Parallel Computation

The HQR computations can be performed on several machines at a time as well as on multiple processors on a given machine. These two modes are not mutually exclusive.

Computation on several processors

The NUMBER_OF_CONCURRENT_JOBS variable into the jobfiles, as well as the Threads spin-box into the Ray tracing tab of the graphics user interface, setup the number of concurrent process which will be used during the ray-tracing phase.

Limitations

Some options are not fully compatible with multi-cpu computation now. This is the case of the radiance precomputation feature, which is not protected by a lock for efficiency reasons. Using it is not forbidden, but it may crash the computation once every 300,000 pixels.

Computation on several machines

The computation on several machines of a single image is performed by separating a given jobfile for this image into sub-job files and building a queue of corresponding commands to disptach on different machines. This is made easy by a set of three scripts located into the HQR/Builds/i686-linux-g++/bin directory, which must be accessible on your path:

hqr_cluster.sh: generates sub-job files and a bunch of cool scripts to clean/recombine/launch jobs.
ChildProcess_ita: handles processing of the job queue, taking care of lock files, etc.
launch_process: this script launches the computation on a given machine.

Notes:

the different machines used do not need to have the same system (e.g. a RedHat and a Mandrake), but the HQR executable must be able to run on each of them. Check for this before.
be careful about image normalization. The NORMALIZE variable in the master job file should be set to anything but AUTO, otherwise the different images will be scale differently and may produce horizontal bands when combined together after the computation.
check that the scene file always is a relative path and not an absolute path, which may not be reachable from the distant machines.

Let's process into a commented example:

Example

In this commented example, we compute an image of a large scene with high parameter settings. The total expected computation time is a couple of days, but distributing it on 6 machines will put it down to around 8 hours.

Step 1: separate the job file into 12 sub-job files:

	> hqr_cluster.sh LBxStudio.hard.3.hqr 12
	Generating job files for 12 cluster nodes
	job number 0 concerns lines 0 to 99
	job number 1 concerns lines 100 to 199
	job number 2 concerns lines 200 to 299
	job number 3 concerns lines 300 to 399
	job number 4 concerns lines 400 to 499
	job number 5 concerns lines 500 to 599
	job number 6 concerns lines 600 to 699
	job number 7 concerns lines 700 to 799
	job number 8 concerns lines 800 to 899
	job number 9 concerns lines 900 to 999
	job number 10 concerns lines 1000 to 1099
	job number 11 concerns lines 1100 to 1199
	job number 12 concerns lines 1200 to 1204
	Created command files for your convenience:
	   sh.LBxStudio.hard.3.hqr : command to launch on each station.
	    => Use ssh -n "cd /home/artis/csoler/SRC/REALREFLECT/doc/manual/tmp2; nice +17 sh.LBxStudio.hard.3.hqr"
	   joblist.LBxStudio.hard.3.hqr : job list
	   clean.LBxStudio.hard.3.hqr : clean everything when finished.
	   sh_exrcombine.LBxStudio.hard.3.hqr : to combine result exr files.
	   sh_launch.LBxStudio.hard.3.hqr : launch the computation

As said by the hqr_cluster.sh script, some files were created in the current directory. These are:

	clean.LBxStudio.hard.3.hqr*
	joblist.LBxStudio.hard.3.hqr
	sh.LBxStudio.hard.3.hqr*
	sh_exrcombine.LBxStudio.hard.3.hqr*
	sh_launch.LBxStudio.hard.3.hqr*

	subjob.10.LBxStudio.hard.3.hqr
	subjob.11.LBxStudio.hard.3.hqr
	subjob.12.LBxStudio.hard.3.hqr
	subjob.9.LBxStudio.hard.3.hqr
	subjob.1.LBxStudio.hard.3.hqr
	subjob.2.LBxStudio.hard.3.hqr
	subjob.3.LBxStudio.hard.3.hqr
	subjob.4.LBxStudio.hard.3.hqr
	subjob.5.LBxStudio.hard.3.hqr
	subjob.6.LBxStudio.hard.3.hqr
	subjob.7.LBxStudio.hard.3.hqr
	subjob.8.LBxStudio.hard.3.hqr
	subjob.0.LBxStudio.hard.3.hqr

The subjob.* files contain the description of the computation to perform as a subpart of what is described into LBxStudio.hard.3.hqr. Every parameters of the original file are kept as such. Only the output file names and Y_START and Y_END values are made specific.

The joblist.LBxStudio.hard.3.hqr file contains the queue of unfinished jobs. This file should not be used directly, but through the sh.LBxStudio.hard.3.hqr dedicated script, which handles lock files from distant machines.

Step 2: launch the computation on 5 distant machines:

	> launch_process aigrette  sh.LBxStudio.hard.3.hqr
	> launch_process cumin     sh.LBxStudio.hard.3.hqr
	> launch_process aneth     sh.LBxStudio.hard.3.hqr
	> launch_process houblon   sh.LBxStudio.hard.3.hqr
	> launch_process psilocybe sh.LBxStudio.hard.3.hqr

... and for instance an additional one on the local machine:

	> ./sh.LBxStudio.hard.3.hqr

Notes:

the launch_process script basically perform something like:
```
 ssh -n <machine name> "cd $PWD ; nice +17 sh.LBxStudio.hard.3.hqr >& <machine name>.sh.LBxStudio.hard.3.hqr.std" 
```
Each job launched this way thus produces a output text file which can be read at any time.
the number of machine used in not related to the number of sub-job files generated at step 1. Considering the fixed cost of launching photons on each machine, the number of sub-jobs should be greater than the number of machines used, but not too much.

At the end of the calculation, you should get the separate images which are shown here ten/two times smaller:

Step 3: recombine results.

	> ./sh_exrcombine.LBxStudio.hard.3.hqr

This adds the separately computed parts of the image into a final HDR image with the name specified in the original job file, which you can see with exrdisplay:

	> exrdisplay ./out.LBxStudio.hard.3.exr

This command can be used anytime during the computation to see what the current state is. In particular the separate results do not cause errors when they are not present. Here is the final image in full resolution.

Last modified on Thursday, March 16, 2005.