When using mem_io together with worker, one of course has to set the mem_io password once using mem_io_set_password
. Again, this has to be done only once, see Setting your mem_io password.
mem_io will be started from a worker prolog script, i.e.,
module load mem_io
mem_io_start -nr_channels 300
Here, the number of mem_io channels should be adapted to fit your computations.
The actual work is specified in the worker batch file, and the standard output is redirected to the appropriate mem_io channel.
module load mem_io
some_program "some_input_${id}" | mem_io_store -channel_id ${id}
Note that the channel ID (i.e., the $id
variable in this example) should be in the range 0 to number of channels - 1, inclusive.
The worker epilog file will retrieve the data, save it to a file all_output
, and stop mem_io:
module load mem_io
mem_io_retrieve > all_ouput
mem_io_stop