Without mem_io

Consider the following script that would perform parallel, indepedent computations on a compute node that has a total of 20 cores:

nr_cores=20
for id in `seq 0 $(( ${nr_cores} - 1 ));
do
    some_program  "some_input_${id}"  >  "some_output_${i}" &
done
wait

for id in `seq 0 $(( ${nr_cores} - 1 ));
do
    cat  "some_output_${i}"  >>  "all_output"
done

Each of the, in this case, 20 instances of the some_program application will perform writes on standard output, which is redirected to individual files. When the computations have been performed in parallel, the resulting individual output files are aggregated into a single file.

Needless to say that such an approach will lead to many I/O, and meta data operations. This particular case is pretty mild when compared to more massively parallelized computation using the worker framework or nitro (Adaptive Computing).

With mem_io

mem_io can be of help here, with a very minor modification to the workflow, and none to the application some_program. Consider:

nr_cores=20

mem_io_start  -nr_channels ${nr_cores}

for id in `seq 0 $(( ${nr_cores} - 1 ));
do
    some_program  "some_input_${id}"  |  mem_io_store  -channel_id ${id}
done
wait

mem_io_retrieve > all_ouput
mem_io_stop

The mem_io_start command prepares a configuration file, and starts an in-memory database that will be used to store the I/O. For details, see Starting mem_io.

Rather than redirecting the standard output of processes to a file, it is piped through mem_io_store which will deal with the file I/O on behalf of the processes. For details, see Writing data to mem_io.

When all computations are done, the output is retrieved from the database by the mem_io_retrieve command, and can be redirected to a file, all_output in this example. For details, see Retrieving data from mem_io.

Lastly, the in-memory will be shut down cleanly by mem_io_stop. For detals, see Stopping mem_io.

Note that mem_io_store is agnostic about the nature of the processes' output, which may be text or binary. However, for binary output, a simple concatenation of individual output will probably not result in a valid aggregated output file, so the output needs to be retrieved and redirected on a per channel basis using mem_io_retrieve -channel_id option.