Effective IO Bandwidth b_eff_io Benchmark

It covers two goals:

achieve a characteristic average number for the I/O bandwidth achievable with parallel MPI-I/O applications
get detailed information about several access patterns and buffer lengths. The benchmark examines "first write", "rewrite" and "read" access, strided (individual and shared pointers) and segmented collective patterns on one file per application and non-collective access to one file per process. The number of parallel accessing processes is also varied and wellformed I/O is compared with non-wellformed. On systems, meeting the rule that the total memory can be written to disk in 10 minutes, the benchmark should not need more than 15 minutes for a first pass of all patterns. The benchmark is designed analogously to the effective bandwidth benchmark for message passing (b_eff) that characterizes the message passing capabilities of a system in a few minutes.

The Latest Releases

Current release as gzip'ed tar archive: b_eff_io_v2.1.tar.gz

Files of this release: b_eff_io.c, beffio_eps, man page (html formatted), man/man1/b_eff_io.1
Helper for b_eff_io_eps: beffio_part_eps, beffio_part_eps.awk, beffio_n_eps.gnuplot, beffio_n_eps_rewrt.gnuplot, beffio_part_eps.gnuplot, beffio_part_eps_rewrt.gnuplot
Source of "on 1 page": beffio_part_eps_on1page.tex

Old releases: 1.3, 1.2, 1.1, 1.0, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1.
A detailed report with first results of b_eff_io release 0.5 can be obtained here.

Latest publications:

Rolf Rabenseifner, Alice E. Koniges, Jean-Pierre Prost, and Richard Hedges: The Parallel Effective I/O Bandwidth Benchmark: b_eff_io.
To be published in Special Issue of Calculateurs Parallèles Journal on Parallel I/O for Cluster Computing. (CFP) Files (2nd submission): reference, PDF document, postscript, gzip'ed postscript.

Rolf Rabenseifner and Alice E. Koniges: Effective Communication and File-I/O Bandwidth Benchmarks.
In J. Dongarra and Yiannis Cotronis (Eds.), Recent Advances in Parallel Virtual Machine and Message Passing Interface, proceedings of the 8th European PVM/MPI Users' Group Meeting, EuroPVM/MPI 2001, Santorini, Greece, Sep. 23-26, LNCS 2131, pp 24-35. Files: Paper as reference, PDF document (180kB), postscript (336kB), gzip'ed postscript (92kB); slides as reference, PDF document (0.4MB), postscript (3.1MB), gzip'ed postscript (1.3MB).

Rolf Rabenseifner and Alice E. Koniges: The Parallel Communication and I/O Bandwidth Benchmarks: b_eff and b_eff_io.
In proceedings of the 43nd Cray User Group Conference, CUG SUMMIT 2001, May 21-25, Indian Wells, California, USA. Files: Abstract, full paper as reference, PDF document, postscript, gzip'ed postscript; slides as reference, PDF document, postscript, gzip'ed postscript.

Alice E. Koniges, Rolf Rabenseifner and Karl Solchenbach: Benchmark Design for Characterization of Balanced High-Performance Architectures.
In IEEE Computer Society Press, www.computer.org/proceedings/, proceedings of the 15th International Parallel and Distributed Processing Symposium (IPDPS'01), Workshop on Massively Parallel Processing (WMPP), April 23-27, 2001, San Francisco, USA. Files: full paper as PDF and gzip'ed postscript, slides as PDF and gzip'ed postscript.

Rolf Rabenseifner and Alice E. Koniges: Effective File-I/O Bandwidth Benchmark.
In proceedings, Arndt Bode, Thomas Ludwig, Roland Wismüller (editors), Euro-Par 2000 -- Parallel Processing, Aug. 29 - Sept. 1, 2000, München, Germany, LNCS 1900, pp 1273-1283. Files: full paper as PDF and gzip'ed postscript, slides as gzip'ed postscript.

Rolf Rabenseifner and Alice E. Koniges: The Effective I/O Bandwidth Benchmark (b_eff_io).
In proceedings of the Message Passing Interface and High-Performance Clusters Developer's and User's Conference (MPIDC 2000), March 20-23, 2000, Ithaca, NY, USA. Files: full paper as US paper, gzip'ed postscript, A4 gzip'ed postscript; slides as gzip'ed postscript.

Usage

Installation and the first test

Download the tar file of the current release b_eff_io_v2.1.tar.gz

Unpack with: gunzip -c b_eff_io_v2.1.tar.gz | tar -xvf -

Change directory: cd b_eff_io

Compile it: mpicc -o b_eff_io b_eff_io.c -lm
If shared filepointer MPI-2 routines are not available (e.g. you are using an old ROMIO), then for a first test you can use:
mpicc -o b_eff_io -D WITHOUT_SHARED b_eff_io.c -lm
But, to achieve valid b_eff_io results, you have to install an MPI library that allows shared file-pointers.

For a first test, I recommend a short run (60 sec) without Shared Filepointer and with only one fixed number of processes, e.g.:

mpirun -np 32 ./b_eff_io -MB 2048 -MT 81920 -noshared -rewrite -N 32 -T 60 -p /MY_fast_filesystem/MY_directory -f MYSYS_32pe_0060sec_noshared

The option -MB 2048 means, that each MPI process runs on a processor with 2 GB memory (= 2048 MB).
The option -MT 81920 means, that the total system has 80 GB (=81920MB), i.e. in our example we are on a 40 processor system with 40*2 GB.
The option -rewrite is necessary to produce all detailed results for plotting.
The option -T 60 means, that the test is scheduled to run at least 60 seconds. In reality, this test needs more than 60 seconds, because each pattern is done at least once, but it should run in not more than 4 minutes.
The option -N 32 is necessary, because the default is -N 32,16,4 if MPI_COMM_WORLD has 32 processes.
This I/O benchmark uses large scratch files. They are stored in /MY_fast_filesystem/MY_directory/.
After execution, you should check MYSYS_32pe_0060sec_noshared.prot on errors.
You will get back:
- on standard output : the b_eff_io value
- on b_eff_io.sum : a human readable summary
- on b_eff_io.prot : the full benchmark protocol

Print the summary, e.g., with: a2ps -C -1 -l 120 b_eff_io.sum

If there are serious problems, e.g., the benchmark took more than 4 minutes on a dedicated system or there are errors reported, then please, feel free to contact the author and send him all your commands and attach the files MYSYS_008pe_0060sec_noshared.sum and MYSYS_008pe_0060sec_noshared.prot.

If all is okay and gnuplot and dvips are available, you can generate some plots from the summary with:
beffio_eps MYSYS_008pe_0060sec_noshared. The 32-processes-results in MYSYS_008pe_0060sec_noshared.sum will be extracted and plotted (.eps and .png). The integrated call to dvips will produce a sheet: MYSYS_008pe_0060sec_noshared_32_on1page.ps

Print the summary sheet, e.g., with: lpr MYSYS_008pe_0060sec_noshared_32_on1page.ps

Now, you can test the 2 pattern type with shared filepointers:

mpirun -np 32 ./b_eff_io -MB 2048 -MT 81920 -rewrite -N 32 -T 60 -p /MY_fast_filesystem/MY_directory -f MYSYS_32pe_0060sec_withshared

After this, I recommend a medium test (10min = 600sec) and now with 3 different processor numbers (here 32/16/4):

mpirun -np 32 ./b_eff_io -MB 2048 -MT 81920 -rewrite -N 32,16,4 -T 600 -p /MY_fast_filesystem/MY_directory -f MYSYS_32.16.4pe_0600sec_withshared

The "time driven" concept is realized only partially, therefore the real execution time can be significantly longer.

CAUTION: Because this first test is scheduled with only 60 seconds, the results will never tell you anything about the I/O bandwidth of your system. The test only should tell you whether the benchmark is running on your system.

To get a first b_eff_io impression

And now you are prepared for the real benchmark on 3 different processor numbers (here 32/16/4) and with 45 minutes (2700sec) because you are using -rewrite (the default 1800 sec should be used only without -rewrite, but you want to see the details, and therefore you use -rewrite).

mpirun -np 32 ./b_eff_io -MB 2048 -MT 81920 -rewrite -N 32,16,4 -T 2700 -p /MY_fast_filesystem/MY_directory -f MYSYS_32.16.4pe_2700sec_withshared

Running beffio_eps MYSYS_32.16.4pe_2700sec_withshared gives the following results:
MYSYS_32.16.4pe_2700sec_withshared.sum, MYSYS_32.16.4pe_2700sec_withshared_32_on1page.ps, MYSYS_32.16.4pe_2700sec_withshared_16_on1page.ps, MYSYS_32.16.4pe_2700sec_withshared_04_on1page.ps, MYSYS_32.16.4pe_2700sec_withshared.prot

#MPROC is used in the next section to abbreviate the memory size of each MPI process (in MBytes), e.g., in our example #MPROC=2048.

#MTOTAL is used to abbreviate the total memory size of the system (in MBytes), e.g., in our example #MTOTAL=81920.

The official execution of b_eff_io
The b_eff_io benchmark has to be done with three mandatory and one optional number of processes:

First, #MPI_PROC_PER_SMP_NODE must be chosen by the person who makes the benchmark. This number is the number of MPI processes that should run on each SMP node. This number is tunable and must not be more than the number of processors of each SMP node.
On systems that are not a cluster of SMP nodes, #MPI_PROC_PER_SMP_NODE is 1.

Next, #NODES_FULL is defined as the number of SMP nodes of the system. On systems that are not a cluster of SMP nodes, #NODES_FULL is the number of processors available for parallel computation.

Thus you have to compute the following numbers:

          #NODES_MEDIUM =  2 ** ( round ( log_2(#NODES_FULL) * 0.70 ) )
          #NODES_SMALL  =  2 ** ( round ( log_2(#NODES_FULL) * 0.35 ) )

Examples: The following table shows the MID and SMALL values for given FULL values

          #NODES_FULL       = 2048 1024 512 256 128  64  32  16  8  4  2  1
          ==> #NODES_MEDIUM =  256  128  64  64  32  16  16   8  4  2  2  1
          ==> #NODES_SMALL  =   16   16   8   8   4   4   4   2  2  2  1  1

And last, you can freely choose #NODES_TUNE as the number of nodes that gives the best b_eff_io value.

Each of these b_eff_io measurements should be done with 30 minutes scheduled time (-T 1800), i.e., we would like to see the results of
```
          mpirun 'on #NODES_SMALL  with #MPI_PROC_PER_SMP_NODE' \
                 ./b_eff_io -T 1800 -MB #MPROC -MT #MTOTAL -T 1800 \
                            -p /my/fast/scratch/dir \
                            -f my_system_#NODES_SMALL_1800sec

          mpirun 'on #NODES_MEDIUM with #MPI_PROC_PER_SMP_NODE' \
                 ./b_eff_io -T 1800 -MB #MPROC -MT #MTOTAL -T 1800 \
                            -p /my/fast/scratch/dir \
                            -f my_system_#NODES_MEDIUM_1800sec
 
	  mpirun 'on #NODES_FULL   with #MPI_PROC_PER_SMP_NODE' \
		 ./b_eff_io -T 1800 -MB #MPROC -MT #MTOTAL -T 1800 \
			    -p /my/fast/scratch/dir \
                            -f my_system_#NODES_FULL_1800sec
 
          mpirun 'on #NODES_TUNED  with #MPI_PROC_PER_SMP_NODE' \
                 ./b_eff_io -T 1800 -MB #MPROC -MT #MTOTAL -T 1800 \
                            -p /my/fast/scratch/dir \
                            -f my_system_#NODES_TUNED_1800sec
	  
```
#NODES_TUNED may be chosen as one of the three other values that has the best b_eff_io result (i.e. #NODES_TUNED is optional).
Filesystem parameters should be chosen as for normal users. The size of data written to /my/fast/scratch/dir by each of these four benchmarks is about real_execution_time * accumulated_write_bandwidth / 3. The real execution time may differ from the scheduled time (30 min.) due to following reasons:
1. The sync operation is outside of the time-driven loop and may consume time after the scheduled iterations.
2. The loop is finished by an iteration that con sumed much more time than the previous iterations.
3. The pattern types 3 (segmented) and 4 (seg-coll) are not time-driven. The estimation for adequate repeating factors is based on results with pattern types 0-2. This estimation may be to high if the implementation of pattern types 3 and 4 is worse than that of pattern type 0 and 2.
4. The same reason is valid for all patterns with the access methods "rewrite" and "read".

Publishing the results
Now, you can publish these four b_eff_io values together with all commands and parameters you have used to run these benchmarks and together with the protocol files (my_system_..._1800sec.prot) and the summary files (my_system_..._1800sec.sum). The Top Cluster initiative of the TFCC Open Forum has nominated this benchmark for evaluating the I/O performance of clusters (see discussion archive).
It is planned to include the b_eff_io results into the TOPClusters list.

References

[1]: Karl Solchenbach: Benchmarking the Balance of Parallel Computers. SPEC Workshop on Benchmarking Parallel and High-Performance Computing Systems (copy of the slides), Wuppertal, Germany, Sept. 13, 1999.
[2]: Karl Solchenbach, Hans-Joachim Plum and Gero Ritzenhoefer: Pallas Effective Bandwidth Benchmark - source code and sample results ( EFF_BW.tar.gz , 43 KB)
[3]: Rolf Hempel: Basic message passing benchmarks, methodology and pitfalls. SPEC Workshop on Benchmarking Parallel and High-Performance Computing Systems ( copy of the slides ), Wuppertal, Germany, Sept. 13, 1999.
[4]: William Gropp and Ewing Lusk: Reproducible Measurement of MPI Performance Characteristics. In J. Dongarra et al. (eds.), Recent Advances in Parallel Virtual Machine and Message Passing Interface, proceedings of the 6th European PVM/MPI Users' Group Meeting, EuroPVM/MPI'99, Barcelona, Spain, Sept. 26-29, 1999, LNCS 1697, pp 11-18. (Summary on the web)

Effective IO Bandwidth b_eff_io Benchmark

The Latest Releases

Usage

References

Links