Exercise 3

Overview:

• Login to the LC workshop cluster, if you are not already logged in.
• Following the Exercise 3 instructions will take you through all sorts of MPI programs - pick any/all that are of interest.
• The intention is review the codes and see what’s happening - not just compile and run.
• Several codes provide serial examples for a comparison with the parallel MPI versions.
• Check out the “bug” programs.

1. Still logged into the workshop cluster?

If so, then continue to the next step. If not, then login as you did previously for Exercises 1 and 2.

2. Review the array decomposition example code.

First, review the serial version of this example code, either ser_array.c or ser_array.f.

After you understand what’s going on, review the parallel MPI version, either mpi_array.c or mpi_array.f. The comments explain how MPI is used to implement a parallel data decomposition on an array.

3. Compile the parallel MPI and serial versions of the array decomposition example code

As with Exercises 1 & 2, use the compiler command of your choice to compile the mpi_array example code.

Use the appropriate serial compiler command for the serial version. For example:

C:

icc ser_array.c  -o ser_array
mpicc mpi_array.c  -o mpi_array

Fortran:

ifort ser_array.f -o ser_array
mpif77 mpi_array.f -o mpi_array

4. Run the executables interactively

For the MPI executable, use the special workshop pool and 8 tasks. For example:

Serial:

ser_array

MPI:

srun -n8 -ppReserved mpi_array

Note: The srun command is covered in detail in the “Starting Jobs” section of the Linux Clusters Overview tutorial, located here. There is also a man page.

5. Compare other serial codes to their parallel version

If we had more time, you might even be able to start with a serial code or two and create your own parallel version. Feel free to try if you’d like.

6. Try any/all of the other MPI example codes

• First, review the code(s) so that you understand how MPI is being used.
• Then, using the MPI compiler command(s) of your choice, compile the codes of interest.
• For convenience, the included Makefiles can be used to compile any or all of the exercise codes. For example:

C:

make -f Makefile.MPI.c
make -f Makefile.MPI.c  mpi_mm
make -f Makefile.Ser.c

Fortran:

make -f Makefile.MPI.f
make -f Makefile.MPI.f  mpi_mm
make -f Makefile.Ser.f

Note: you can change the compiler being used by editing the Makefile.

• Run the executables interactively in the special workshop pool. Use the srun command for this as shown previously. Most of the executables only need 4 MPI tasks or less. Some exceptions and notes:

Code	Requirement(s)
mpi_bandwidth	Requires an even number of tasks.
mpi_cartesian	Requires 16 MPI tasks
mpi_group	Requires 8 MPI tasks
mpi_heat2D mpi_wave	These examples attempt to generate an Xwindows display for results. You will need to make sure that your Xwindows environment and software is setup correctly if you want to see the graphic. Ask the instructor if you have any questions.
mpi_latency	Requires only 2 MPI tasks that should be on DIFFERENT nodes

• Some things to try:
  • Different compilers
  • Experiment with compiler flags (see respective man pages).
  • Vary the number of tasks and nodes used.

7. Compare per task and aggregate communications bandwidths

• Compile the mpi_bandwidth code if you haven’t already.
• Run the code interactively with 2 tasks on two different nodes:

  srun -N2 -n2 -ppReserved mpi_bandwidth
  

• Note the overall average bandwidth for the largest message size of 1,000,000 bytes.
• Now run the code interactively with 4, 8, 16, 32 and 64 tasks on two different nodes:

srun -N2 -n4 -ppReserved mpi_bandwidth
srun -N2 -n8 -ppReserved mpi_bandwidth
srun -N2 -n16 -ppReserved mpi_bandwidth
srun -N2 -n32 -ppReserved mpi_bandwidth
srun -N2 -n64 -ppReserved mpi_bandwidth

• Note the average bandwidths as before.
• What are your observations? Why?

Explanation (Click to expand!)

As the number of tasks increase, the per task bandwidth decreases because they must compete for use of the network adapter. Aggregate bandwidth will increase until it plateaus.

8. Compare blocking send/receive with non-blocking send/receive

• Copy your mpi_bandwidth source file to another file called mpi_bandwidthNB. Modify your new file so that it performs non-blocking sends/receives instead of blocking. An example mpi_bandwidth_nonblock file has been provided in case you need it.
• After you’re satisfied with your new non-blocking version of the bandwidth code, compile both.
• Run each code using two tasks on two different nodes in the special workshop pool:

  srun -N2 -ppReserved mpi_bandwidth
  srun -N2 -ppReserved mpi_bandwidthNB
  

• Compare the results. Which one performs best?

Explanation (Click to expand!)

Non-blocking send/receive operations are often significantly faster than blocking send/receive operations.

9. When things go wrong…

There are many things that can go wrong when developing MPI programs. The mpi_bug series of programs demonstrate just a few. See if you can figure out what the problem is with each case and then fix it.

Compile with the compile command(s) of your choice and run interactively using 4 tasks in the special workshop pool.

The buggy behavior will differ for each example. Some hints are provided below.

Hints (Click to expand!)

Code	Behavior	Hints/Notes
mpi_bug1	Hangs	mpi_bug1 demonstrates how miscoding even a simple parameter like a message tag can lead to a hung program. Verify that the message sent from task 0 is not exactly what task 1 is expecting and vice versa. Matching the send tags with the receive tags solves the problem.
mpi_bug2	Wrong results or abnormal termination	mpi_bug2 shows another type of miscoding. The data type of the message sent by task 0 is not what task 1 expects. Nevertheless, the message is received, resulting in wrong results or abnormal termination - depending upon the MPI library and platform. Matching the send data type with the receive data type solves the problem.
mpi_bug3	Error message and/or abnormal termination	mpi_bug3 shows what happens when the MPI environment is not initialized or terminated properly. Inserting the MPI init and finalize calls in the right locations will solve the problem.
mpi_bug4	Gives the wrong result for "Final sum". Compare to mpi_array	Number of MPI tasks must be divisible by 4; mpi_bug4 shows what happens when a task does not participate in a collective communication call. In this case, task 0 needs to call MPI_Reduce as the other tasks do.
mpi_bug5	Dies or hangs - depends upon platform and MPI library	mpi_bug5 demonstrates an unsafe program, because sometimes it will execute fine, and other times it will fail. The reason why the program fails or hangs is due to buffer exhaustion on the receiving task side, as a consequence of the way an MPI library has implemented an eager protocol for messages of a certain size. One possible solution is to include an MPI_Barrier call in the both the send and receive loops.
mpi_bug6	Terminates or is ignored (depends on platform/language)	Requires 4 MPI tasks; mpi_bug6 has a bug that will terminate the program in some cases but be ignored in other cases. The problem is that task 2 performs a blocking operation, but then hits the MPI_Wait call near the end of the program. Only the tasks that make non-blocking calls should hit the MPI_Wait. The coding error in this case is easy to fix - simply make sure task 2 does not encounter the MPI_Wait call.
mpi_bug7	Hangs	mpi_bug7 performs a collective communication broadcast but erroneously codes the count argument incorrectly resulting in a hang condition.

If you’re just finishing the tutorial and haven’t filled out our evaluation form yet, please do!