This tutorial presents tools and techniques to optimize the runtime tunable parameters exposed by the MPI using the TAU Performance System® [http://tau.uoregon.edu]. MVAPICH2 [http://mvapich.cse.ohio-state.edu] exposes MPI performance and control variables using the MPI_T interface that is now part of the MPI-3 standard. The tutorial will describe how to use TAU and MVAPICH2 for assessing the application and runtime system performance. We present the complete workflow of performance engineering, including instrumentation, measurement (profiling and tracing, timing, and PAPI hardware counters), data storage, analysis, and visualization. Emphasis is placed on how tools are used in combination for identifying performance problems and investigating optimization alternatives. We will request remote access to the Stampede system at TACC for hands-on exercises. We will also provide the ECP E4S OVA image [https://e4s.io] containing all of the necessary tools (running within a virtual machine) for the hands-on sessions. Participants will learn how to use the TAU Performance System with MPI, OpenMP (OMPT), CUDA, HIP, and OneAPI runtimes, and use the MPI-T interface from the MVAPICH2 library on the Frontera system at TACC and on the VM. This will help to prepare participants to locate and diagnose performance bottlenecks in their own parallel programs.
