Abstract-Six of the ten fastest supercomputers in the world in 2014 use a torus interconnection network for message passing between compute nodes. Torus networks provide high bandwidth links to near-neighbors and low latencies over multiple hops on the network. However, large diameters of such networks necessitate a careful placement of parallel tasks on the compute nodes to minimize network congestion. This paper presents a methodological study of optimizing application performance on a five-dimensional torus network via the technique of topology-aware task mapping. Task mapping refers to the placement of processes on compute nodes while carefully considering the network topology between the nodes and the communication behavior of the application. We focus on the IBM Blue Gene/Q machine and two production applicationsa laser-plasma interaction code called pF3D and a lattice QCD application called MILC. Optimizations presented in the paper improve the communication performance of pF3D by 90% and that of MILC by up to 47%. Keywords-task mapping, 5D torus, performance, congestion The scale of large parallel machines at the disposal of computational scientists is unprecedented. The top five machines on the Top500 list [1] have over half a million cores. These machines have complex on-node architectures and their compute nodes are connected together by high-speed interconnection networks. The rapid increase of on-node computational power in proportion to network speeds suggests that optimizing communication is important to ensure high efficiency and optimal scaling of parallel applications. Task mapping is a technique to optimize communication of parallel applications on the interconnection network without having to modify the source code [2] . The technique places tasks or processes on compute nodes based on a careful consideration of the interconnection topology between the nodes and the communication behavior of the application. This is especially important on torus networks because their large diameters can require messages to travel multiple hops to reach their destination, thereby increasing the burden on the shared links. A torus or mesh network topology has been commonly used to connect compute nodes since the Cray T3D machine was developed twenty years ago. Six of the ten fastest supercomputers in 2014 use a torus network for message passing between compute nodes. The K computer uses the Tofu interconnect which is a six-dimensional (6D) mesh/torus. The IBM Blue Gene/P architecture uses a three-dimensional torus and its successor, the IBM Blue Gene/Q, uses a five-dimensional (5D) torus. Several generations of Cray machines (T3D, T3E, XT3/4/5, XE6, XK7) from 1993 to the present have used a threedimensional torus as their communication backbone. In this paper, we focus on the 5D torus network deployed in the IBM Blue Gene/Q (BG/Q) machines. Task mapping is an NP-hard problem [3] . The complexity of conceptualizing a 5D torus (as opposed to a 3D one) makes it even more challenging to develop near-optimal mapping heuristics. We use Rubik, a tool developed at LLNL to map applications with structured/Cartesian process grids to arbitrary n-dimensional meshes/tori [4] . We present a stepby-step methodology to improve application performance using topology-aware task mapping, based on our experience with optimizing production codes on BG/Q. There are three steps involved in improving application performance using topology-aware task mapping: 1) Performance debugging via profiling, 2) Performance optimization via task mapping, and 3) Performance analysis via profiling and visualization. We present our experience of applying task mapping to improve the performance of two highly scalable, production applications -pF3D, a laser-plasma interaction code, and MILC, a lattice quantum chromodynamics (QCD) application. We provide a detailed analysis of the cases in which task mapping improves application performance and the reasons for the performance benefits. We compare the improvements to the change in the average number of hops traveled by messages and the maximum load/congestion on the network links. Task mapping improves the communication performance of pF3D by 2.8× on 131,072 processes and that of MILC by 47% on 65,536 processes. An easily correctable performance bug found when profiling pF3D for this study gives an additional 3.9× improvement. We believe that the methodology presented here will be useful for improving the performance of a broad class of computational science and engineering applications on torus architectures.
The study of optimizing application performance on a 5-D torus network via the topology-aware task mapping technique has been presented in REF .
Optimizing the performance of parallel applications on a 5D torus via task mapping
2014 21st International Conference on High Performance Computing (HiPC)
2014 21st International Conference on High Performance Computing (HiPC)
Computer Science