Breaking Band: A Breakdown of Highperformance Communication The critical path of internode communication on largescale systems is composed of multiple components. When a supercomputing application initiates the transfer of a message using a highlevel communication routine such as an MPI_Send, the payload of the message traverses multiple software stacks, the IO subsystem on both the host and target nodes, and network components such as the switch. In this paper, we analyze where, why, and how much time is spent on the critical path of communication by modeling the overall injection overhead and endtoend latency of a system. We focus our analysis on the performance of small messages since finegrained communication is becoming increasingly important with the growing trend of an increasing number of cores per node. The analytical models present an accurate and detailed breakdown of time spent in internode communication. We validate the models on Arm ThunderX2based servers connected with Mellanox InfiniBand. This is the first work of this kind on Arm. Alongside our breakdown, we describe the methodology to measure the time spent in each component so that readers with access to precise CPU timers and a PCIe analyzer can measure breakdowns on systems of their interest. Such a breakdown is crucial for software developers, system architects, and researchers to guide their optimization efforts. As researchers ourselves, we use the breakdown to simulate the impacts and discuss the likelihoods of a set of optimizations that target the bottlenecks in todayu0027s highperformance communication.

Evaluating the Impact of Energy Efficient Networks on HPC Workloads Interconnection networks grow larger as supercomputers include more nodes and require higher bandwidth for performance. This scaling significantly increases the fraction of power consumed by the network, by increasing the number of network components (links and switches). Typically, network links consume power continuously once they are turned on. However, recent proposals for energy efficient interconnects have introduced lowpower operation modes for periods when network links are idle. Lowpower operation can increase messaging time when switching a link from lowpower to active operation. We extend the TraceRCODES network simulator for power modeling to evaluate the impact of energy efficient networking on power and performance. Our evaluation presents the first study on both singlejob and multijob execution to realistically simulate power consumption and performance under congestion for a largescale HPC network. Results on several workloads consisting of HPC proxy applications show that singlejob and multijob execution favor different modes of low power operation to have significant power savings at the cost of minimal performance degradation.