Topologyaware Sparse Allreduce for Largescale Deep Learning Data parallelism is the dominant method used to scaleup deep learning (DL) training across multiple compute nodes. Collective communication of the local gradients between nodes is a critical bottleneck due to the significant increase in complexity and size of DL models. Researchers cope with this problem by one of the following solutions: a) optimizing the collective communication algorithm to account for the underlying network topology (topologyaware), and b) reducing the amount of transferred data. In the latter approach, sparse communication techniques communicate only the essential data, which helps to significantly cut down the communication cost. However, the diversity of message sizes, and unknown overlap of indicessets among compute nodes (i.e. irregular size), restricts their communication into the manytomany scheme, which is ineffective in largescale implementations. In this paper, we present allreduce algorithms that can exploit both sparsecommunication and topologyaware techniques by mixing heterogeneous data formats (dense and sparse) with a trivial cost of computation.

On the Feasibility of Hybrid ElectricalOptical Switch Architecture for LargeScale Training of Distributed Deep Learning Data parallelism is the dominant method used to train deep learning (DL) model on HighPerformance Computing systems such as largescale GPU clusters. When training a DL model on a large number of nodes, internode communication becomes bottleneck due to its relatively higher latency and lower link bandwidth (than intranode communication). To cope with this problem, some techniques have been proposed to (a) optimize the collective communication algorithms that take into account the network topology, (b) reduce the message size, and (c) overlap the communication and computation. All of these approaches target to deal with the large message size issue while diminishing the effect of the limitation of the internode network. In this study, we investigate the benefit of increasing internode link bandwidth by using the hybrid switching systems, i.e., Electrical Packet Switching and Optical Circuit Switching. We found that the typical datatransfer of synchronous dataparallelism training are longlive and rarely changed that can be speedup with optical switching. Simulation results on Simgrid simulator show that our approach speedup the training time of deep learning application around 10%.