PSNet: Reconfigurable network topology design for accelerating parameter server architecture based distributed machine learning Abstract   The bottleneck of Distributed Machine Learning (DML) has shifted from computation to communication. Lots of works have focused on speeding up communication phase from perspective of Parameter Server (PS) architecture, for example resource scheduling. Nonetheless, the performance improvement of these schemes is limited due to the agnostic of the physical topology to the communication pattern of the applications. Concurrently, some articles have also pointed out the impact of topology on DML performance. Besides, our analysis and experimental results also indicate that the general topologies cannot match well with the communication characteristics of DML based on PS architecture. However, to the best of our knowledge, no special topology is tailored for DML. Therefore, in this paper, we propose PSNet, a reconfigurable modular network topology for DML with consideration of the communication characteristics of PS architecture. The main idea of PSNet is that servers are firstly divided into two categories, namely workers configured with highperformance computing capability and parameter servers equipped with multiple Network Interface Cards (NICs). Then Electrical Circuit Switch (ECS) is exploited to connect workers and Top of Rack (ToR) switches for flexibility and reconfigurability in each module. Our theoretical analysis proves that PSNet not only provides high performance for DML tasks, but also achieves high fault tolerance and flexibility. In order to validate the performance of PSNet, we conduct largescale simulations and smallscale testbed experiments, and the results of experiments demonstrate that PSNet performs     1  .  89  xc3x97     and     1  .  92  xc3x97     faster than FatTree for VGG16 and ResNet50, respectively.

DiameterASPLBased Mapping of Applications with Uncertain Communication over Random Interconnection Networks The steadily increasing number of compute nodes in highperformance computing (HPC) systems calls for more efficient interconnection networks and application mapping strategies. Traditionally, the user applications are mapped onto a regular network with nearby compute nodes to reduce the network distances. However, it is likely that the rigid mapping limitation on regular topologies may largely delay the dispatch time when the workload is heavy at a short period. In this work, we target the applications with unknown communication patterns over random interconnection networks, which have drawn increasing attention in the HPC world due to their low diameter and low average shortest path length (ASPL). We propose to use topology embedding metrics, i.e., diameter and ASPL, for application mapping to overcome the rigid limitation on regular topologies. We investigate the timespace tradeoff among several application mapping algorithms using a compound application workload. Evaluation results show that, when compared to the baseline random connected topology mapping, the proposed diameterASPLbased topology mapping algorithms reduce up to 48.0% makespan and up to 78.1% average turnaround time and improve up to 1.9X system utilization when the instantaneous workload is heavy over random interconnection networks.