AccUDNN: A GPU Memory Efficient Accelerator for Training UltraDeep Neural Networks With the implementation of mainstream DL frameworks, scarce GPU memory resource is the primary bottleneck that hinders the trainability and training efficiency of ultradeep neural networks (UDNN). Prior memory optimization works focus on removing the trainability restriction but leave the training efficiency out of consideration. To fill the gap, we present AccUDNN, an accelerator that aims to make full use of finite GPU memory resource to speed up the training process of UDNN in this paper. AccUDNN mainly includes two modules: memory optimizer and hyperparameter tuner. Memory optimizer develops a novel performancemodel guided dynamic swap outin strategy to meet trainability first and further remedy the efficiency degradation in other swapping strategies. Then, a hyperparameter tuner is designed to explore the efficiencyoptimal minibatch size and the matched learning rate after applying the dynamic swapping strategy. Evaluations demonstrate that AccUDNN cuts down the GPU memory requirement of ResNet152 from more than 24GB to 8GB. In turn, given 12GB GPU memory budget, the efficiencyoptimal minibatch size can reach 4.2x larger than Caffe and finally improve the scaling efficiency (speedup) of 8 GPUsu0027 cluster by 1.9x.

Memory and Interconnect Optimizations for PetaScale Deep Learning Systems Hardware accelerators are a promising solution to the stringent computational requirements of Deep Neural Networks (DNNs). Ranging from lowpower IP cores to server class systems, various accelerator architectures with high TOPSW peak processing efficiencies and flexibility to execute different DNN topologies have been proposed. Prior efforts improve core utilization through better dataflows and computation sequencing, but little effort has thus far been devoted to systematically programming DNN accelerators to extract best possible system utilization, particularly for DNN training, which can be parallelized across petascale systems. In this work, we address the hitherto open challenge of systematically mapping computations onto Petascale accelerator systems, comprising many (thousands of) processing cores spanning many chips, while maximizing overall system performance. We achieve this by characterizing the design space of possible mapping configurations, building a detailed performance model that incorporates every computation and datatransfer involved in DNN training, and using a design space exploration tool called DEEPSPATIALMATRIX to identify the performance optimal configuration. We highlight 4 key optimizations built within DEEPSPATIALMATRIX  hybrid datamodel parallelism, interlayer memory reuse, timestep pipelining, and dynamic spatial minibatching  each of which improve system utilization by carefully managing the available memory capacity and interconnect bandwidth to balance the compute vs. communication costs. On a 8petaFLOP accelerator system, we demonstrate 1.36xc3x9732xc3x97 improvement in training performance through our design space exploration and optimizations across image recognition (VGG16, ResNet50) and machine translation (GNMT) DNN models.