ArchitectureCentric Bottleneck Analysis for Deep Neural Network Applications The evergrowing complexity and popularity of machine learning and deep learning applications have motivated an urgent need of effective and efficient support for these applications on contemporary computing systems. In this paper, we thoroughly analyze the various DNN algorithms on three widely used architectures (CPU, GPU, and Xeon Phi). The DNN algorithms we choose for evaluation include i) Unet  for biomedical image segmentation, based on Convolutional Neural Network (CNN), ii) NMT  for neural machine translation based on Recurrent Neural Network (RNN), iii) ResNet50, and iv) DenseNet  both for image processing based on CNNs. The ultimate goal of this paper is to answer four fundamental questions: i) whether the different DNN networks exhibit similar behavior on a given execution platform? ii) whether, across different platforms, a given DNN network exhibits different behaviors? iii) for the same execution platform and the same DNN network, whether different execution phases have different behaviors? and iv) are the current major generalpurpose platforms tuned sufficiently well for different DNN algorithms? Motivated by these questions, we conduct an indepth investigation of running DNN applications on modern systems. Specifically, we first identify the most timeconsuming functions (hotspot functions) across different networks and platforms. Next, we characterize performance bottlenecks and discuss them in detail. Finally, we port selected hotspot functions to a cycleaccurate simulator, and use the results to direct architectural optimizations to better support DNN applications.

Memory and Interconnect Optimizations for PetaScale Deep Learning Systems Hardware accelerators are a promising solution to the stringent computational requirements of Deep Neural Networks (DNNs). Ranging from lowpower IP cores to server class systems, various accelerator architectures with high TOPSW peak processing efficiencies and flexibility to execute different DNN topologies have been proposed. Prior efforts improve core utilization through better dataflows and computation sequencing, but little effort has thus far been devoted to systematically programming DNN accelerators to extract best possible system utilization, particularly for DNN training, which can be parallelized across petascale systems. In this work, we address the hitherto open challenge of systematically mapping computations onto Petascale accelerator systems, comprising many (thousands of) processing cores spanning many chips, while maximizing overall system performance. We achieve this by characterizing the design space of possible mapping configurations, building a detailed performance model that incorporates every computation and datatransfer involved in DNN training, and using a design space exploration tool called DEEPSPATIALMATRIX to identify the performance optimal configuration. We highlight 4 key optimizations built within DEEPSPATIALMATRIX  hybrid datamodel parallelism, interlayer memory reuse, timestep pipelining, and dynamic spatial minibatching  each of which improve system utilization by carefully managing the available memory capacity and interconnect bandwidth to balance the compute vs. communication costs. On a 8petaFLOP accelerator system, we demonstrate 1.36xc3x9732xc3x97 improvement in training performance through our design space exploration and optimizations across image recognition (VGG16, ResNet50) and machine translation (GNMT) DNN models.