MoNA: Mobile Neural Architecture with Reconfigurable Parallel Dimensions Emerging mobile deep neural networks (DNNs) are prevalent in resourcelimited devices because of their shrunken parameter size and reduced computation. However, when deploying mobile DNNs on conventional DNN accelerators, there will be a performance loss due to the reduction of processing element (PE) utilization. Conventional accelerators are generally designed base on a fixed pattern of data reuse, but the data reuse opportunities of different layers in mobile DNNs are diverse. To process mobile DNNs with high performance, we propose an architecture called MoNA (Mobile Neural Architecture), which includes a flexible dataflow and a reconfigurable computing core. The dataflow efficiently supports reconfigurable processing parallelism to maximize the PE utilization of mobile DNNs. The computing core is a 3D PE array, which supports the dataflow and avoids high bandwidth requirement.

Memory and Interconnect Optimizations for PetaScale Deep Learning Systems Hardware accelerators are a promising solution to the stringent computational requirements of Deep Neural Networks (DNNs). Ranging from lowpower IP cores to server class systems, various accelerator architectures with high TOPSW peak processing efficiencies and flexibility to execute different DNN topologies have been proposed. Prior efforts improve core utilization through better dataflows and computation sequencing, but little effort has thus far been devoted to systematically programming DNN accelerators to extract best possible system utilization, particularly for DNN training, which can be parallelized across petascale systems. In this work, we address the hitherto open challenge of systematically mapping computations onto Petascale accelerator systems, comprising many (thousands of) processing cores spanning many chips, while maximizing overall system performance. We achieve this by characterizing the design space of possible mapping configurations, building a detailed performance model that incorporates every computation and datatransfer involved in DNN training, and using a design space exploration tool called DEEPSPATIALMATRIX to identify the performance optimal configuration. We highlight 4 key optimizations built within DEEPSPATIALMATRIX  hybrid datamodel parallelism, interlayer memory reuse, timestep pipelining, and dynamic spatial minibatching  each of which improve system utilization by carefully managing the available memory capacity and interconnect bandwidth to balance the compute vs. communication costs. On a 8petaFLOP accelerator system, we demonstrate 1.36xc3x9732xc3x97 improvement in training performance through our design space exploration and optimizations across image recognition (VGG16, ResNet50) and machine translation (GNMT) DNN models.