A LowPower HighThroughput InMemory CMOSReRAM Accelerator for LargeScale Deep Residual Neural Networks We present an inmemory accelerator circuit design for ResNet50, a largescale residual neural network with 49 convolutional layers, 2.6 xc3x97 107 parameters and 4.1 xc3x97 109 f1oatingpoint operations (FLOPS). A 4bit quantized ResNet50 is first chosen among various bitwidths for the best tradeoff. It is then trained and fully mapped onto a 4608 xc3x97 512 ReRAM crossbar, yielding a storage reduction from 195.2Mb to 24.3Mb and an 88.1% top5 accuracy on ImageNet, only 2.5% lower than the fullprecision original. Two versatile CMOS 4bit DAC and ADC are designed for input and readout, allowing the proposed CMOSReRAM accelerator to achieve up to 15.2xc3x97 runtime speedup and 498xc3x97 higher energy efficiency versus the stateoftheart CMOSASIC implementation.

Memory and Interconnect Optimizations for PetaScale Deep Learning Systems Hardware accelerators are a promising solution to the stringent computational requirements of Deep Neural Networks (DNNs). Ranging from lowpower IP cores to server class systems, various accelerator architectures with high TOPSW peak processing efficiencies and flexibility to execute different DNN topologies have been proposed. Prior efforts improve core utilization through better dataflows and computation sequencing, but little effort has thus far been devoted to systematically programming DNN accelerators to extract best possible system utilization, particularly for DNN training, which can be parallelized across petascale systems. In this work, we address the hitherto open challenge of systematically mapping computations onto Petascale accelerator systems, comprising many (thousands of) processing cores spanning many chips, while maximizing overall system performance. We achieve this by characterizing the design space of possible mapping configurations, building a detailed performance model that incorporates every computation and datatransfer involved in DNN training, and using a design space exploration tool called DEEPSPATIALMATRIX to identify the performance optimal configuration. We highlight 4 key optimizations built within DEEPSPATIALMATRIX  hybrid datamodel parallelism, interlayer memory reuse, timestep pipelining, and dynamic spatial minibatching  each of which improve system utilization by carefully managing the available memory capacity and interconnect bandwidth to balance the compute vs. communication costs. On a 8petaFLOP accelerator system, we demonstrate 1.36xc3x9732xc3x97 improvement in training performance through our design space exploration and optimizations across image recognition (VGG16, ResNet50) and machine translation (GNMT) DNN models.