Memory and CommunicationAware Model Compression for Distributed Deep Learning Inference on IoT. Model compression has emerged as an important area of research for deploying deep learning models on InternetofThings (IoT). However, for extremely memoryconstrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communicationaware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained u0027teacheru0027 deep network into several disjoint and highlycompressed u0027studentu0027 modules, without loss of accuracy. Moreover, we propose a network sciencebased knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for userdefined memoryperformance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24x), performance (up to 12x), and energy per node (up to 14x) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33x reduction in total latency w.r.t. a stateoftheart model compression baseline.

Optimizing Deep Learning Inference on Embedded Systems Through Adaptive Model Selection Deep neural networks ( DNNs ) are becoming a key enabling technology for many application domains. However, ondevice inference on batterypowered, resourceconstrained embedding systems is often infeasible due to prohibitively long inferencing time and resource requirements of many DNNs. Offloading computation into the cloud is often unacceptable due to privacy concerns, high latency, or the lack of connectivity. While compression algorithms often succeed in reducing inferencing times, they come at the cost of reduced accuracy. This paper presents a new, alternative approach to enable efficient execution of DNNs on embedded devices. Our approach dynamically determines which DNN to use for a given input, by considering the desired accuracy and inference time. It employs machine learning to develop a lowcost predictive model to quickly select a pretrained DNN to use for a given input and the optimization constraint. We achieve this by first offline training a predictive model, and then using the learned model to select a DNN model to use for new, unseen inputs. We apply our approach to two representative DNN domains: image classification and machine translation. We evaluate our approach on a Jetson TX2 embedded deep learning platform and consider a range of influential DNN models including convolutional and recurrent neural networks. For image classification, we achieve a 1.8x reduction in inference time with a 7.52% improvement in accuracy, over the mostcapable single DNN model. For machine translation, we achieve a 1.34x reduction in inference time over the mostcapable single model, with little impact on the quality of translation.