Accelerating Deep Learning Inference via Freezing Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on realworld tasks. However, this increase in accuracy comes at the cost of computationally expensive models leading to higher prediction latencies. Prior efforts to reduce this latency such as quantization, model distillation, and anytime prediction models typically tradeoff accuracy for performance. In this work, we observe that caching intermediate layer outputs can help us avoid running all the layers of a DNN for a sizeable fraction of inference requests. We find that this can potentially reduce the number of effective layers by half for 91.58% of CIFAR10 requests run on ResNet18. We present Freeze Inference, a system that introduces approximate caching at each intermediate layer and we discuss techniques to reduce the cache size and improve the cache hit rate. Finally, we discuss some of the open research challenges in realizing such a design.

Optimizing Deep Learning Inference on Embedded Systems Through Adaptive Model Selection Deep neural networks ( DNNs ) are becoming a key enabling technology for many application domains. However, ondevice inference on batterypowered, resourceconstrained embedding systems is often infeasible due to prohibitively long inferencing time and resource requirements of many DNNs. Offloading computation into the cloud is often unacceptable due to privacy concerns, high latency, or the lack of connectivity. While compression algorithms often succeed in reducing inferencing times, they come at the cost of reduced accuracy. This paper presents a new, alternative approach to enable efficient execution of DNNs on embedded devices. Our approach dynamically determines which DNN to use for a given input, by considering the desired accuracy and inference time. It employs machine learning to develop a lowcost predictive model to quickly select a pretrained DNN to use for a given input and the optimization constraint. We achieve this by first offline training a predictive model, and then using the learned model to select a DNN model to use for new, unseen inputs. We apply our approach to two representative DNN domains: image classification and machine translation. We evaluate our approach on a Jetson TX2 embedded deep learning platform and consider a range of influential DNN models including convolutional and recurrent neural networks. For image classification, we achieve a 1.8x reduction in inference time with a 7.52% improvement in accuracy, over the mostcapable single DNN model. For machine translation, we achieve a 1.34x reduction in inference time over the mostcapable single model, with little impact on the quality of translation.