Recent advances in deep learning motivate the use of deep neutral networks in sensing applications, but their excessive resource needs on constrained embedded devices remain an important impediment. A recently explored solution space lies in compressing (approximating or simplifying) deep neural networks in some manner before use on the device. We propose a new compression solution, called DeepIoT, that makes two key contributions in that space. First, unlike current solutions geared for compressing speci c types of neural networks, DeepIoT presents a uni ed approach that compresses all commonly used deep learning structures for sensing applications, including fully-connected, convolutional, and recurrent neural networks, as well as their combinations. Second, unlike solutions that either sparsify weight matrices or assume linear structure within weight matrices, DeepIoT compresses neural network structures into smaller dense matrices by nding the minimum number of non-redundant hidden elements, such as lters and dimensions required by each layer, while keeping the performance of sensing applications the same. Importantly, it does so using an approach that obtains a global view of parameter redundancies, which is shown to produce superior compression. e compressed model generated by DeepIoT can directly use existing deep learning libraries that run on embedded and mobile systems without further modi cations. We conduct experiments with ve di erent sensing-related tasks on Intel Edison devices. DeepIoT outperforms all compared baseline algorithms with respect to execution time and energy consumption by a signi cant margin. It reduces the size of deep neural networks by 90% to 98.9%. It is thus able to shorten execution time by 71.4% to 94.5%, and decrease energy consumption by 72.2% to 95.7%. ese improvements are achieved without loss of accuracy. e results underscore the potential of DeepIoT for advancing the exploitation of deep neural networks on resource-constrained embedded devices. A key direction in embedded sensing literaure is to enable running progressively more interesting applications under the more pronounced resource constraints of embedded and mobile devices. Brouwers et al. reduced the energy consumption of Wi-Fi based localization with an incremental scanning strategy [5]. Hester et al. proposed an ultra-low-power hardware architecture and a companion so ware framework for energy-e cient sensing system [25]. Ferrari et al. and Schuss et al. focused on low-power wireless communication protocols [13, 41]. Wang et al. enabled energy ecient reliable broadcast by considering poorly correlated links [49]. Saifullah et al. designed a scalable and energy-e cient wireless sensor network (WSN) over white spaces [40]. Alsheikh et al. discussed about data compression in WSN with machine learning techniques [3]. Recent studies focused on compressing deep neural networks for embedded and mobile devices. Han et al. proposed a magnitudebased compression method with ne-tuning, which illustrated
Yao et al. propose a reinforcement learning based adaptive dropout solution to explore the less-redundant network structure for mobile and embedded devices REF .
DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
SenSys '17
Mathematics Computer Science