Mobile applications are benefiting significantly from the advancement in deep learning, e.g. providing new features . Given a trained deep learning model, applications usually need to perform a series of matrix operations based on the input data, in order to infer possible output values. Because of model computation complexity and increased model sizes, those trained models are usually hosted in the cloud. When mobile apps need to utilize those models, they will have to send input data over the network. While cloud-based deep learning can provide reasonable response time for mobile apps, it also restricts the use case scenarios, e.g. mobile apps need to have access to network . With mobile specific deep learning optimizations, it is now possible to employ device-based inference. However, because mobile hardware, e.g. GPU and memory size, can be very different and limited when compared to  desktop counterpart, it is important to understand the feasibility of this new device-based deep learning inference architecture. In this paper, we empirically evaluate the inference efficiency of three Convolutional Neural Networks  using a benchmark Android application we developed. Based on our application-driven analysis , we have identified several performance bottlenecks for mobile applications powered by on-device deep learning inference .
Modern mobile applications are benefiting significantly from the advancement in deep learning, e.g. , implementing real-time image recognition and conversational system . Given a trained deep learning model, applications usually need to perform a series of matrix operations based on the input data, in order to infer possible output values. Because of computational complexity and size constraints, these trained models are often hosted in the cloud. To utilize these cloud-based models, mobile apps will have to send input data over the network. While cloud-based deep learning can provide reasonable response time for mobile apps, it restricts the use case scenarios, e.g. mobile apps need to have network access . With mobile specific deep learning optimizations, it is now possible to employ on-device inference. However, because mobile hardware, such as GPU and memory size, can be very limited when compared to its desktop counterpart, it is important to understand the feasibility of this new on-device deep learning inference architecture. In this paper, we empirically evaluate the inference performance of three Convolutional Neural Networks (CNNs) using a benchmark Android application we developed. Our measurement and analysis suggest that on-device inference can cost up to two orders of magnitude greater response time and energy when compared to cloud-based inference, and that loading model and computing probability are two performance bottlenecks for on-device deep inferences .
R
Mobile
Modern mobile
R
providing new features
, implementing real-time image recognition and conversational system
R
model computation complexity and increased model sizes, those
computational complexity and size constraints, these
R
usually
often
R
When mobile apps need to utilize those models, they
To utilize these cloud-based models, mobile apps
D
also
R
access to network
network access
R
device-based
on-device
R
e.g.
such as
D
different and
A
its
R
device-based
on-device
R
efficiency
performance
A
(CNNs)
R
Based on our application-driven analysis , we have identified several
Our measurement and analysis suggest that on-device inference can cost up to two orders of magnitude greater response time and energy when compared to cloud-based inference, and that loading model and computing probability are two
D
mobile applications powered by
R
learning inference
inferences