  Recent breakthroughs in deep learning (DL) have led to the emergence of many
intelligent mobile applications and services, but in the meanwhile also pose
unprecedented computing challenges on resource-constrained mobile devices. This
paper builds a collaborative deep inference system between a
resource-constrained mobile device and a powerful edge server, aiming at
joining the power of both on-device processing and computation offloading. The
basic idea of this system is to partition a deep neural network (DNN) into a
front-end part running on the mobile device and a back-end part running on the
edge server, with the key challenge being how to locate the optimal partition
point to minimize the end-to-end inference delay. Unlike existing efforts on
DNN partitioning that rely heavily on a dedicated offline profiling stage to
search for the optimal partition point, our system has a built-in online
learning module, called Autodidactic Neurosurgeon (ANS), to automatically learn
the optimal partition point on-the-fly. Therefore, ANS is able to closely
follow the changes of the system environment by generating new knowledge for
adaptive decision making. The core of ANS is a novel contextual bandit learning
algorithm, called $\mu$LinUCB, which not only has provable theoretical learning
performance guarantee but also is ultra-lightweight for easy real-world
implementation. We implement our system on a video stream object detection
testbed to validate the design of ANS and evaluate its performance. The
experiments show that ANS significantly outperforms state-of-the-art benchmarks
in terms of tracking system changes and reducing the end-to-end inference
delay.
