The development of smart vehicles brings drivers and passengers a comfortable and safe environment. Various emerging applications are promising to enrich users' traveling experiences and daily life. However, how to execute computing-intensive applications on resource-constrained vehicles still faces huge challenges. In this article, we construct an intelligent offloading system for vehicular edge computing by leveraging deep reinforcement learning. First, both the communication and computation states are modelled by finite Markov chains. Moreover, the task scheduling and resource allocation strategy is formulated as a joint optimization problem to maximize users' Quality of Experience (QoE). Due to its complexity, the original problem is further divided into two sub-optimization problems. A two-sided matching scheme and a deep reinforcement learning approach are developed to schedule offloading requests and allocate network resources, respectively. Performance evaluations illustrate the effectiveness and superiority of our constructed system. With the rapid development of ubiquitous systems and smart vehicles, artificial intelligence-based vehicular networks [37] , known as a subset of Cyber Physical Systems (CPS) [22] , have drawn increased attention. Many researchers all over the world have been working on novel automotive applications to create a more comfortable and safer driving environment. However, how to execute these computing-intensive applications on vehicles still faces huge challenges, e.g., how to enable real-time feedbacks between vehicles and network servers based on Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication modes; how to provide efficient computing capabilities for resource-constraint applications and reasonable resource allocations for vehicles and infrastructures. Initially, researchers propose the paradigm of Mobile Cloud Computing (MCC) [40] , accumulating rich computing and storage resources into cloud servers. Despite its powerful computing capability, it is still difficult to satisfy the real-time response requirements of vehicular applications. Therefore, Mobile Edge Computing (MEC) is a promising alternative, the nodes of which are in proximity of users. Compared with MCC, MEC can greatly reduce communication latency, which scales exponentially with the increase of routing hops in vehicular networks [8] . Moreover, the diversity of MEC nodes significantly exploits potential computing resources over the network, which also alleviates the workload of the central Base Station (BS) [4] . It should be noted that more than RoadSide Units (RSUs)-any entities with the capability of computing, caching, or networking-can be the platform of MEC. Since the constraint resource limits the capability of MEC nodes, it is envisioned that the performance of traditional time-and energy-consuming networking methods [3, 5] can drop abruptly in vehicular networks. Therefore, it is urgent to develop an efficient solution for the MEC environment. Deep Reinforcement Learning (DRL) is a prospective technology to replace traditional methods. Recently, machine learning has achieved remarkable achievements in many fields, such as image processing, pattern recognition, and natural language processing. It is also involved in computingintensive applications, including autopilot and real-time navigation through V2V or V2I. However, machine learning in MEC-enabled vehicular networks (i.e., vehicular edge computing) is still in its infancy. A few researches attempt to leverage deep learning and convolutional neural networks to predict traffic flows. However, DRL is rarely considered. To construct an intelligent offloading system for vehicular edge computing and make it work well, there are three main challenges: (1) Although DRL has achieved great success in Atari games and Go [17] , its application in vehicular networks is almost nonexistent. This is because vehicular networks are highly dynamic, and the constraint in offloading systems is more implicit, flexible, and diverse than the explicit rules of chess. (2) Both DRL and traditional vehicular networks are investigated based on a sequence of captured images. However, no sequential images exist in our intelligent offloading system. How to migrate DRL to vehicular networks without images is rather challenging. Recently, deep learning and edge computing-based CPS have attracted the attention of many researchers. We review three categories, including deep learning-based CPS, MEC-based CPS, and ACM 60:4 Z. Ning et al. deep learning for MEC. Since CPS contains various network systems, we choose vehicular networks as an example to introduce recent research progress. Deep learning is generally utilized for pattern recognition and traffic prediction. The authors in Reference [18] utilize unsupervised learning to cluster the urban smart card data, involving the station-oriented and passenger-oriented views. The real-world data of the metropolitan area in Rennes, France, are collected to find the distribution of traffic flow and the similarity among passengers. A cooperative neural network is tailored to construct a structural lane detection system with the captured images in a real-world traffic scene [13] . First, a deep convolutional network is devised to detect traffic signs and their geometric attributes. Second, recurrent neural networks dealing with signal spatial distribution are difficult to be explicitly recognized. After millions of traffic images are captured, how to efficiently retrieve these large amounts of data is a challenging issue. A supervised hash coding scheme is designed to generate high-quality binary codes [35] . Convolutional neural networks are implemented to analyze the feature representation of images. The quantized loss function compels that similar images are encoded by similar codes. Authors in Reference [1] adopt Long Short Term Memory (LSTM)-based recurrent neural networks to predict the real-time taxi demand. Different from traditional prediction, they do not forecast a deterministic value, but leverage mixture density networks to predict the probability distribution of taxi demands. A new neural network training method is applied in Reference [16] . It takes the correlations between space and time into consideration to predict traffic flow. The key idea of the training method is using the greedy layerwise unsupervised learning scheme to preprocess the deep neural network layer-by-layer, which reduces training time significantly.
Deep neural network is the most successful representation learning technology and has made remarkable achievements in many scenes REF . And deep neural networks are more efficient to solve the problems in anomaly detection.
Deep Reinforcement Learning for Vehicular Edge Computing: An Intelligent Offloading System
TIST
Computer Science