Abstract-With the rapid increase in demand for mobile data, mobile network operators are trying to expand wireless network capacity by deploying wireless local area network (LAN) hotspots on to which they can offload their mobile traffic. However, these network-centric methods usually do not fulfill the interests of mobile users (MUs). Taking into consideration many issues such as different applications' deadlines, monetary cost and energy consumption, how the MU decides whether to offload their traffic to a complementary wireless LAN is an important issue. Previous studies assume the MU's mobility pattern is known in advance, which is not always true. In this paper, we study the MU's policy to minimize his monetary cost and energy consumption without known MU mobility pattern. We propose to use a kind of reinforcement learning technique called deep Q-network (DQN) for MU to learn the optimal offloading policy from past experiences. In the proposed DQN based offloading algorithm, MU's mobility pattern is no longer needed. Furthermore, MU's state of remaining data is directly fed into the convolution neural network in DQN without discretization. Therefore, not only does the discretization error present in previous work disappear, but also it makes the proposed algorithm has the ability to generalize the past experiences, which is especially effective when the number of states is large. Extensive simulations are conducted to validate our proposed offloading algorithms. Index Terms-wireless LAN, multiple-flow, mobile data offloading, reinforcement learning, deep Q-network, DQN ! I T mobile data traffic demand is growing rapidly. According to the investigation of Cisco Systems [1], the mobile data traffic is expected to reach 24.3 exabytes per month by 2019, while it was only 2.5 exabytes per month at the end of 2014. On the other hand, the growth rate of the mobile network capacity is far from satisfying that kind of the demand, which has become a major problem for wireless mobile network operators (MNOs). Even though 5G technology is promising for providing huge wireless network capacity [2], the development process is long and the cost is high. Economic methods such as time-dependent pricing [3] [4] have been proposed to change users' usage pattern, which are not user-friendly. Up to now, the best practice for increasing mobile network capacity is to deploy complementary networks (such as wireless LAN and femtocells), which can be quickly deployed and is cost-efficient. Using such methods, part of the MUs' traffic demand can be offloaded from a MNO's cellular network to the complementary network. The process that a mobile device automatically changes its connection type (such as from cellular network to wireless LAN) is called vertical handover [5] . Mobile data offloading is facilitated by new standards such as Hotspot 2.0 [6] and the 3GPP Access Network Discovery and Selection Function (ANDSF) standard [7] , with which information of network (such as price and network load) can be broadcasted to MUs in real-time. Then MUs can make offloading decisions intelligently based on the real-time network information. There are many works related to the wireless LAN offloading problem. However, previous works either considered the wireless LAN offloading problem from the network providers' perspective without considering the MU's quality of service (QoS) [14] [15] studied wireless LAN offloading problem from MU's perspective. While single-flow mobile date offloading was considered in [14] , multi-flow mobile data offloading problem is studied in [15] in which a MU has multiple applications to transmit data simultaneously with different deadlines. MU's target was to minimize its total cost, while taking monetary cost, preference for energy consumption, application's delay tolerance into consideration. This was formulated the wireless LAN offloading problem as a finite-horizon discrete-time Markov decision process [16] [17] [18] . A high time complexity dynamic programming (DP) based optimal offloading algorithm and a low time complexity heuristic offloading algorithm were prosed in [15] . One assumption in [15] was that MU's mobile pattern from one place to another is known in advance, then the transition probability, which is necessary for optimal policy calculation in MDP, can be obtained in advance. However, the MU's mobility pattern may not be easily obtained or the accuracy is not high. Even though a Q-learning [19] based algorithm was proposed in [14] for the unknown MU's mobility pattern case, the learning, the convergence rate of the proposed algorithm is rather low due to the large number of states. It takes time for the reinforcement learning agent to experience all the states to estimate the Q-value. In this paper, we propose a deep reinforcement learning algorithm, specifically, a deep Q-network (DQN) [20] based algorithm, to solve the multi-flow offloading problem with high convergence rate without knowing MU's mobility pattern. In
Reference REF study the mobile user's policy to minimize the user's monetary cost and energy consumption without a known mobility pattern and use a deep Q-network (DQN) for the user to learn the optimal offloading policy from past experiences.
A Deep Reinforcement Learning Based Approach for Cost- and Energy-Aware Multi-Flow Mobile Data Offloading
IEICE Transactions on Communications, Vol.E101-B, No.7, Jul. 2018
Computer Science