Model-Based Stabilisation of Deep Reinforcement Learning

Though successful in high-dimensional domains, deep reinforcement learning exhibits high sample complexity and suffers from stability issues as reported by researchers and practitioners in the field. These problems hinder the application of such algorithms in real-world and safety-critical scenarios. In this paper, we take steps towards stable and efficient reinforcement learning by following a model-based approach that is known to reduce agent-environment interactions. Namely, our method augments deep Q-networks (DQNs) with model predictions for transitions, rewards, and termination flags. Having the model at hand, we then conduct a rigorous theoretical study of our algorithm and show, for the first time, convergence to a stationary point. En route, we provide a counterexample showing that ’vanilla’ DQNs can diverge confirming practitioners’ and researchers’ experiences. Our proof is novel in its own right and can be extended to other forms of deep reinforcement learning. In particular, we believe exploiting the relation between reinforcement (with deep function approximators) and online learning can serve as a recipe for future proofs in the domain. Finally, we validate our theoretical results in 20 games from the Atari benchmark. Our results show that following the proposed model-based learning approach not only ensures convergence but leads to a reduction in sample complexity and superior performance. Introduction Model-free deep reinforcement learning methods have recently demonstrated impressive performance on a range of complex learning tasks (Hessel et al. 2018; Lillicrap et al. 2017; Jaderberg et al. 2017). Deep Q-Networks (DQNs) (Mnih et al. 2015), in particular, stand out as a versatile and effective tool for a wide range of applications. DQNs offer an end-to-end solution for many reinforcement learning problems by generalizing tabular Q-learning to highdimensional input spaces. Unfortunately, DQNs still suffer from a number of important drawbacks that hinder wide-spread adoption. There is a general lack of theoretical understanding to guide the use of deep Q-learning algorithms. Combining non-linear function approximation with off-policy reinforcement learning is known to lead to possible divergence issues, even in very simple tasks (Boyan and Preliminary work. Moore 1995). Despite improvements introduced by deep Qnetworks (Mnih et al. 2015), Q-learning approaches based on deep neural networks still exhibit occasional instability during the learning process, see for example (van Hasselt, Guez, and Silver 2016). In this paper, we aim to address the instability of current deep RL approaches by augmenting model-free value learning with a predictive model. We propose a hybrid model and value learning objective, which stabilises the learning process and reduces sample complexity when compared to model-free algorithms. We attribute this reduction to the richer feedback signal provided by our method compared to standard deep Q-networks. In particular, our model provides feedback at every time step based on the current prediction error, which in turn eliminates one source of sample inefficiency caused by sparse rewards. These conclusions are also in accordance with previous research conducted on linear function approximation in reinforcement learning (Parr et al. 2008; Sutton et al. 2012; Song et al. 2016). While linear function approximation results provide a motivation for model-based stabilisation, such theories fail to generalise to deep non-linear architectures (Song et al. 2016). To close this gap in the literature and demonstrate stability of our method, we prove convergence in the general deep RL setting with deep value function approximators. Theoretically analysing deep RL algorithms, however, is challenging due to nonlinear functional dependencies and objective non-convexities prohibiting convergence to globally optimal policies. As such, the best one would hope for is a stationary point (e.g., a first-order one) that guarantees vanishing gradients. Even understanding gradient behavior for deep networks in RL can be difficult due to explorationbased policies and replay memory considerations. To alleviate some of these problems, we map deep reinforcement learning to a mathematical framework explicitly designed for understanding the exploration/exploitation trade-off. Namely, we formalise the problem as an online learning game the agent plays against an adversary (i.e., the environment). Such a link allows us to study a more general problem combining notions of regret, reinforcement learning, and optimisation. Given such a link, we prove that for any > 0, regret vanishes as O ( T 1−2 lnT ) with T being the total number of rounds. This, in turn, guarantees convergence to a stationary point. ar X iv :1 80 9. 01 90 6v 1 [ cs .L G ] 6 S ep 2 01 8 Guided by the above theoretical results, we validate that our algorithm leads to faster learning by conducting experiments on 20 Atari games. Due to the high computational and budgeting demands, we chose a benchmark that is closest to our setting. Concretely, we picked DQNs as our template algorithm to improve on both theoretically and empirically. It is worth noting that extending our theoretical results to more complex RL algorithms is an orthogonal future direction. Background: Reinforcement Learning We consider a discrete time, infinite horizon, discounted Markov Decision Process (MDP) 〈 S,A, P , P , γ 〉 . Here S denotes the state space, A the set of possible actions, P (R) : S × A × R → [0, 1] is the reward function, P (S) : S × A × S → [0, 1] is the state transition function, and γ ∈ (0, 1) is a discount factor. An agent, being in state St ∈ S at time step t, executes an action at ∈ A sampled from its policy π(at|St), a conditional probability distribution where π : S × A → [0, 1]. The agent’s action elicits from the environment a reward signal rt ∈ R indicating instantaneous reward, a terminal flag ft ∈ {0, 1} indicating a terminal event that restarts the environment, and a transition to a successor state St+1 ∈ S. We assume that the sets S, A, and R are discrete. The reward rt is sampled from the conditional probability distribution P (rt|St,at). Similarly, ft ∼ P (F (ft|St,at) with P (F ) : S × A × {0, 1} → [0, 1], where a terminal event (ft = 1) restarts the environment according to some initial state distribution S0 ∼ P (S) 0 (S0). The state transition to a successor state is determined by a stochastic state transition function according to St+1 ∼ P (St+1|St,at). The agent’s goal is to maximise future cumulative reward E P (S) 0 ,π,P (R),P (F ),P (S) [ ∞ ∑ t=0 ( t ∏ t′=0 (1− ft′−1) ) γrt ] with respect to the policy π. An important quantity in RL are Q-values Q(St,at), which are defined as the expected future cumulative reward value when executing action at in state St and subsequently following policy π. Qvalues enable us to conveniently phrase the RL problem as maxπ ∑ S p (S) ∑ a π(a|S)Q(S,a), where p(S) is the stationary state distribution obtained when executing π in the environment (starting from S0 ∼ P (S) 0 (S0)). Deep Q-Networks: Value-based reinforcement learning approaches identify optimal Q-values directly using parametric function approximators Qθ(S,a), where θ ∈ Θ represents the parameters (Watkins 1989; Busoniu et al. 2010). Optimal Q-value estimates then correspond to an optimal policy π(a|S) = δa,argmaxa′Qθ(S,a′). Deep Q-networks (Mnih et al. 2015) learn a deep neural network based Qvalue approximation by performing stochastic gradient descent on the following training objective: L(θ) = ES,a,r,f,S′ [( r + (1− f)γmax a′ Qθ−(S ′,a′) −Qθ(S,a) )2] . (1) The expectation ranges over transitions S,a, r, f,S′ sampled from an experience replay memory (S′ denotes the state at the next time step). Use of this replay memory, together with the use of a separate target network Qθ− (with different parameters θ− ∈ Θ) for calculating the bootstrap values maxa′ Qθ−(S ′,a′), helps stabilise the learning process. Background: Online Learning In this paper, we employ a special form of regret minimisation games that we briefly review here. A regret minimisation game is a triple 〈Θ,F , T 〉, where Θ is a non-empty decision set, F is the set of moves of the adversary which contains bounded convex functions from R to R and T is the total number of rounds. The game commences in rounds, where at round j = 1, . . . , T , the agent chooses a prediction θj ∈ Θ and the adversary a loss function Lj ∈ F . At the end of the round, the adversary reveals its choice and the agent suffers a loss Lj(θj). In this paper, we are concerned with the full-information case where the agent can observe the complete loss function Lj at the end of each round. The goal of the game is for the agent to make successive predictions to minimise cumulative regret defined as: