In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace(λ), with three desired properties: (1) low variance; (2) safety, as it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) efficiency, as it makes the best use of samples collected from near on-policy behaviour policies. We analyse the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. To our knowledge, this is the first return-based off-policy control algorithm converging a.s. to Q * without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q(λ), which was still an open problem. We illustrate the benefits of Retrace(λ) on a standard suite of Atari 2600 games. One fundamental trade-off in reinforcement learning lies in the definition of the update target: should one estimate Monte Carlo returns or bootstrap from an existing Q-function? Return-based methods (where return refers to the sum of discounted rewards t γ t r t ) offer some advantages over value bootstrap methods: they are better behaved when combined with function approximation, and quickly propagate the fruits of exploration (Sutton, 1996) . On the other hand, value bootstrap methods are more readily applied to off-policy data, a common use case. In this paper we show that learning from returns need not be at cross-purposes with off-policy learning. We start from the recent work of Harutyunyan et al. (2016) , who show that naive off-policy policy evaluation, without correcting for the "off-policyness" of a trajectory, still converges to the desired Q π value function provided the behavior µ and target π policies are not too far apart (the maximum allowed distance depends on the λ parameter). Their Q π (λ) algorithm learns from trajectories generated by µ simply by summing discounted off-policy corrected rewards at each time step. Unfortunately, the assumption that µ and π are close is restrictive, as well as difficult to uphold in the control case, where the target policy is always greedy with respect to the current Q-function. In that sense this algorithm is not safe: it does not handle the case of arbitrary "off-policyness". Alternatively, the Tree-backup (TB) (λ) algorithm (Precup et al., 2000) tolerates arbitrary target/behavior discrepancies by scaling information (here called traces) from future temporal differences by the product of target policy probabilities. TB(λ) is not efficient in the "near on-policy" case (similar µ and π), though, as traces may be cut prematurely, blocking learning from full returns. In this work, we express several off-policy, return-based algorithms in a common form. From this we derive an improved algorithm, Retrace(λ), which is both safe and efficient, enjoying convergence guarantees for off-policy policy evaluation and -more importantly -for the control setting. Retrace(λ) can learn from full returns retrieved from past policy data, as in the context of experience replay (Lin, 1993) , which has returned to favour with advances in deep reinforcement learning (Mnih
Based on the work in REF , several off-policy return-based methods based on target policy probability: importance sampling (IS), tree-backup and Retrace(λ) can be expressed in a unified operator R as follows:
Safe and Efficient Off-Policy Reinforcement Learning
ArXiv
ArXiv
Computer Science Mathematics