The Nature of Temporal Difference Errors in Multi-step Distributional Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the multi-step off-policy learning approach to distributional RL.
Despitethe apparent similarity between value-based RL and distributional RL, our studyreveals intriguing and fundamental differences between the two cases in the multistep setting.
We identify a novel notion of path-dependent distributional TDerror, which is indispensable for principled multi-step distributional RL.
Thedistinction from the value-based case bears important implications on concepts suchas backward-view algorithms.
Our work provides the ﬁrst theoretical guarantees onmulti-step off-policy distributional RL algorithms, including results that apply tothe small number of existing approaches to multi-step distributional RL.
In addition,we derive a novel algorithm, Quantile Regression-Retrace, which leads to a deep
RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN onthe Atari-57 benchmark.
Collectively, we shed light on how unique challenges inmulti-step distributional RL can be addressed both in theory and practice.
Introduction
The return (cid:80) ∞ t =0 γ t R t is a fundamental concept in reinforcement learning (RL).
In general, the returnis a random variable, whose distribution captures important information such as the stochasticity infuture events.
While the classic view of value-based RL typically focuses on the expected return[1–3] learning the full return distribution is of both theoretical and practical importance [4–10].
To design efﬁcient algorithms for learning return distributions a natural idea is to construct distributional equivalents of existing multi-step off-policy value-based algorithms.
In value-based RL,multi-step learning tends to propagate useful information more efﬁciently and off-policy learning isubiquitous in modern RL systems.
Meanwhile, the return distribution shares inherent commonalitieswith the expected return, thanks to the close connection between the distributional Bellman equation[4–6, 10] and the celebrated value-based Bellman equation [2].
The Bellman equation is foundationalto value-based RL algorithms, including many multi-step off-policy methods [11–14].
Due to theapparent similarity between distributional and value-based Bellman equations, should we expect keyvalue-based concepts and algorithms to seamlessly transfer to distributional learning?
Our study indicates that the answer is no.
There are critical differences between distributional andvalue-based RL, which requires a distinct treatment of multi-step learning.
Indeed, thanks to thefocus on expected returns, the value-based setup offers many unique conceptual and computationalsimpliﬁcations in algorithmic design.
However, we ﬁnd that such simpliﬁcations do not hold fordistributional learning.
Multi-step distributional RL requires a deeper look at the connections
betweenfundamental concepts such as n -step returns, TD errors and importance weights for off-policylearning.
To this end, we make the following conceptual, theoretical and algorithmic contributions:
Distributional TD error.
We demonstrate the emergence of a novel notion of path-dependent distributional TD error (Section 4).
Intriguingly, as the name suggests, path-dependent distributional
TD errors are path-dependent , i.e., distributional TD errors at time t depend on the sequence of immediate rewards ( R s ) t − 1 s =0 .
This differs from value-based TD errors, which are path-independent.
We will show that the path-dependency property is not an artifact, but rather a fundamental propertyof distributional learning.
We show numerically that naively constructing certain path-independentdistributional TD errors does not produce convergent algorithms.
The path-dependency property alsohas conceptual and computational impacts on forward-view estimates and backward-view algorithms.
Theory of multi-step distributional RL.
We derive distributional Retrace, a novel and genericmulti-step off-policy operator for distributional learning.
We prove that distributional Retrace iscontractive and has the target return distribution as its ﬁxed point.
Distributional Retrace interpolatesbetween the one-step distributional Bellman operator [6] and Monte-Carlo (MC) estimation withimportance weighting [15], trading-off the strengths from the two extremes.
Approximate multi-step distributional RL.
Finally, we derive Quantile Regression-Retrace, anovel algorithm combining distributional Retrace with quantile representations of distributions [16](Section 5).
One major technical challenge is to deﬁne the quantile regression (QR) loss against signedmeasures, which are unavoidable in sample-based settings.
We bypass the issue of ill-deﬁned QR lossand derive unbiased stochastic estimates to the QR loss gradient.
This leads up to QR-DQN-Retrace,a deep RL agent with performance improvements over QR-DQN on Atari-57 games.
In Fig 1, we illustrate how the back-up target is computed for multi-step distributional RL.
Insummary, we take our ﬁndings to demonstrate how the set of unique challenges presented by multistep distributional RL can be addressed both theoretically and empirically.
Our study also opens upmany exciting research pathways in this domain, paving the way for future investigations.
Background
Consider a Markov decision process (MDP) represented as the tuple ( X , A , P R , P, γ ) where X isthe state space,
A the action space, P R : X ×
A → P ( R ) the reward kernel (with R a ﬁnite set ofpossible rewards), P : X
× A → P ( X ) the transition kernel and γ ∈ [0 , 1) the discount factor.
Ingeneral, we use P ( A ) denote a distribution over set A .
We assume the reward to take a ﬁnite set ofvalues mainly because it is notationally simpler to present results; it is straightforward to extend ourresults to the general case.
Let π : X → P ( A ) be a ﬁxed policy.
We use ( X t , A t , R t ) ∞ t =0 ∼ π todenote a random trajectory sampled from π , such that A t ∼ π ( ·| X t ) , R t ∼ P R ( ·| X t , A t ) , X t +1 ∼
P ( ·| X t , A t ) .
Deﬁne G π ( x, a ) :
= (cid:80) ∞ t =
γ t R t as the random return, obtained by following π startingfrom ( x, a ) .
The Q-function Q π ( x, a ) :
=
E [ G π ( x, a )] is deﬁned as the expected return under policyπ .
For convenience, we also adopt the vector notation Q ∈ R X×A .
Deﬁne the one-step value-based
Bellman operator T π : R X×A
→ R X×A such that T π Q ( x, a ) :
=
E [ R 0 + γQ ( X 1 , A π 1 ) | X 0 =x, A 0 = a ] where Q ( X t , A πt ) :
= (cid:80) a π ( a | X t ) Q ( X t , a ) .
The Q-function Q π satisﬁes Q π
=
T π Q πand is also the unique ﬁxed point of T π .
2.1 Distributional reinforcement learning
In general, the return G π ( x, a ) is a random variable and we deﬁne its distribution as η π ( x, a ) :
=
Law π ( G π ( x, a )) .
The return distribution satisﬁes the distributional Bellman equation [4–6, 17, 10],
[Equation]
where ( b r,γ ) # : P ( R ) → P ( R ) is the pushforward operation deﬁned through the function b r,γ ( z )
=
r + γz [17].
Here, for convenience, we adopt the notation η π ( X t , A πt ) :
= 76 (cid:80) a π ( a | X t )
η π ( X t , a ) .
Throughout the paper, we focus on the space of distributions with boundedsupport P ∞ ( R ) .
Let η ∈ P ∞
( R ) X×A be any distribution vector, we deﬁne the distributional
Bellman operator T π : P ∞ ( R ) X×A → P ∞ ( R ) X×A as follows [17, 10],
[Equation]
Let η π
be the collection of return distributions under π ; the distributional Bellman equation canthen be rewritten as η π = T
π η π .
The distributional Bellman operator T π is γ -contractive underthe supremum p -Wasserstein distance [16, 10], so that η π is the unique ﬁxed point of T π .
See
Appendix B for details of the distance metrics.
2.2 Multi-step off-policy value-based learning
We provide a brief background on the value-based multi-step off-policy setting as a referencefor the distributional case discussed below.
In off-policy learning, the data is generated under abehavior policy µ , which potentially differs from target policy π .
The aim is to evaluate the target
Q-function Q π .
As a standard assumption, we require supp ( π ( ·| x )) ⊆ supp ( µ ( ·| x )) , ∀ x ∈ X .
Letρ t :=
π ( A t | X t ) /µ
( A t | X t ) be the step-wise importance sampling (IS) ratio at time step t .
Step-wise
IS ratios are critical in correcting for the off-policy discrepancy between π and µ .
Let c t ∈ [0 , ρ t ] be a time-dependent trace coefﬁcient.
We denote c 1: t = c 1 · · · c t and deﬁne c 1:0 = 1by convention.
Consider a generic form of the return-based off-policy operator R π,µ as in [13],
R π,µ Q ( x, a ) :
= Q ( x, a ) + E µ
[Equation]
In the above and below, we omit the notation conditioning on X 0 = x, A
= a for conciseness.
Thegeneral form of R π,µ encompasses many important special cases: when on-policy and c t = λ , itrecovers the Q-function variant of TD( λ ) [2, 12]; when c t = λ min( c, ρ t ) , it recovers a speciﬁc formof Retrace [13]; when c t = ρ t , it recovers the importance sampling (IS) operator.
The back-up targetis computed as a mixture over TD errors δ πt , each calculated from the one-step transition data.
Wealso deﬁne the discounted TD error (cid:101) δ πt =
γ t δ πt , which can be interpreted as the difference betweenn -step returns from two time steps t and t + 1 , as we discuss in Section 4.
As we will detail, theproperty of (cid:101) δ πt marks a signiﬁcant difference from the distributional RL setting.
By design, R π,µ has Q π as the unique ﬁxed point.
Multi-step updates make use of rewards frommultiple time steps, propagating learning signals more efﬁciently.
This is reﬂected by the fact that
R π,µ is β -contractive with β ∈ [0 , γ ] [13] and often contracts to Q π faster than the one-step Bellmanoperator T π .
Our goal is to design distributional equivalents of multi-step off-policy operators, whichcan lead to concrete algorithms with sample-based learning.
Multi-step off-policy distributional reinforcement learning
We now present the core theoretical results relating to multi-step distributional operators.
In general,the aim is to evaluate the target distribution η π with access to off-policy data generated under µ .
Below, we use G t (cid:48) :
t = (cid:80) ts = t (cid:48) γ s − t (cid:48)
R s to denote the partial sum of discounted rewards between twotime steps t (cid:48) ≤ t .
We deﬁne the generic form of multi-step off-policy distributional operator R π,µsuch that for any η ∈ P ∞ ( R ) X×A , its back-up target R π,µ η ( x, a ) is computed as [Equation] 
[Equation]
As an effort to simplify the naming, we call R π,µ the distributional Retrace operator.
Distributional
Retrace only requires c t ∈ [0 , ρ t ] and represents a large family of distributional operators.
Throughout,we will heavily adopt the pushforward notations.
This is mainly because instead of directly working [Equation] with the random variable G π , we ﬁnd it much more convenient to express various important multi-stepoperations with pushfoward notations.
The back-up target R π,µ η ( x, a ) is written as a weighted sum of the path-dependent distributional
TD errors (cid:101) ∆ π 0: t , which we extensively discuss in Section 4.
Though the form of R π,µ seems tobear certain similarities to the value-based operator in Equation (3), the critical differences lie insubtle deﬁnitions of the distributional TD errors (cid:101)
∆ π 0: t and where to place the traces c 1: t for off-policycorrections.
We resume to unpack the insights entailed by the design of the operator in Section 4.
Below, we ﬁrst present theoretical properties of the distributional Retrace operator.
We start with akey property which underlies many ensuing theoretical results.
Given a ﬁxed n -step reward sequencer 0: n − 1 and a ﬁxed state-action pair ( x, a ) ∈
X × A , we call pushfoward distributions of the form 124 (cid:16) b
[Equation]
n (cid:17) # η ( x, a ) the n -step target distributions .
Our result shows that the back-up target of
Retrace is a convex combination of n -step target distributions with varying values of n .
Lemma 3.1.
( Convex combination )
The Retrace back-up target is a convex combination of nstep target distributions.
Formally, there exists an index set
I ( x, a ) such that R π,µ η ( x, a ) = 128 (cid:80) i ∈ I ( x,a ) w i η i where w i ≥ 0 , (cid:80) i ∈
I ( x,a ) w i = 1 and ( η i ) i ∈ I ( x,a ) are n i -return target distributions.
Since R π,µ η ∈ P ∞ ( R ) X×A , we can measure the contraction of R π,µ under probability metrics.
Proposition 3.2.
( Contraction ) R π,µ is β -contractive under supremum p -Wasserstein distance,where β = max x ∈X ,a ∈A (cid:80) ∞ t =1 E µ [ c 1 ...
c t − 1 (1 − c t )] γ t ≤ γ .
The contraction rate of the distributional Retrace operator is determined by its effective horizon.
Atone extreme, when c t = 0 , the effective horizon is 1 and β = γ , in which case Retrace recovers theone-step operator.
At the other extreme, when c t = ρ t , the effective horizon is inﬁnite which givesβ = 0 .
This latter case can be understood as correcting for all the off-policy discrepancies with IS,which is very efﬁcient in expectation but incurs high variance under sample-based approximations.
Proposition 3.2 also implies that the distributional Retrace operator has a unique ﬁxed point.
Proposition 3.3.
( Unique ﬁxed point )
R π,µ has η π as the unique ﬁxed point in P ∞ ( R ) X×A .
The above result suggests that starting with η 0 ∈ P ∞ ( R ) X×A , the recursion η k +
=
R π,µ η kproduces iterates ( η k ) ∞ k =0 ∈ P ∞ ( R ) X×A which converge to η π in W p at a rate of O ( β k ) .
Understanding multi-step distributional reinforcement learning
Now, we pause and take a closer look at the construction of the distributional Retrace operator.
Wepresent a number of insights that distinguish distributional learning from value-based learning.
4.1 Path-dependent TD error
The value-based Retrace back-up target can be written as a mixture of value-based TD errors.
Tobetter parse the distributional Retrace operator and draw comparison to the value-based setting, weseek to rewrite the distributional back-up target R π,µ η ( x, a ) into a weighted sum of some notion ofdistributional TD errors.
To this end, we start with a natural analogy to the value-based TD error.
Deﬁnition 4.1.
( Distributional TD error )
Given a transition ( X t , A t , R t , X t +1 ) , deﬁne the associated distributional TD error as ∆ π ( X t , A t , R t , X t +1 ) :
= ( b R t ,γ ) # η (cid:0) X t
+1 , A πt +1 (cid:1) − η
( X t , A t ) .
When the context is clear, we also adopt the concise notation ∆ πt =
∆ π ( X t , A t , R t , X t +1 ) .
Byconstruction, distributional TD errors are signed measures with zero total mass [10].
The distributional
TD error is a natural counterpart to the value-based TD error, because they both stem directly fromthe corresponding one-step Bellman operators.
However, unlike in value-based RL, where TD errorsalone sufﬁce to specify the multi-step learning operator (Equation (3)), in distributional RL this is notenough.
We introduce the path-dependent distributional TD error, which serves as the building blockto distributional Retrace.
Deﬁnition 4.2.
( Path-dependent distributional TD error )
Given a trajectory ( X s , A s , R s ) ∞ s =0 ,deﬁne the path-dependent distributional TD error at time t ≥ 0 as follows,
[Equation]
Path-dependent distributional TD errors are deﬁned as a pushforward measures from ∆ πt , wherethe pushforward operations depend on G 0: t − 1 .
This equips (cid:101) ∆ π 0: t with an intriguing property, path- 162 dependency .
Concretely, this means that the path-dependent distributional TD error depends onthe sequence of rewards ( R s ) t − 1 s =0 leading up to step t .
With the above deﬁnitions, we can ﬁnallyrewrite the back-up target of distributional Retrace as a weighted sum of path-dependent distributional
TD errors R π,µ η ( x, a )
=
η ( x, a ) + E µ [ (cid:80) ∞ t =
c 1: t (cid:101) ∆
π 0: t ] .
We now illustrate the difference betweenvalue-based and distributional TD errors.
Comparison with value-based TD equivalents.
The value-based equivalent to the path-dependentdistributional TD error is the discounted value-based TD error (cid:101) δ πt =
γ t δ πt which we brieﬂy mentionedin Section 2.
To see why, note that discounted value-based TD errors allow us to rewrite the valuebased Retrace back-up target as R π,µ Q ( x, a )
=
Q ( x, a ) + E µ [ (cid:80) ∞ t =0 c 1: t (cid:101) δ πt ] .
For direct comparisonbetween the two settings, we rewrite both (cid:101) ∆ π 0: t and (cid:101) δ πt as the difference between two n -step predictionsevaluated at two time steps t and t + 1 ,(cid:101) ∆ π 0:
[Equation]
(Value-based)
The above rewriting attributes the path-dependency to the fact that the n -step distributional prediction 174 (cid:0) b G 0: t ,γ t +1 (cid:1) # η (cid:0) X t +1 , A πt +1 (cid:1) is non-linear in G 0: n − 1 .
Indeed, in the value-based setting, because
G 0: t = G 0: t − 1 + γ t R t the partial sum of rewards G 0: t − 1 cancels out as a common term.
Thisleaves the discounted TD error (cid:101) δ πt path-independent .
In other words, the computation of (cid:101) δ πt does notdepend on past rewards ( R s ) t − 1 s =0 .
In contrast, in the distributional setting, the pushforward operationsare non-linear in the partial sum of rewards G 0: t − 1 .
As a result, G 0: t − 1 does not cancel out in thedeﬁnition of (cid:101) ∆ π 0: t , making the path-dependent TD error (cid:101)
∆ π 0: t depend on the past rewards ( R s ) t − 1 s =0 .
The path-dependent property is not an artifact of the distributional Retrace operator R π,µ ; instead, itis an indispensable element for convergent multi-step distributional learning in general.
We showthis by empirically verifying that multi-step learning operators based on alternative deﬁnitions of
Numerically non-convergent path-independent operators.
Consider the path-independent distributional TD error ∆ πt :=( b 0 ,γ t ) # ∆ πt .
We arrived at this deﬁnition by dropping thepath-dependent term G 0: t − 1 in the pushforward of (cid:101) ∆ π 0: t .
Sucha deﬁnition seems appealing because when η = η π , the error is zero in expectation E µ (cid:104) ∆ π t | X t , A t (cid:105) = 0 .
This impliesthat we can construct a multi-step operator by a weighted sumof the alternative path-independent TD error R π,µn η ( x, a ) :
=η ( x, a ) + E µ (cid:104)(cid:80) ∞ t =0 c 1: t ∆ πt (cid:105) .
By construction, R π,µn has η πas one ﬁxed point.
We provide a very simple counterexample on which R π,µ isnot contractive: consider an MDP with one state and one action.
The state transitions back to itself with a deterministic reward
R t
= 1 .
When the discount factor is γ = 0 . 5 , η π is a Diracdistribution centered at 2 .
We consider the simple case c 1 =
ρ 1and c t = 0 , ∀ t ≥ 2 .
We use the L p distance to measure theconvergence of the distribution iterates [10].
Fig 2 shows that ( R π,µn ) k
η 0 does not converge to η π ,while the one-step Bellman operator T π and distributional Retrace R π,µ are convergent.
In Appendix C, we discuss yet another alternative to (cid:101)
∆ π 0: t designed to be path-independent γ t ∆ πt .
Though the resulting multi-step operator still has η π as one ﬁxed point, we show numerically that it is not contractive on the same simple example.
These results demonstrate that naively removing thepath-dependency might lead to non-convergent multi-step operators.
4.2 Backward-view of distributional multi-step learning
To highlight the difference between distributional and value-based multi-step learning, we discussthe impact that path-dependent distributional TD errors have on the backward-view distributionalalgorithm.
Thus far, distributional back-up targets are expressed in the forward-view , i.e., the back-uptarget at time t is calculated as a function of future transition tuples ( X s , A s , R s ) s ≤ t .
The forwardview algorithms, unless truncated, wait until the episode ﬁnishes to carry out the update, which mightbe undesirable when the problem is non-episodic or has a very long horizon.
In the backward-view , when encountering a distributional TD error ∆ πt , the algorithm carries outupdates for all predictions at time t (cid:48) ≤ t [2].
To this end, the algorithm needs to maintain additionalpartial return traces , i.e., the partial sum of rewards G t (cid:48) : t , in order to calculate the path-dependent
TD error (cid:101) ∆ πt .
Unlike the value-based state-dependent eligibility traces [2, 18], partial return traces aretime-dependent.
This implies that in an episode of T steps, value-based backward-view algorithmsrequire memory of size min( |X||A| , O ( T )) while the distributional algorithms requires O ( T ) .
In addition to the added memory complexity, the incremental updates of distributional algorithmsare also much more complicated due to the path-dependent TD errors.
We remark that the pathindependent nature of value-based TD errors greatly simplify the value-based backward-view algorithm.
For a more detailed discussion, see Appendix D.
4.3 Importance sampling for multi-step distributional RL
In our initial derivation, we arrived at R π,µ through the application of importance sampling (IS) in adifferent way from the value-based setting.
We now highlight the subtle differences and caveats.
For a ﬁxed n ≥ 1 , consider the trace coefﬁcient c t = ρ t
I [ t < n ] .
The back-up target of the resulting
Retrace operator reduces to E µ (cid:104) ρ 1: n − 1 · (cid:0) b G 0: n − 1 ,γ n (cid:1) # η ( X n , A πn ) (cid:105) .
This can be seen as applying
IS to the n -step prediction (cid:0) b G 0: n − 1 ,γ n (cid:1)
# η ( X n , A πn ) .
As a caveat, note that an appealing alternativeapproach is to apply IS to G 0: n − 1 , producing the estimate (cid:0) b ρ 1: n − 1 G 0: n − 1 ,γ n (cid:1) # η ( X n , A πn ) .
Thislatter estimate does not properly correct for the off-policy discrepancy between π and µ .
To see why,note that applying the IS ratio to G 0: n − 1 , instead of to the probability of its occurrence, is an artifactof value-based RL because the expected return is linear in G 0: t [11].
In general for distributional RL,one should importance weigh the measures instead of sum of rewards.
Approximate multi-step distributional reinforcement learning algorithm
We now discuss how the distributional Retrace operator combines with parametric distributions, usingthe construction of the novel Quantile Regression-Retrace algorithm as a practical example.
Wefocus on the quantile representation because it entails the best empirical performance of large-scaledistributional RL [16, 19].
Speﬁcially, we present an application of quantile regression with signedmeasures, which is interesting in its own right.
Below, we start with a brief background on quantilerepresentations [16], followed by details on the proposed algorithm.
Consider parametric distributions of the form: 1 m (cid:80) mi =
δ z i for a ﬁxed m ≥ 1 , where ( z i ) mi =1 ∈ Rare a set of parameters indicating the support of the distribution.
Let P Q ( R ) denote the family ofdistribution P Q ( R ) :
= { 1 m (cid:80) mi =1 δ z i | z i ∈ R } .
We deﬁne the projection Π Q : P ∞ ( R ) → P Q ( R )as Π Q η = arg min ν ∈ P Q ( R ) W 1 ( η, ν ) , which projects any distribution onto the space of representabledistributions in the parametric class under the W 1 distance.
With an abuse of notation, we also letΠ Q denote the component-wise projection when applied to vectors.
See [16, 10] for more details.
Gradient-based learning via quantile regression.
We can use quantile regression [20–22] tocalculate the projection Π Q η .
Let F η ( z ) , z ∈ R denote the CDF of a given distribution η .
Let F − 1 ηbe the generalized CDF inverse, we deﬁne the τ -th quantile as F − 1 η ( τ ) for τ ∈ [0 , 1] .
The projectionΠ Q is equivalent to computing z
i =
F − 1 η
( τ i ) for τ ∈ ( 2 i − 1 2 m ) mi =1 [16].
To learn the τ -th quantile forany τ ∈ [0 , 1] , it sufﬁces to solve the quantile regression problem whose optimal solution is F − 1 η ( τ ) :min θ L τθ ( η ) :=
E Z ∼ η
[ f τ ( Z − θ )]
where f τ ( u ) =
u ( τ − I [ u < 0]) .
In practice, we carry out thegradient update θ ← θ − α ∇ θ L τθ ( η ) to ﬁnd the optimal solution and learn the quantile θ ≈ F − 1 η ( τ ) .
5.1 Distributional Retrace with quantile representations
Given an input distribution vector η , we use the distributional Retrace operator to construct theback-up target R π,µ η .
Then, we use the quantile projection to map the back-up target onto thespace of representations Π Q R π,µ η .
Overall, we are interested in the recursive update: start withany η 0 ∈ P Q ( R ) X×A , consider the sequence of distributions generated via η k
+
=
Π Q R π,µ η k .
A direct application of Proposition 3.2 allows us to characterize the convergence of the sequence,following the approach of [10].
Theorem 5.1.
( Convergence of quantile distributions )
The projected distributional Retrace operator Π Q R π,µ is β -contractive under W ∞ distance in P Q ( R ) .
As a result, the above η k converges toa limiting distribution η π R in W ∞ , such that W ∞ ( η k , η π R ) ≤ ( β ) k W ∞ ( η 0 , η π R ) .
Further, the qualityof the ﬁxed point is characterized as W ∞ ( η π R , η π ) ≤ (1 − β ) − 1 W ∞ (Π Q η π , η π ) .
Thanks to the faster contraction rate β ≤ γ , the advantage of the projected operator Π Q R π,µ istwo-fold: (1) the operator often contracts faster to the limiting distribution η π R than the one-stepoperator T π contracts to its own limiting distribution η T π [16]; (2) the limiting distribution η π R alsoenjoys a better approximation bound to the target distribution.
We verify such results in Section 7.
5.2 Quantile Regression-Retrace: distributional Retrace with quantile regression
Below, we use z i ( x, a ) to represent the i -th quantile of the distribution at ( x, a ) .
Overall, we havea tabular quantile representation η z ( x, a ) = 1 m (cid:80) mi =
δ z i ( x,a ) , ∀ ( x, a ) ∈
X × A , where we use thenotation η z to stress the distribution’s dependency on parameter z i ( x, a ) .
For any given bootstrappingdistribution vector η ∈ P ∞ ( R ) X×A , in order to approximate the projected back-up target Π Q R π,µ ηwith the parameterized quantile distribution η z , we solve the set of quantile regression problems forall 1 ≤ i ≤ m, ( x, a ) ∈
X × A ,
min z i ( x,a ) L τ i
z i ( x,a ) ( R π,µ η ( x, a )) , where τ i = (2 i − 1) / 2 m .
For any ﬁxed ( x, a, i ) , to solve the quantile regression problem, we apply gradient descent on z i ( x, a ) .
In practice, with one sampled trajectory ( X s , A s , R s ) ∞ s =0 ∼ µ , the aim is to construct an unbiasedstochastic gradient estimate of the QR loss
L τ i z i ( x,a ) ( R π,µ η ( x, a )) .
Below, let b t =
b G 0: t − 1 ,γ t forsimplicity.
We start with a stochastic estimate (cid:98) L τ i z
i ( x,a ) ( R π,µ η ( x, a )) for the QR loss,
L τ i z i ( x,a ) ( [Equation] ( x, a [Equation]
+ [Equation] t =
[Equation]
Since (cid:98) L τ i z i ( x,a ) ( R π,µ η ( x, a )) is differentiable with z i ( x, a ) , we use ∇ z i ( x,a ) (cid:98)
L τ i z i ( x,a ) ( R π,µ η ( x, a ))
asthe stochastic gradient estimate.
This gradient estimate is unbiased under mild conditions.
Lemma 5.2.
( Unbiased stochastic QR loss gradient estimate ) Assume that the trajectory terminates within H < ∞ steps almost surely, then we have E µ [ (cid:98) L τ i z
i ( x,a ) ( R π,µ η ( x, a ))] =
L τ i z i ( x,a ) ( R π,µ η ( x, a )) and E µ [ ∇ z i ( x,a ) (cid:98)
L τ i z i ( x,a ) ( R π,µ η ( x, a ))]
= ∇ z i ( x,a ) L τ i z i ( x,a ) ( R π,µ η ( x, a )) .
The above stochastic estimate bypasses the challenge that the QR loss is only deﬁned againstdistributions, whereas sampled back-up targets (cid:98) R π,µ η ( x, a ) =
η ( x, a ) + (cid:80) ∞ t =0 c 1: t (cid:101) ∆ π 0: t are signedmeasures in general.
In Quantile Regression-Retrace, we use η z itself as the bootstrapping distribution,such that the algorithm approximates the ﬁxed point iteration η z ← Π Q R π,µ η z .
Concretely, we carryout the following sample-based update
[Equation]
5.3 Deep reinforcement learning: QR-DQN-Retrace
We introduce a deep RL implementation of the Quantile Regression-Retrace: QR-DQN-Retrace,where the parametric representation is combined with function approximations [23, 16, 19].
Thebase agent QR-DQN [23] parameterizes the quantile locations
z i ( x, a ; w ) with the output of a neuralnetwork with weights w .
Let η ( x, a ; w )
=
1 m (cid:80) mi =
δ z i ( x,a ; w ) denote the parameterized distribution.
QR-DQN-Retrace updates its parameters by stochastic gradient descent on the estimated QR loss,averaged across all m quantile levels w ← w − α 1 m (cid:80) mi =1 ∇ w (cid:98)
L τ i z i ( x,a ; w ) ( R π,µ η ( x, a ; w )) .
Inpractice, the update is further averaged over state-action pairs sampled from a replay buffer.
[Figure]
Discussions
Categorical representations.
The categorical representation is another commonly used class ofparameterized distributions in prior literature [23, 17, 24, 10].
We obtain contractive guarantees forthe categorical representation similar to Theorem 5.1.
As with QR, this leads both to improved ﬁxedpoint approximations and faster convergence.
Further, this leads to a deep RL algorithm C51-Retrace.
The actor-critic Reactor agent [25] uses C51-Retrace as a critic training algorithm, although withoutexplicit consideration or analysis of the associated distributional operator.
See Appendix E for details.
We empirically evaluate the stand-alone improvements of C51-Retrace over C51 in Section 7.
Uncorrected methods.
The uncorrected methods do not correct for the off-policyness and henceobtain a biased ﬁxed point [26–28].
The Rainbow agent [26] combined n -step uncorrected learningwith C51, effectively implementing a distributional operator whose ﬁxed point differs from η π .
On-policy distributional TD( λ ).
Nam et al.
[29] propose SR( λ ), a distributional version of onpolicy TD ( λ ) [30].
In operator form, this can be viewed as a special case of Equation (4) with µ = π ,c t = λ ; [29] also introduce a sample-replacement technique for more efﬁcient implementation.
Experiments
We carry out a number of experiments to validate the theoretical insights and empirical improvements.
7.1 Illustration of distributional
Retrace properties on tabular MDPs
We verify a few important properties of the distributional Retrace operator on a tabular MDP.
Theresults corroborate the theoretical results from previous sections.
Throughout, we use quantilerepresentations with m = 100 atoms; we obtain similar results for categorical representations.
See
Appendix F for details on the experiment setup.
Let η 0 be the initial distribution, we carry outdynamic programming with R π,µ and denote η k =
( R π,µ )
k
η 0 as the k th distribution iterate.
Impact of off-policyness.
We control the level of off-policyness by setting the behavior policy µto be a uniform policy and the target policy to π =
(1 − ε ) µ + επ d where π d is a ﬁxed deterministicpolicy.
Moving from ε = 0 to ε = 1 , we transition from on-policy to very off-policy.
We use
L p ( η k , η π R ) to measure the contraction rate to the ﬁxed point.
Fig 3 shows that as the behaviorbecomes more off-policy, the contraction slows down, degrading the efﬁciency of multi-step learning.
Impact of trace coefﬁcient c t .
Throughout, we set c t = min( ρ t , c ) with c to control the effectivetrace length.
With a ﬁxed level of off-policyness ε = 0 .
5 , Fig 3(b) shows that increasing c speeds upthe contraction to the ﬁxed point as predicted by Proposition 3.2.
Quality of ﬁxed point.
We next examine how the quality of the ﬁxed point is impacted by c ,by measuring L p ( η k , Π Q η π ) as a proxy to L p ( η k , η π ) .
As k increases the error ﬂattens, at whichpoint we take the converged value to be L p ( η π R , Π Q η π ) which measures the ﬁxed point quality.
Fig 3(c) shows when c increases, the ﬁxed point quality improves, in line with the Theorem 5.1.
Thisphenomenon does not arise in tabular non-distributional reinforcement learning, although relatedphenomena do occur when using function approximation techniques.
[Figure]
Bias of uncorrected methods.
Finally, we illustrate a critical difference between Retrace anduncorrected n -step methods [26]: the bias to the ﬁxed point.
Fig 3(d) shows that uncorrected n -steparrives at a ﬁxed point in between η π and η µ , showing an obvious bias from η π .
7.2 Deep reinforcement learning
We consider the control setting where the target policy π is the greedy policy with respect to the
Q-function induced by the parameterized distribution.
Because the training data is sampled from areplay, the behavior policy µ is ε -greedy with respect to Q-functions induced by previous copies ofthe parameterized distribution.
We evaluate the performance of deep RL agents on 57 Atari games[31].
To ensure fair comparison across games, we compute the human normalized scores for eachagent, and compare their evaluated mean and median scores across all 57 games during training.
Deep RL agents.
The multi-step agents adopt exactly the same hyperparameters as the baselineagents.
The only difference is the back-up target.
For completeness of results, we show the combination of Retrace with both C51 and QR-DQN.
For QR-DQN, we use the Huber loss for quantileregression, which is a thresholded variant of the QR loss [16].
Throughout, we use c t = λ min( ρ t , c )with c = 1 as in [13].
See Appendix F for details.
In practice, sampled trajectories are truncated atlength n .
We also adapt Retrace to the n -step case, see Appendix A.
Results.
Fig 6 compares one-step baseline, Retrace and uncorrected n -step [26].
For C51, bothmulti-step methods clearly improve the median performance over the one-step baseline.
Retraceslightly outperforms uncorrected n -step towards the end of learning.
For QR-DQN, all multi-stepalgorithms achieve clear performance gains.
Retrace signiﬁcantly outperforms the uncorrected n -stepwith the mean performance, while obtaining similar results on the median performance.
Overall,distributional Retrace achieves a clear improvement over the one-step baselines.
The uncorrectedn -step method typically takes off faster than Retrace but may to slightly worse performance.
Finally, note that in the value-based setting, uncorrected methods are generally more high-performingthan Retrace, potentially due to a favorable trade-off between contraction rate and ﬁxed-point bias[32].
Our results add to the beneﬁts of off-policy corrections in the control setting.
Conclusion
We have identiﬁed a number of fundamental conceptual differences between value-based and distributional RL in multi-step settings.
Central to such differences is the novel notion of path-dependentdistributional TD error, which naturally arises from the multi-step distributional RL problem.
Building on this understanding, we have developed the ﬁrst principled multi-step off-policy distributionaloperator Retrace.
We have also developed an approximate distributional RL algorithm, Quantile
Regression-Retrace, which makes distributional Retrace highly competitive in both tabular andhigh-dimensional setups.
This paper also opens up a several avenues for future research, such as theinteraction between multi-step distributional RL and signed measures, and the convergence theory ofstochastic approximations for multi-step distributional RL.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work
[Yes]
(c) Did you discuss any potential negative societal impacts of your work [No] (d)
Have
you read the ethics review guidelines and ensured that your paper
conforms to
them? [Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]