U NDERSTANDING AND L EVERAGING O VERPARAMETERIZATION IN R ECURSIVE V ALUE E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The theory of function approximation in reinforcement learning (RL) typically considers low capacity representations that incur a tradeoff between approximation error, stability and generalization.
Current deep architectures, however, operate in an overparameterized regime where approximation error is not necessarily a bottleneck.
To better understand the utility of deep models in RL we present an analysis of recursive value estimation using overparameterized linear representations that provides useful, transferable ﬁndings.
First, we show that classical updates such as temporal difference (TD) learning or ﬁtted-value-iteration (FVI) converge to different ﬁxed points than residual minimization (RM) in the overparameterized linear case.
We then develop a uniﬁed interpretation of overparameterized linear value estimation as minimizing the Euclidean norm of the weights subject to alternative constraints.
A practical consequence is that RM can be modiﬁed by a simple alteration of the backup targets to obtain the same ﬁxed points as FVI and TD (when they converge), while universally ensuring stability.
Further, we provide an analysis of the generalization error of these methods, demonstrating per iterate bounds on the value prediction error of FVI, and ﬁxed point bounds for
TD and RM.
Given this understanding, we then develop new algorithmic tools for improving recursive value estimation with deep models.
In particular, we extract two regularizers that penalize out-of-span top-layer weights and co-linearity in top-layer features respectively.
Empirically we ﬁnd that these regularizers dramatically improve the stability of
TD and FVI, while allowing RM to match and even sometimes surpass their generalization performance with assured stability
I NTRODUCTION
Model-free value estimation remains a core method of reinforcement learning (RL), lying at the heart of some of the most prominent achievements in this area (Mnih et al., 2015; Bellemare et al., 2020).
Such success appears paradoxical however, given that value estimation is subject to the deadly triad : any value update that combines off-policy estimation with Bellman-bootstrapping and function approximation diverges in the worst case (Sutton and Barto, 2018).
Without additional assumptions it is impossible to ensure the viability of iterative value estimation schemes, yet this remains a dominant method in RL—its popularity supported by empirical success in many applications.
Such a sizable gap between theory and practice reﬂects limited understanding of such methods, how they behave in practice, and what accounts for their empirical success (van Hasselt et al., 2018; Achiam et al., 2019).
Decomposing the deadly triad indicates that off-policy estimation and bootstrapping are difﬁcult to forego: off-policy estimation is supported by the empirical effectiveness of action value maximization and replay buffers, while Bellman-bootstrapping provides signiﬁcant advantages over Monte Carlo estimation (Sutton, 1988).
On the other hand, our understanding of the third factor, the relationship between function representation and generalization, has evolved dramatically in recent years.
Although it was once thought that restrictive function approximation—representations that lack capacity to ﬁt all data constraints—might be essential for generalization, we now know that this view is oversimpliﬁed (Belkin et al., 2019).
The empirical success of deep learning (Krizhevsky et al., 2012), extremely large models (Brown et al., 2020) and associated theoretical advances (Jacot et al., 2018) have made it clear that gradient-based training of overparameterized models embodies implicit biases that encourage generalization even after all data constraints are ﬁt exactly.
This success suggests a
new opportunity for breaking the deadly triad: by leveraging overparameterized value representations
one can avoid some of the most difﬁcult tradeoffs in value-based RL (Lu et al., 2018).
The use of overparameterized deep models in value-based RL, however, still exhibits mysteries in stability and performance.
Although one might expect larger capacity models to improve the stability of Bellman-bootstrapping, in fact the opposite appears to occur (van Hasselt et al., 2018).
Our own empirical experience indicates that classical value estimation with deep models always diverges eventually in non-toy problems.
It has also been shown that value updating leads to premature rankcollapse in deep models (Kumar et al., 2021), coinciding with instability and degrading generalization.
In practice, some form of early-stopping is usually necessary to obtaining successful results, a fact that is not often emphasized in the literature (Agarwal et al., 2021).
Meanwhile, there is a long history of convergent methods being proposed in the RL literature—starting from residual gradient (Baird, 1995), to gradient-TD (Sutton et al., 2008; Maei et al., 2009), prox gradient TD (Liu et al., 2015; 2016), and emphatic TD (Yu, 2015; Sutton et al., 2016)—yet none of these has demonstrated sufﬁcient generalization quality to supplant unstable methods.
The current state of development leaves an awkward tradeoff between stability and generalization.
A stable recursive value estimation method that ensures generalization quality with overparametrization remains elusive.
In this paper we investigate whether overparameterized value representations might allow the stabilitygeneralization tradeoff to be better managed, enabling stable estimation methods that break the deadly triad and generalize well.
To this end, we ﬁrst consider policy evaluation with overparameterized linear value representations, a simpliﬁed setting that still imposes the deadly triad (Zhang et al., 2021).
Here we ﬁrst show that alternative updates, such as temporal differencing (TD), ﬁtted value iteration (FVI) and residual minimization (RM) converge to different ﬁxed points in the overparameterized case (when they converge), even though these updates share a common ﬁxed point when the approximation error is zero and there are no extra degrees of freedom (Dann et al., 2014).
That is, these algorithms embody certain implicit biases that only become distinguishable in the overparameterized case.
From this result, we observe that the ﬁxed points lie in different bases, which we use to develop a uniﬁed view of iterative value estimation as minimizing the Euclidean norm of the weights subject to alternative constraint sets.
This uniﬁcation allows us to formulate alternative updates that share common ﬁxed points with TD and FVI but guarantee stability without requiring regularization or prox constraints (Zhang et al., 2021).
Next, we analyze the generalization performance of these algorithms and provide a per-iterate bound on the value estimation error of FVI, and ﬁxed point bounds on the value estimation error of TD.
From these results, we identify two novel regularizers, one that closes the gap between RM and TD and another that quantiﬁes the effect of the feature representation on the generalization bound.
We deploy these regularizers in a realistic study of deep model training for optimal value estimation and observe systematic stability and generalization improvements.
We also observe that the performance gap between RM and TD/FVI can be closed and in some cases eliminated.
R ELATED WORK
Value estimation has a lengthy history throughout RL research.
Our main focus is on off-policy value estimation with parametric function representations and iterative (i.e., gradient based) updates.
We do not consider exploration nor full planning problems (i.e., approximately solving an entire Markov decision process (MDP)) in the theoretical development, but instead focus on ofﬂine value estimation; however, we do apply the ﬁndings to policy improvement experiments in the empirical investigation.
Dann et al.
(2014) provide a comprehensive survey of value estimation with parametric function representations.
Signiﬁcant attention has been focused on under parameterized representations where backed up values are not necessarily expressible in the function class, however we focus on the overparameterized case where any backed up values can be assumed to be exactly representable with respect to ﬁnite data.
This change fundamentally alters the conclusions one can draw about algorithm behavior, as we see below.
One of the key consequences is that classical distinctions (Scherrer, 2010;
Dann et al., 2014) between objectives—e.g., mean squared Bellman error (MSBE), mean squared projected Bellman error (MSPBE), mean squared temporal difference error (MSTDE), and the norm of the expected TD update (NEU)—all collapse when the Bellman errors can all be driven to zero.
Despite this collapse, we ﬁnd that algorithms targetting the different objectives—TD and LSTD for
MSPBE (Sutton, 1988; Bradtke and Barto, 1996) and RM without double sampling (DS) for MSTDE (Maei et al., 2009; Dann et al., 2014)—converge to different ﬁxed points given overparameterization, even when they ultimately satisfy the same set of temporal consistency constraints.
It is well known that classical value updates can diverge given off-policy data and parametric function representations (Baird, 1995; Tsitsiklis and Van Roy, 1996; 1997).
The stability of these methods has therefore been studied extensively with many mitigations proposed, including restricting the function representation (Gordon, 1995; Szepesv´ari and Smart, 2004) or adjusting the representation to ensure contraction (Kolter, 2011; Ghosh and Bellemare, 2020;
Wang et al., 2021b), or modifying the updates to achieve convergent variations, such as LSTD (Bradtke and Barto, 1996; Yu, 2010), FVI (Ernst et al., 2005; Munos and Szepesv´ari, 2005; Szepesv´ari and Munos, 2008; Lizotte, 2011) or the introduction of target networks (Mnih et al., 2015; Lillicrap et al., 2016; Zhang et al., 2021; Carvalho et al., 2020).
Others have considered modiﬁed the updates to combat various statistical inefﬁciencies (van Hasselt, 2010; Weng et al., 2020; Konidaris et al., 2011).
Another long running trend has been to consider two time-scale algorithms and analyses, reﬂected in gradient-TD methods (Sutton et al., 2008;
Maei et al., 2009), prox gradient TD (Liu et al., 2015; 2016), primal-dual TD (Dai et al., 2017; Du et al., 2017), and emphatic TD (Yu, 2015; Sutton et al., 2016).
Beyond mere convergence, however, we discover a greater diversity in ﬁxed points among algorithms in the overparameterized case, which play a critical but previously unacknowledged role in generalization quality.
The fact that minimizing MSPBE via TD methods still dominates practice appears surprising given the theoretical superiority of other objectives.
It has been argued, for example, that direct policy gradient methods (Sutton et al., 1999) dominate minimizing Bellman error objectives (Geist et al., 2017).
Even among Bellman based approaches, it is known that MSBE can upper bound the value estimation error (MSE) whereas MSPBE cannot (Kolter, 2011; Dann et al., 2014), yet MSPBE minimization (via TD based methods) empirically dominates minimizing MSBE (via residual methods).
This dominance has been thought to be due to the double sampling bias of residual methods (Baird, 1995; Dann et al., 2014), but we uncover a more interesting ﬁnding that their ﬁxed points lie in different bases in the overparameterized setting, and that reducing this difference closes the performance gap.
We analyze the convergence of classical updates given ofﬂine data and provide associated generalization bounds, with the primary goal of understanding the discrepancy between previous theory and the empirical success of TD/FVI versus RM.
Although this theory sheds new light in exploitable ways, it cannot overcome theoretical limits on ofﬂine value estimation, such as lower bounds on worst case error that are exponential in horizon length (Wang et al., 2021a;b; Zanette, 2021; Xiao et al., 2021).
We analyze the convergence of the expected updates, extendible to the stochastic case using known techniques (Yu, 2010; Bhandari et al., 2018;
Dalal et al., 2018; Prashanth et al., 2021; Patil et al., 2021).
We expand the coverage of these earlier works by including alternative updates and focusing on the overparameterized case, uncovering previously unobserved differences in the ﬁxed points.
There is a growing body of work on linear value estimation and planning that leverages the insight of (Parr et al., 2008; Taylor and Parr, 2009) that linear value estimation is equivalent to linear model approximation.
A number of works have strived to obtain provably efﬁcient algorithms for approximating the optimal policy values in this setting, but these generally rely on exploration or strong assumptions about data coverage (Song et al., 2016;
Yang and Wang, 2019; Duan et al., 2020;
Agarwal et al., 2020; Jin et al., 2020; Yang et al., 2020; Hao et al., 2021) that we do not make.
Instead we study linear value estimation to gain insight, but rather than focus on linear planning we leverage the ﬁndings to improve the empirical performance of value estimation with deep models.
P RELIMINARIES
Notation We let R denote the set of real numbers, I n an n × n identity matrix, and I the indicator function.
For a ﬁnite set X , we use ∆( X ) to denote the set of probability distributions over X .
For a vector µ we let | supp ( µ ) | denote the size of the support of µ (i.e., the number of nonzero entries in µ ).
For a matrix A ∈ R n × m , we let A † be the Moore-Penrose pseudoinverse of A , (cid:107) A (cid:107) be its spectral norm, λ max ( A ) and λ min ( A ) be its maximum and minimum non-zero eigenvalues.
We also use Π A
=
A † A to denote the projection matrix to the row space of A .
For a vector x ∈ R d , we let (cid:107) x (cid:107) be its l 2 norm and (cid:107) x (cid:107)
A = √ x (cid:62) A x be the associated norm for a positive deﬁnite matrix A .
We also use diag ( x ) ∈ R d × d to denote a diagonal matrix whose diagonal elements are x .
Markov reward processes We consider the problem of predicting the value of a given stationary policy in a Markov Decision Process (MDP).
For a stationary policy, this problem can naturally formulated in terms of a Markov reward process M = {S , P, r, γ } , such that S is a ﬁnite set of states, r : S → R and P : S → ∆( S ) are the reward and transition functions respectively, and γ ∈ [0 , 1) is the discount factor.
For a given state s ∈ S , the function r ( s ) gives the immediate reward incurred at s , while P ( ·| s ) gives the next-state transition probability of s .
The value function speciﬁes the future discounted total reward obtained from each state, which is deﬁned as
[Equation]
To simplify the presentation we identify functions as vectors to allow vector-space operations.
In particular, the reward function r is identiﬁed as a vector r ∈ R |S| , and the transition P is identiﬁed as an |S| × |S| transition matrix, where the s -th row P s speciﬁes the transition probability P ( ·| s ) of state s .
These deﬁnitions allow the value function to be expressed using Bellman’s equation
[Equation]
Linear Function Approximation It is usually not possible to consider tabular value representations in practice, since the state set is usually combinatorial or inﬁnite.
In our theoretical development we focus on linear function approximations, where v is approximated by a linear combination of features describing states; i.e., v ( s ) ≈
φ ( s ) (cid:62) θ , where θ ∈ R
d is a parameter vector and φ : S → R d maps a given state s ∈ S to a d -dimensional feature vector φ ( s ) ∈ R d .
We let Φ ∈ R |S|×
d denote the feature matrix, with the s -th row corresponding to the feature vector φ ( s ) , so that the value approximation can be written as v
≈ Φ θ .
We assume (cid:107) φ ( s ) (cid:107) ≤ 1 for any s ∈ S , and for simplicity we also assume that there is no redundant or irrelevant features in the feature map; that is, Φ is full rank.
3. B ATCH V ALUE E STIMATION
We consider batch mode (“ofﬂine”) estimation of the value function.
Let µ ∈ ∆( S ) be an arbitrary probability distribution over states and D µ = diag ( µ ) .
The data set consists of { s i , r i , s (cid:48) i } ni = transition tuples, which are generated by s ∼ µ, r i =
r ( s i ) , s (cid:48)
i ∼ P ( ·| s i ) .
Let n ( s )
=
(cid:80) ni
=
I ( s i = s ) be the number of counts of state s .
We deﬁne the empirical data distribution matrix D = diag (ˆ µ ) , where ˆ µ ( s ) =
n ( s ) /n is the empirical data distribution over states.
The goal is to estimate the value function by ﬁnding a weight vector θ ∈ R d that minimizes the value prediction error ,
[Equation]
Let ˆ P be the empirical transition matrix, where the s -th row represents the estimated transition of state s : if n ( s ) > 0 , ˆ P s ( s (cid:48) )
= (cid:80) ni
=
I ( s i = s, s
[Equation]
i = s (cid:48) ) /n
( s ) ; if n ( s )
= 0 , ˆ P s ( s (cid:48) )
=
for all s (cid:48) ∈ S .
The empirical mean squared Bellman error on the batch data can be deﬁned as
[Equation]
Over vs Underparameterized Features In this paper we are particularly interested in the overparameterized regime d > | supp (ˆ µ ) | where one can exactly satisfy the temporal consistencies on all transitions in the batch data set, achieving zero Bellman error.
(Obviously this would also be possible if d = | supp (ˆ µ )
| but the strictly overparameterized case is more interesting, as we will see below.)
By contrast, in the underparameterized regime d < | supp (ˆ µ ) | , one can only expect to ﬁnd an approximate solution that in general has nonzero Bellman error.
We consider three core algorithms in our analysis, covering major classical approaches.
Residual Minimization (RM) RM directly minimizes the empirical mean squared Bellman error
Eq.
(4) (MSBE)
(Baird, 1995).
The gradient update (Dann et al., 2014) can be expressed as
[Equation]
) where θ t is the estimated weight at step t , and η is the learning rate.
As a gradient descent method, the convergence of this update is robust, and applies to both linear and nonlinear function approximation.
Temporal Difference (TD) Learning The simplest variant of TD (Sutton, 1988), known as TD(0), also updates weights iteratively using transition data to approximate the value function.
Let θ t be the weight vector at step t .
Then the so-called “ semi-gradient ” of Eq.
(4) is used to compute the update,
[Equation]
where η is the learning rate.
From Eq.
(6), it is clear that in the underparameterized ( d < | supp (ˆ µ ) | ) regime, if the system converges, it must converge to parameters θ ∗ D
such that
[Equation]
where θ ∗ D is the TD ﬁxed point .
That is, given limited representational power, the TD ﬁxed point minimizes the squared projected Bellman error (MSPBE) by solving the projected Bellman equation :
[Equation]
such that Π D Φ = Φ(Φ (cid:62) D Φ) − 1 Φ (cid:62) D is a weighted projection matrix.
It is well-known that TD(0) can diverge if the data sampling distribution µ is not the stationary distribution of the Markov process.
One can still compute the TD ﬁxed point directly using batch data, for example using the LSTD algorithm (Bradtke and Barto, 1996), but this requires computation on the order of O ( d 2 ) compared to O ( d ) of the iterative update algorithm Eq.
The value prediction error of TD is discussed in (Tsitsiklis and Van Roy, 1997; Kolter, 2011;
Dann et al., 2014; Bhandari et al., 2018).
Fitted Value Iteration (FVI) FVI iteratively updates the weight vector by solving a regression problem where the target is constructed from the current estimate (Ernst et al., 2005;
Dann et al., 2014), which is also known as approximate dynamic programming (Sutton and Barto, 2018).
In particular, given the current weight θ t at iteration t , the objective Eq.
(9) is minimized to obtain θ t +1 ,
[Equation]
A simple calculation shows the TD ﬁxed point matches the ﬁxed point of FVI whenever θ 0 is in the row-span of D Φ .
Although convergence of FVI can be established under strong conditions (Szepesv´ari and Munos, 2008), the algorithm can be quite unstable in the general batch setting (Chen and Jiang, 2019;
Wang et al., 2021b).
O VER -P ARAMETERIZED L INEAR V ALUE F UNCTION A PPROXIMATION
In this section, we study the convergence properties of the value estimation algorithms introduced in Section 3.1 in the overparameterized regime where d > | supp (ˆ µ ) | .
To faciliate analysis, we ﬁrst introduce additional notation to simplify the derivations.
Let { x i } ki =1 denote the states in the support of ˆ µ , such that n ( x i ) > 0 for all
i = { 1 , . . .
, k } and k
= | supp (ˆ µ ) | .
Deﬁne a mask matrix
H ∈ R k ×|S| and a truncated empirical data distribution matrix D k ∈ R k × k according to
[Equation]
where 1 x i ∈ { 0 , 1 } |S| is an indicator vector such that φ ( x i ) =
Φ (cid:62) 1 x i .
We can then translate between the full distribution and its support via the following.
Proposition 1.
The empirical data distribution matrix D can be decomposed as D = H (cid:62) D k H .
Let M =
H Φ , N =
H ˆ P Φ and R =
H r denote the state features, the expected next state features under the empirical transitions, and the rewards on the support of the data distribution respectively.
Overparameterized Residual Minimization
We ﬁrst study the convergence of RM given a ﬁxed
D .
First note that the update Eq.
(5) can be re-written as
[Equation]
In the overparameterized regime, one can easily verify that there are inﬁnitely many solutions θ ∈ R d satisfying ( M − γ N ) θ
= R .
The gradient of Eq.
(11) is zero at any of these solutions, which implies that RM can have inﬁnitely many ﬁxed points.
However, given that RM minimizes the MSBE objective via gradient descent, as we show in the following theorem, the RM update initialized from θ 0 = 0 will converge to a unique ﬁxed point.
Theorem 1.
With η ≤ 1 (1+ γ ) 2 and starting from θ 0 = 0 , RM converges to θ RM =
( M − γ N ) † R .
Remark 1.
For simplicity we present the ﬁxed points of RM and TD starting from θ 0 = 0 .
The ﬁxed points given an arbitrary initial weight vector θ 0 ∈ R d are shown in Appendices A.1 and A.2.
This result parallels similar ﬁndings in the supervised learning literature, that training overparameterized deep models with gradient descent (or related algorithms) encodes implicit regularization
[Equation]
M =
[ φ 1 , φ 2 ] (cid:62) .
Without loss of generality, we let φ
=
(cos τ, sin τ, 0) and φ 2 = (1 , 0 , 0) .
N =
[ φ (cid:48) 1 , φ (cid:48) 2 ] (cid:62) , where φ (cid:48)
= φ 2 , φ (cid:48)
2 = ( − cos τ,
[Equation]
22 sin τ, √ 22 sin τ ) .
Then
[Equation] [Equation]
Clearly, the spectral norm of W increases as the angle τ between φ 1 and φ 2 decreases.
that drives the model solution to particular outcomes in the overparameterized regime (Soudry et al., 2018; Gunasekar et al., 2018; Neyshabur et al., 2019).
Moreover, this implicit regularization is often associated with generalization beneﬁts.
However, unlike the case for supervised learning, RM solutions do not often generalize well.
Below we uncover a key difference between the RM ﬁxed point and those of TD and FVI that sheds new light on the source of generalization differences.
Overparameterized TD Learning We next consider the convergence properties of the TD(0) update in the overparameterized setting.
First, rewrite the TD(0) update formula Eq.
(6) as
[Equation]
Similar to RM, in the overparameterized regime any solutions θ ∈ R d that satisfy ( M − γ N ) θ = R are the ﬁxed points of Eq.
(12), which implies an inﬁnite set of ﬁxed points.
This is quite unlike the underparameterized case where there is a unique TD ﬁxed point Eq.
(7) given by the solution of projected Bellman equation.
However, we now show that in the overparameterized setting, similar to solving RM using gradient descent, TD also encodes an implicit bias toward a particular ﬁxed point.
This of course requires TD to converge, which can be assured by a simple condition.
Let W = NM † , which has a geometric interpretation that we will exploit later in Section 5.
Observe that
[Equation]
i.e., N can be decomposed into its projection onto the row-span of M plus a perpendicular component.
Eq.
(13) shows that W projects N onto the row space of M ; see Fig.
1 for an illustration.
We refer to W as the core matrix since its norm determines the convergence of TD.
Theorem 2.
Choosing η < 1 (1+ γ ) (cid:107) Φ (cid:107) and starting from θ 0 = 0 , if (cid:107) W (cid:107) <
1 γ , TD(0) converges to θ TD
= M † ( I k − γ W ) − 1 R .
If (cid:107) W (cid:107) ≥ 1 γ there is an initial θ 0 for which TD(0) does not converge.
A few key observations.
First, note that the RM ﬁxed point in Theorem 1 and the TD ﬁxed point in
Theorem 2 are not identical.
That is, the different value estimation algorithms continue to demonstrate different preferences for ﬁxed points, but in the overparameterized setting these differences are implicit in the algorithms and cannot be captured by the MSBE versus MSPBE objectives, since both are zero for any θ that satisﬁes ( M − γ N )
θ =
R .
Second, the ﬁxed point of TD lies in different basis than
RM .
That is, θ TD lies in the row space of the state features M , whereas θ RM lies in the row space of the residual features M − γ N , and these two spaces are not identical in general.
We revisit the signiﬁcance of this difference below, but intuitively, the parameter vector θ is being trained to predict values rather than temporal differences, and the future test states from which value predictions are made will tend to be closer to the space spanned by M than M − γ N .
Overparameterized Fitted Value Iteration
Finally, we consider the convergence of FVI.
Recall that at iteration t , FVI solves the least squares problem Eq.
(9) to compute the next weight vector.
Using the notation established above, the normal equations for this problem can be expressed as
M
[Equation]
M
[Equation]
=
M
[Equation]
D k
( R + γ N θ t ) , but this system cannot be directly used to compute the solution since M (cid:62)
D k M is not invertible.
Furthermore, just like RM and TD, any θ ∈ R d that satisﬁes ( M − γ N ) θ
= R is a ﬁxed point of FVI.
If one solves the least squares problem Eq.
(9) using gradient descent, it is known (Bartlett et al., 2021; Soudry et al., 2018) that the optimization converges to the minimum norm solution
[Equation]
Interestingly, by choosing this solution, each iteration of FVI corresponds to applying a linear backup on the current value estimate, where the backup operator is deﬁned by the core matrix.
Deﬁnition 1.
Deﬁne the core matrix linear operator T W by T W ν
=
R + γ W ν for any ν ∈ R |S| .
Using this operator we can characterize the convergence condition of FVI, reaching the conclusion that whenever T W is a non-expansion, FVI converges to the same ﬁxed point as TD.
Theorem 3.
Let θ 0 be the initial weight and θ t ∈ R
d be the output of FVI at iteration t .
We have
[Equation]
Furthermore, given that (cid:107) W (cid:107) < 1 /γ , the algorithm converges to θ TD = M † ( I k − γ W ) − 1 R .
If (cid:107) W (cid:107) ≥ 1 γ there is an initial θ 0 for which FVI does not converge.
4. U NIFIED V IEW OF O VERPARAMETERIZED V ALUE E STIMATORS
We now show that the convergence points above can be characterized as solutions to related constrained optimization problems, providing a uniﬁed perspective on the respective algorithm biases.
Theorem 4.
θ RM is the solution of the following constrained optimization,
[Equation]
and θ TD is the solution of the following constrained optimization,
[Equation]
That is, the convergence points of RM, TD and FVI in the overparameterized case can all be seen as minimizing the Euclidean norm of the weights θ subject to satisfying the Bellman constraints
M θ = R + γ N θ , where TD and FVI implicitly add the additional constraint that θ must lie in the row span of M ; moreover, this is the only constraint that differentiates TD from RM.
From this perspective, the algorithms can all be seen as iterative procedures for solving a particular form of quadratic program, when they converge.
Of course, proper constrained optimization techniques would be able to stably compute solutions in scenarios where TD or FVI diverge (Boyd and Vandenberghe, 2004), but a more direct way to ensure convergence is implied by the following corollary.
Corollary 1.
θ TD is also the solution of the following constrained optimization,
[Equation]
Note that the right hand side of the constraint simply pre-projects next state value predictions onto the row space of M before determining the Bellman backed up value.
This allows a novel objective to be formulated whose minimizer recovers the same ﬁxed point as TD,
[Equation]
which stands for mean squared corrected Bellman error.
Note that MSCBE is not identical to MSPBE because the projection is applied before not after the Bellman backup.
Gradient descent minimization of MSCBE yields the same ﬁxed point as θ TD , which is essentially equivalent to applying RM to corrected target values while ensuring stability.
Note also that in the linear case the projection matrix Π M only needs to be precomputed once.
4. V ALUE P REDICTION E RROR B OUNDS
One can also establish generalization bounds on the value estimation error of these methods in the overparameterized regime.
We ﬁrst provide a ﬁnite time analysis of the value prediction error of FVI.
Theorem 5.
Let ˆΣ =
M (cid:62) D k M be the empirical covariance matrix, and θ t be the output of FVI starting from θ 0 as deﬁned in Theorem 3.
Then for any θ ∗ ∈ arg min θ ∈ R d E ( θ ) ,
[Equation]
where ε = (cid:107) N
( I d − Π M ) θ ∗ (cid:107) and σ = (cid:107) H ( ˆ P − P ) v (cid:107) .
Intuitively, ε measures the length of next-state features along the direction θ ∗ , and σ is the expected value prediction error under the empirical transition model, which can be bounded using standard concentration inequalities.
The proof of this theorem is given in Appendix A.5.
Observe that for any step t ≥ 1 , the output of FVI θ t is within the row-span of M .
This allows us to decompose the prediction error into a component within the row-span, controlled by leveraging the core matrix linear operator T W , and an orthogonal component that can be bounded by (cid:107) θ ∗ (cid:107)
I d − Π M .
Under the convergence conditions of Theorems 2 and 3, we also have the following generalization bound for the value prediction error of θ TD .
Corollary 2.
Suppose that (cid:107) W (cid:107) ≤ 1 , and the value prediction for any s ∈ S is bounded by v ( s ) ∈ [0 , v max ] .
For any θ ∗ ∈ arg min θ ∈ R d E ( θ ) ,
[Equation]
where n min = min s : n ( s ) > 0 n ( s ) is the minimum counts given the data set.
This result automatically implies the requirements for ensuring ofﬂine generalization, accounting both for distribution shift (Wang et al., 2021b) and policy completeness (Munos and Szepesv´ari, 2005; Duan et al., 2020) in feature space.
In particular, for Eq.
(20) and Eq.
(21), we characterize the distribution shift using well known concentration bounds in Appendix A.6, which leads to the denominators kλ min ( ˆ Σ) and n min E [ λ min ( ˆ Σ)] respectively.
In addition, we explicitly characterize the misalignment between the features of current states and next states using the core matrix, which can be used to bound misalignment between values, replacing the feature completeness assumption.
We note that if the convergence condition cannot be satisﬁed, that is when (cid:107) W (cid:107) ≥ 1 /γ , the estimation error could be arbitrarily large.
The sources of value estimation error are explicit in Corollary 2.
The ﬁrst term measures the error due to sampling (statistical error), while the second term considers out-of-span components of the optimal weight vector θ ∗ with respect to M (approximation error).
The smallest eigenvalue of the empirical covariance matrix E [ λ min (ˆΣ)] , as well as the length of the orthogonal components E [ (cid:107) θ ∗ (cid:107)
I d − Π M ] , can both be controlled using classical techniques for concentration properties of random matrix.
In Appendix A.7 we present the exact approach for bounding these two terms.
Furthermore, by Corollary 1, one can also apply Corollary 2 to an algorithm that directly optimizes MSCBE.
Although a solution of Eq.
(18) must exist, its value prediction error can be arbitrarily large given that (cid:107) W (cid:107) ≥ 1 /γ .
This also connects to a similar result for the TD ﬁxed point that minimizes MSPBE in the underparameterized regime (Kolter, 2011).
R EGULARIZERS FOR D EEP R EINFORCEMENT L EARNING A LGORITHMS
For tractability, the theory in prior sections assumes ﬁxed representations with a linear parameterization on only the ﬁnal layer parameters of the value function.
However, in practice, deep RL algorithms also learn the representations in an end-to-end fashion.
Inspired by the linear case, we now identify two novel regularizers that are applicable more generally—one that closes the gap between RM and
TD inspired by the uniﬁed view of different ﬁxed points, and another that quantiﬁes the effect of feature representation on the generalization bound.
Two-Part Approximation Most deep RL algorithms rely on approximating the value function with a deep neural network Q ω that predicts the future outcome of given state-action pair (Mnih et al., 2015;
Kalashnikov et al., 2018; Lillicrap et al., 2016).
In practice, Q ω is trained by TD learning that minimizes the objective (cid:80) s,a ( r ( s, a ) + γ ¯ Q ω ( s, a ) − Q ω ( s, a )) , where ¯ Q ω ( s, a ) is known as the target network to increase the learning stability.
We view Q ω as a two part-approximation with ω =
( φ, θ ) , where the output of the penultimate layer is referred as the feature mapping φ , the weight of last fully connected layer is referred as θ , and the Q-function is approximated by
Q ω ( s, a ) =
φ ( s, a ) (cid:62) θ .
Our goal is to deﬁne regularizers on φ and θ that can be effectively applied to practical algorithms.
The ﬁrst regularizer directly takes inspiration from Theorem 4: by restricting the linear weight θ within the row space of M (now deﬁned by exited ( s, a ) pairs in the data), RM ﬁnds the same ﬁxed point as TD.
We implement this idea by penalizing the norm of the perpendicular component of θ ,
[Equation]
In practice we compute this regularizer for each minibatch of data.
The projection step is computed by a least squares algorithm with an additional l 2 regularization for numerical stability.
[Figure]
The second regularizer is designed to address the effect of the feature representation on convergence and value prediction error.
In particular, as shown by Theorems 2 and 3, a sufﬁcient condition that guarantees the convergence of TD and FVI is that the spectral norm of W be upper bounded by 1 /γ , which by Theorem 5 will also reduce the bound on generalization error.
Hence, it is natural to penalize the norm of this matrix using standard automatic differentiation tools.
However, such an approach is prone to numerical difﬁculty, as it involves differentiation through a matrix pseudo inverse.
We instead propose an alternative regularizer inspired by the geometric interpretation of the core matrix Eq.
(13): recall from Fig.
1 that W can be viewed as the weights that project N onto the row space of M .
To ensure that an arbitrary feature vector can be well approximated using W , it would be ideal if M was orthonomral, which would imply an ideally-behaved basis to represent N .
This intuition justiﬁes the following regularization:
[Equation]
(23) where β is a scale parameter designed to approximate the column norm.
That is, the regularizer forces the neural network to learn an orthogonal feature embedding by normalizing the empirical feature covariance matrix.
The gradient of R φ can also approximated using mini-batches.
We augment the original learning objectives by adding both R θ and R φ weighted by hyper-parameters.
5. E MPIRICAL J USTIFICATION OF R EGULARIZERS
The goal of our experiments is to assess the applicability of the proposed regularization schemes based on orthogonality and projection operations to practical deep RL algorithms.
To avoid the confounding effects of exploration, we restrict our study to learning from a frozen batch of data with a ﬁxed number of transitions collected prior to learning.
We use a randomly initialized policy in this initial collection step.
We consider both discrete and continuous control benchmarks in this analysis.
For the discrete action environments, we use DQN (Mnih et al., 2015) as the baseline algorithm to add our regularizers.
For continuous control environments, we use QT-Opt (Kalashnikov et al., 2018) as the baseline algorithm, which is an actor-critic method that applies the cross-entropy method to perform policy optimization.
Our modiﬁcations add R φ and R θ to the standard MSBE objective on the critic Q-network.
Additional details describing the complete experiment setup for each environment are provided in Appendix B. Experimental results contrasting vanilla TD and
RM with their regularized variants are summarized in Fig.
These ﬁndings demonstrate that our regularization schemes can be used to improve the performance of both vanilla TD learning and
RM.
Note that RM is typically less popular than TD due to its worse empirical performance.
On
Acrobot and Reacher, our method was able to fully close the gap between RM and TD.
On Cartpole, (where vanilla RM dominates vanilla TD), and on Pendulum, our regularizers also deliver signiﬁcant improvements to the TD learning baseline and modest improvements to the RM baseline.
C ONCLUSION
We have investigated the ﬁxed points of classical updates for value estimation in the overparameterized setting, where there is sufﬁcient capacity to ﬁt all the Bellman constraints in a given data set.
We ﬁnd that TD and FVI have different ﬁxed points than RM, but in the linear case the difference can be entirely attributed to a constraint missing from RM that the solution lie in the row space of the predecessor state features.
We devised two novel regularizers based on these ﬁndings, which stabilized the performance of TD without sacriﬁcing generalization, while improving the generalization of RM, in the setting of estimating optimal values with a deep model.
Characterizing the implicit bias of other algorithms, such as gradient or emphatic TD variants remains open.
Identifying other regularizers that further close the gap between TD and RM is also an interesting direction for future investigation.
A P ROOFS A. P ROOF OF T HEOREM
Theorem 6 (Restatement of Theorem 1) .
Let θ
∈
R
d be the initial weight vector.
With η ≤ 1 (1+ γ ) 2 ,
RM converges to θ RM =
( M − γ N ) † R
+
( I d − Π M − γ N ) θ 0 .
Proof.
Let A =
M − γ N for simplicity.
First recall the residual minimization update,
[Equation]
Let θ ∗ =
A † R .
It can be veriﬁed θ ∗ is one of the feasible solution as A θ ∗ =
R .
Then a simple recursive argument shows that for any θ 0 ∈ R d ,
[Equation]
Thus,
[Equation]
We let V Λ V (cid:62) be its eigendecomposition of A (cid:62) D k A , which is the empirical covariance matrix of residual features.
Let V − be the null space of V .
Then
[Equation]
Let λ max be the largest eigenvalue of A (cid:62) D k A .
We now show that λ max ≤ 1 + γ .
[Equation]
) where we use the fact that λ max is a convex function and we assume (cid:107) φ ( s ) (cid:107) ≤ 1 for all s ∈ S .
Thus, given that η ≤ 11+ γ , η ≤ 1 λ max .
Then I d − ( I d − η A (cid:62) D k A ) t +1 =
V V (cid:62) as t → ∞ .
Thus
[Equation]
(38) where the last equality follows by that θ ∗ is in the row space of A by deﬁnition.
When θ 0 = 0 , we have the algorithm converges to θ ∗ .
We next show the result for general θ 0 .
Let θ
= θ 10 + θ 20 , where θ
10 =
Π M − γ N θ 0 is the component of θ 0 that is in the row space of A , θ
=
( I d − Π M − γ N ) θ 0 is the perpendicular residual.
Then,
[Equation]
where the last step follows by the choice of η .
This ﬁnishes the proof.
A. P ROOF OF T HEOREM
Theorem 7 (Restatement of Theorem 2) .
Let θ
∈
R
d be the initial weight vector.
Assuming that
M (cid:62) D k ( M − γ N ) is diagonalizable, with η < 1 (1+ γ ) (cid:107) Φ (cid:107) , if (cid:107) W (cid:107) < 1 γ , TD(0) converges to θ TD
= M † ( I k − γ W ) − 1 R + β , where β
= Q 0 Q − 1 0 θ 0 , Q 0 are eigenvectors of M (cid:62)
D k
( M − γ N ) with zero eigenvalues.
If (cid:107) W (cid:107) ≥ 1 γ there is an initial θ 0 for which TD(0) does not converge.
Proof.
We ﬁrst rewrite the TD update formulate as
[Equation] [Equation]
A simple recursive argument shows that for any θ 0 ∈ R d ,
[Equation]
Then note that
[Equation]
By writing N as the projection to the row-span of M and the perpendicular component, we have
[Equation]
where the last step follows by ( I d − M † M ) M (cid:62)
= 0 .
Thus we can rewrite θ t +1 as
[Equation]
Given (cid:107) W (cid:107) < 1 /γ , we have that all eigenvalues of I k − γ W are positive.
Let η < 1 (1+ γ ) (cid:107) Φ (cid:107) , then (cid:107) η
[Equation]
I k
[Equation]
MM
[Equation]
<
[Equation]
k
[Equation]
MM
[Equation]
D k
[Equation]
otherwise the matrix power series diverges.
Thus
[Equation]
Therefore, given that θ 0 = 0 , we have the algorithm converge to M † ( I k − γ W ) − 1 R .
We now show the convergence point for an arbitrary θ 0 .
Let Q Λ Q − 1 be the eigen decomposition of M (cid:62)
D k ( M − γ N ) .
By the low rank structure of this matrix, it has at most h ≤ k non-zero eigenvalues.
Let Q 0 be the eigenvectors with eigenvalue zero .
Then
[Equation]
where the last step follows by the choice of η .
Characterization for non-diagonalizable M (cid:62) D k ( M − γ N )
Let Z =
M (cid:62) D
k
( M − γ N ) and Z = QJQ − 1 be the jordan normal form of Z .
We still denote
Q 0 the eigenvectors with eigenvalue zero.
Then there is
[Equation]
Since I − η J has a block diagonal structure, its power can be obtained by ﬁrst computing the power of each block.
Let J i be the jordan block with eigenvalue λ
i .
We write J i
=
λ i I + L , where L is a matrix such that the only non-zero entries of L are on the ﬁrst off-diagonal. Then we can write the i -th block of J as (1 − ηλ i )
I − η L .
Using the binomial theorem we get
[Equation]
Note that L s is the matrix with ones on the s -th diagonal away from the main diagonal, and L s =for s larger than the size of L .
Therefore, ((1 − ηλ i )
I − η L ) t is a triangular matrix with (1 − ηλ i ) t on the main diagonal, − ηt (1 − ηλ i ) t − 1 on the ﬁrst off-diagonal, and so on.
Therefore, the eigenvalues of this matrix are all (1 − ηλ i ) t .
Then given a learning rate that η < 1 /λ max , for any jordan block with λ i > 0 , we have that the matrix power converges.
For λ i = 0 , the jordan block corresponds to eigenvectors that are in the kernel space of Z .
Thus, suppose that all eigenvalues of Z are non-negative, we have
[Equation]
Note that if a negative λ i exists, the above derivations can still be used to characterize the convergent sub-component of θ 0 .
The non-convergent sub-component of θ 0 will diverge with an exponential rate as shown above.
A. P ROOF OF T HEOREM
Proof.
We ﬁrst prove the update formula.
Recall the FVI update,
[Equation]
For t = 1 the result holds by deﬁnition.
For t ≥ 2 , a simple recursive argument reveals that for every θ 0 ∈ R d , θ
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
Clearly the convergence of this algorithm depends on the spectral norm of NM † .
In particular, given that (cid:107) NM † (cid:107) < 1 /γ , we have the algorithm converges to
[Equation]
as t → ∞ .
This ﬁnishes the proof.
A. P ROOF OF T HEOREM 4 AND C OROLLARY
Proof.
We ﬁrst prove the result for residual minimization ﬁxed point θ RM .
Let A =
M − γ N for simplicity.
We write the Lagrange of the optimization problem,
[Equation]
Solving for α ∗ and add it to θ ∗ =
A (cid:62) α ∗ gives that θ ∗ =
A † R .
We next prove Corollary [Equation] which characterizes the TD and FVI ﬁxed point θ TD .
Let W = NM † .
We write the Lagrange of the optimization problem,
[Equation]
Solving for α ∗ and add it to θ ∗ =
M (cid:62) ( I k − γ W ) (cid:62) α ∗ gives that θ ∗ =
M † ( I k − γ W ) − 1 R .
The second part of Theorem 4 is immediately followed by this.
A. P ROOF OF T HEOREM
Lemma 1.
Let θ t be the output of FVI at iteration t with θ 0 as the initial parameter.
We have that
M † M θ t
= θ t for any t ≥ 1 .
Proof.
The claim is implied by the fact that θ t is in the row space of M .
In particular, by Theorem 3, θ t = M † α for some α ∈ R n .
Thus,
[Equation]
This ﬁnishes the proof.
Lemma 2.
E ( θ ) is 1-smoothness.
Proof.
Recall the prediction error of θ ∈ R d
[Equation]
(78) where Σ = Φ (cid:62) D µ Φ .
The gradient of θ is E (cid:48)
( θ ) =
Σ( θ − θ ∗ ) .
Then (cid:107)E [Equation] [Equation] 1 ) − E
[Equation]
[Equation]
2 ) (cid:107) = (cid:107)
[Equation]
≤ λ
[Equation]
[Equation]
(cid:107) (
[Equation]
−
[Equation] (cid:107) ≤ (cid:107) [Equation] (cid:107) , (79) where we use (cid:107) φ ( s ) (cid:107) ≤ 1 for all s ∈ S and λ max (Σ) ≤ (cid:80) s µ ( s ) λ max ( φ ( s )
[Table]
Lemma 3.
Let ε app
=
N Π ⊥ M θ ∗ and σ stat =
H ( P − ˆ P )Φ θ ∗ .
We have
[Equation]
for any T ≥ 1 .
Proof.
A simple recursive argument shows that,
[Equation]
Proof.
By Theorem [Equation] θ t = M † T t − 1 ( R + γ N θ 0 ) is the output of FVI at iteration t .
By noting that
E ( θ ∗ ) = 0 and Lemma 2, for any θ ∈ R d .
[Equation]
We ﬁrst consider the second term.
By Lemma 1,
[Equation]
[Equation]
We now consider the ﬁrst term.
By Lemma 3,
[Equation]
[Equation]
Let ˆΣ =
M (cid:62) D k M be the empirical covariance matrix.
Note that λ min ( M (cid:62) M ) /k ≥ λ min (ˆΣ) .
To show this, let ¯ D = diag ( 1 k , . .
. , 1 k ) , and M = USV (cid:62) be the SVD of M .
Then
[Equation]
[Equation] [Equation] (88) where we replace x = V α since V are orthonormal bases, and min i ∈
[ k ] ˆ µ i ≤ 1 /k .
Therefore, (cid:107) M † (cid:107) =
1 / (cid:113) λ +min ( M † M ) ≤ 1 / (cid:113) kλ +min (ˆΣ) .
Combining the results above we have,
[Equation]
Combine this with Eq.
(81) ﬁnishes the proof.
A. P ROOF OF C OROLLARY
Proof.
Recall that in the proof of Theorem 5 we have
[Equation]
[Equation]
Given that (cid:107) W (cid:107) < 1 , we have for the ﬁxed point θ ∞ ,
[Equation]
We ﬁrst consider σ 2 = (cid:107) H
( P − ˆ P ) v (cid:107) 2 .
By Hoeffding’s inequality and a union bound we have with probability at least 1 − δ , for any s ∈ supp ( D ) , [Equation]
[Equation]
Thus, let n min =
min s : n ( s ) > 0 n ( s ) , we have
[Equation]
Now we consider ε 2 = (cid:107) N Π ⊥ M θ ∗ (cid:107) 2 .
Since N Π ⊥ M is perpendicular to M , and all features have norm bounded by one,
[Equation]
[Equation]
Combine the above we have,
[Equation]
Finally, using the tower rule gives the desired result.
A. C ONCENTRATION OF E IGENVALUES AND B OUNDING THE O RTHOGONAL C OMPLEMENT
We will need the following result (Kuzborskij et al., 2021, Theorem 6), which is concerned with the magnitude of projection onto the eigenspace of a covariance matrix.
The result is based on (Shawe-Taylor et al., 2005, Theorem 1)
Lemma 4.
Let ˆΣ = 1 n (cid:80) i x i x (cid:62) i be the covariance matrix of i.i.d.
data x i ∈ R d .
Denote the h -“tail” of eigenvalues of a covariance matrix ˆΣ = as
[Equation]
Let U r be the ﬁrst r eigenbasis for r ∈ [ n ] .
Then for any z ∈ R d , with probability at least 1 − δ ,
[Equation]
The next lemma gives a non-asymptotic result to understand the behaviour of ˆ λ min (Kuzborskij et al., 2021, Lemma 1).
Lemma 5.
Let X =
[ X 1 , . .
, X n ] ∈ R d × n be a random matrix with i.i.d.
columns, such that max i (cid:107) X i (cid:107) 2 ≤ K , and let ˆΣ =
XX (cid:62) /n , and Σ = E
[ X 1 X (cid:62) 1 ] .
Then, for every α ≥ 0 , with probability at least 1 − 2 e − α , we have
[Equation]
n ≥
[Equation]
[Equation]
and furthermore, assuming that (cid:107) X
i (cid:107) Σ † = √ d , for all i ∈ [ n ] , we have
[Equation]
104) where we have an absolute constant c = 2 3 . 5 √ ln 9 .
A. N EURAL T ANGENT K ERNEL E XTENSION
We show how to extend the results of Theorem 3 using neural tangent kernel.
Let v ( s, ω ) be the value function approximated by a neural network, where ω is all the parameters of the neural network.
The neural tangent kernel is deﬁned as
[Equation]
where s, s (cid:48) are two states, Ω is an initialization distribution over ω .
Let ω 0 be random initial parameters sampled from Ω , there is
[Equation]
See (Arora et al., 2019, Theorem 3.1) for the proof.
Note that in this kernel, an input s is mapped to a feature vector
[Equation]
Consider a least square problem: given a training dataset { x i , y i } ki =1 , consider training the neural network by minimizing the squared loss over training data: 12 (cid:80) ki
=1 ( v ( x i , ω ) − y i ) 2 .
The following result shows that a properly randomly initialized sufﬁciently wide deep neural network trained by gradient descent is equivalent to a kernel regression predictor with neural tangent kernel (Arora et al., 2019, Theorem 3.2),
[Equation]
where ker ( x, X ) = ( ker ( x, x 1 ) , . .
, ker ( x, x k ))
(cid:62) , ker ( X , X ) ∈ R k × k
whose
i, j ) -th element is ker ( x, x i ) , and y ∈ R k whose i -th element is y i .
Recall that { x i , r i , x (cid:48)
i } ki =1 is the batch dataset on the support of the ˆ µ .
Let ker ( X (cid:48) , X ) be a n × n matrix where the ( i, j ) -th element is ker ( x (cid:48)
i , x j ) .
For simplicity, we assume that FVI uses zero value prediction initially.
We now prove that for any s ∈ S , the value prediction at the t -th iteration of FVI is given by
[Equation]
We prove this result by induction.
For the base case i = 1 , the result directly follows by Eq.
[Equation]
Now suppose the induction hypothesis holds for i = t − 1 , then by Eq.
[Equation]
[Equation]
Then, given that the following matrix is a contractor
[Equation]
we have that at the convergence, the value prediction of FVI for any s ∈ S is given by
[Equation]
Note that ker ( X (cid:48) , X ) ker
( X , X ) − 1 corresponds to the core matrix W in the linear setting.
B E XPERIMENT SETUP
In this section, we provide additional details about the experimental setup and hyper-parameters used for each of the environments.
For all of these environments the regularization weights were considered as tunable hyper-parameters.
In addition, for R φ (see Eq 23), the scale factor β was also considered as a parameter to be tuned in order to approximate the feature matrix norm.
B. A CROBOT
• Replay buffer with 10k tuples sampled using a random policy across trajectories with maximum episode length of 64.
• A DQN with hidden units consisting of fully connected layers with (100 , 100) units.
• Batch size 64.
• Learning rate of 1e-3.
• Regularized RM with weight of 2e-2 on R φ and 1e-4 on R w .
• Regularized TD with weight of 0 on R φ and 1e-4 on R w .
B. R EACHER
• Replay buffer with 10k tuples sampled from a random policy across trajectories with maximum steps per episode of length 50.
• Learning rate 1e-4.
• A value network for the continuous action inputs with a fc observation layer with params (50,), a fc action layer with params (50,) and a joint fc layer with params (100,).
• Batch size 64.
• Gradient clipping with a norm of 10.
• Regularized RM with weight of 0.15 on R w and 0 on R φ .
• Regularized TD with weight of 2e-2 on R w and 7e-3 on R φ .
[Figure]
(1e6)
[Figure]
R e t u r n s
HalfCheetah-v2 ddpgreg_ddpg
[Figure]
HalfCheetah 0.
[Figure]
(1e6)
[Figure]
R e t u r n s
[Figure]
-v2 ddpgreg_ddpg
[Figure] [Figure]
(1e6)
[Figure]
e t
[Figure] [Figure]
[Figure]
R e t u r n s
[Figure]
v2 ddpgreg
[Figure] [Figure]
• Replay buffer with 10k tuples sampled using a random policy across trajectories with maximum steps per episode of length 50.
• A DQN with hidden units consisting of fully connected layers with (100 , 100) units.
• Batch size 64.
• Learning rate 1e-3.
• Regularized RM with weight of 0.29 on R w and 0 on R φ .
• Regularized TD with weight of 1.5e-3 on R w and 5e-3 on R φ .
B.
P ENDULUM
• Replay buffer with 1k tuples obtained by sampling directly from a ﬁxed initial state distribution using a random policy.
• A value network for the continuous action inputs with a fc observation layer with params (50,), a fc action layer with params (50,) and a joint fc layer with params (100,).
• Batch size 64.
• Learning rate 1e-3.
• Regularized RM with weight of 1.0 on R w and 5.4e-4 on R φ .
• Regularized TD with weight of 0 on R w and 1.0 on R φ .
B. E XTRA E XPERIMENT R ESULTS
We provide extra experiment results on four mujoco control problems to assess the applicability of the proposed regularization R φ : HalfCheetah, Hopper, Ant, and Walker2d.
The results are provided in Fig.
All results are averaged over 100 runs with different random seeds.
The hyper-parameters are provided below.
• The Q-function is approximated by two hidden layer fully neural networks, where the hidden layer size is 256.
• Batch size 256.
• Learning rate 3e-4.
• Regularized TD weight are tuned from { 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 , 1 }