G EOMETRIC I NSIGHTS INTO THE C ONVERGENCE OF N ONLINEAR TD L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
While there are convergence guarantees for temporal difference (TD) learning when using linear function approximators, the situation for nonlinear models is far less understood, and divergent examples are known.
Here we take a ﬁrst step towards extending theoretical convergence guarantees to TD learning with nonlinear function approximation.
More precisely, we consider the expected learning dynamics of the TD(0) algorithm for value estimation.
As the step-size converges to zero, these dynamics are deﬁned by a nonlinear ODE which depends on the geometry of the space of function approximators, the structure of the underlying
Markov chain, and their interaction.
We ﬁnd a set of function approximators that includes ReLU networks and has geometry amenable to TD learning regardless of environment, so that the solution performs about as well as linear TD in the worst case.
Then, we show how environments that are more reversible induce dynamics that are better for TD learning and prove global convergence to the true value function for well-conditioned function approximators.
Finally, we generalize a divergent counterexample to a family of divergent problems to demonstrate how the interaction between approximator and environment can go wrong and to motivate the assumptions needed to prove convergence.
I NTRODUCTION
The instability of reinforcement learning (RL) algorithms is well known, but not well characterized theoretically.
Notably, there is no guarantee that value estimation by temporal difference (TD) learning converges when using nonlinear function approximators, even in the on-policy case.
The use of a function approximator introduces a projection of the tabular TD update into the class of representable functions.
Since the dynamics of TD do not follow the gradient of any objective function, the interaction of the geometry of the function class with that of the TD algorithm in the space of all functions potentially eliminates any convergence guarantees.
This lack of convergence has motivated many authors to seek variants of TD learning that reestablish convergence guarantees, such as two timescale algorithms.
In contrast, in this work we focus on TD learning directly and examine its behavior under generic function approximation.
We consider the simplest case: on-policy discounted value estimation.
To further simplify the analysis, we only consider the expected learning dynamics in continuous time as opposed to the online algorithm with sampling.
This means that we are eschewing discussions of off-policy data, exploration, sampling variance, and step size.
In this continuous limit, the dynamics of TD learning are modeled as a (nonlinear) ODE.
Stability of this ODE is a pre-requisite for convergence of the algorithm.
However, for general approximators and MDPs it can diverge as demonstrated by Tsitsiklis & Van Roy (1997).
Today, the convergence of this ODE is known in two regimes: under linear function approximation for general environments (Tsitsiklis & Van Roy, 1997) and under reversible environments for general function approximation (Ollivier, 2018).
We signiﬁcantly close this gap through the following contributions:
We prove that the set of smooth homogeneous functions, including ReLU networks, is amenable to the expected dynamics of TD.
In this case, the ODE is attracted to a compact set containing the true value function.
Moreover, when we use a parametrization inspired by ResNets, nonlinear TD will have error comparable to linear TD in the worst case.
We prove global convergence to the true value function when the environment is “more reversible” than the function approximator is “poorly conditioned”.
We generalize a divergent TD example to a broad class of non-reversible environments.
These results begin to explain how the geometry of nonlinear function approximators and the structure of the environment interact with TD learning.
S ETUP 2. D ERIVING THE DYNAMICS ODE
We consider the problem of on-policy value estimation.
Deﬁne a Markov reward process (MRP)
M = ( S , P, r, γ ) where S is the state space with |S| = n ﬁnite, P ( s (cid:48) | s ) is the transition matrix, r ( s, s (cid:48) ) is the ﬁnite reward function, and γ ∈ [0 , 1) is the discount factor.
This is equivalent to a
Markov decision process with a ﬁxed policy.
Throughout we assume that P deﬁnes an irreducible, aperiodic Markov chain with stationary distribution µ .
We want to ﬁnd the true value function:
V ∗ ( s ) :=
E [ (cid:80) ∞ t =0 γ t r ( s t , s t +1 ) | s 0 = s ] , where the expectation is taken over transitions from P 1 .
The true value function satisﬁes the Bellman equation:
[Equation]
It will be useful to think of the value function as a vector in R n and to deﬁne R ( s ) =
E s (cid:48) ∼ P ( ·| s )
[ r ( s, s (cid:48) )] so that the Bellman equation becomes V ∗
= R + γPV ∗ .
The most prominent algorithm for estimating V ∗ is temporal difference (TD) learning, a form of dynamic programming (Sutton & Barto, 2018).
In the tabular setting (with no function approximation) the TD(0) variant of the algorithm with learning rates α k makes the following update at iteration k + 1 :
[Equation]
Under appropriate conditions on the learning rates and noise we have that V k
→ V ∗ as k → ∞ (Robbins & Monro, 1951; Sutton, 1988).
Moreover, under these conditions the algorithm is a discretized and sampled version of the expected continuous dynamics of the following ODE.
[Equation]
Letting D µ be the matrix with µ along the diagonal and applying the Bellman equation we get
[Equation]
We now deﬁne
[Equation]
While A is not necessarily symmetric, it is positive deﬁnite in the sense that x T Ax > 0 for nonzero x since 1 2 ( A + A T ) is positive deﬁnite (Sutton, 1988).
This can be seen by showing that A is a non-singular M-matrix (Horn & Johnson, 1994).
This then implies convergence of the ODE deﬁned in (4) to V ∗ regardless of initial conditions.
In practice, the state space may be too large to use a tabular approach or we may have a feature representation of the states that we think we can use to efﬁciently generalize.
So, we can parametrize a value function V θ by θ ∈ R d .
Then, the “semi-gradient” TD(0) algorithm is
[Equation]
Now by an abuse of notation, deﬁne V : R d → R n to be the function that maps parameters θ to value functions V ( θ ) so that V ( θ ) s
= V θ ( s ) .
Now, the associated ODE which we will study becomes:
[Equation]
The method is called “semi-gradient” because it is meant to approximate gradient descent on the squared expected Bellman error, which is not feasible since it would require two independent samples of the next state to make each update (Sutton & Barto, 2018).
This approximation is what results in the lack of convergence guarantees, as elaborated below.
2. C ONVERGENCE FOR LINEAR FUNCTIONS AND REVERSIBLE ENVIRONMENTS
There are two main regimes where the above dynamics are known to converge.
The ﬁrst is when
V ( θ ) is linear and the second when the MRP is reversible so that A is symmetric.
It is a classic result of Tsitsiklis & Van Roy (1997)
that under linear function approximation, where
V ( θ ) =
Φ θ for some full rank feature matrix Φ , TD(0) converges to a unique ﬁxed point (the result also applies to the more general TD( λ )).
The proof uses the fact that in the linear case, letting θ ∗ =
(Φ T A Φ) − 1 Φ T AV ∗ , the dynamics (7) become:
[Equation]
The positive deﬁniteness of A gives global convergence to θ ∗ .
Recent work has extended this result to give ﬁnite sample bounds for linear TD (Bhandari et al., 2018).
Making such an extension in the nonlinear case is beyond the scope of this paper since even the simpler-to-analyze expected continuous dynamics are not understood for the nonlinear case.
Some concurrent work has also extended linear convergence to particular kinds of neural networks in the lazy training regime where the networks behave like linear models (Agazzi & Lu, 2019;
Cai et al., 2019).
The relationship to our work will be discussed in Section 6.
The other main regime where the dynamics converge is when P deﬁnes a reversible Markov chain which makes TD(0) gradient descent (Ollivier, 2018).
In that case, A is symmetric so (7) becomes:
[Equation]
(9) where A must be symmetric to deﬁne an inner product.
Then for any function approximator this gradient ﬂow will approach some local minima.
Note that without this symmetry the TD dynamics in (7) are provably not gradient descent of any objective since differentiating the dynamics we get a non-symmetric matrix which cannot be the Hessian of any objective function (Maei, 2011).
2. A N EXAMPLE OF DIVERGENCE
To better understand the challenge of proving convergence
Tsitsiklis & Van Roy (1997) provide an example of an MRP with zero reward everywhere and a nonlinear function approximator where both the parameters and estimated value function diverge under the expected TD(0) dynamics.
We provide a description of this idea in Figure
[Figure]
TD WITH HOMOGENEOUS APPROXIMATORS INCLUDING NEURAL NETWORKS
Our ﬁrst result is that the expected dynamics of TD(0) are attracted to a neighborhood of the true value function in any irreducible, aperiodic environment when we use a smooth and homogeneous function approximator.
Deﬁnition 1 (Homogeneous) .
f : R k
→ R m is h -homogeneous for h ∈ R if f ( x )
=
h ∇ f ( x ) x .
Note that by Euler’s homogeneous function theorem, this is equivalent to f ( αx )
=
α h f ( x ) for all positive α .
Remark 1.
The ReLU activation as well as the square and several others are homogeneous.
Moreover, neural networks of any depth with such activations remain homogeneous.
This can be found in
Lemma 2.1 of (Liang et al., 2017) and we include a proof in Appendix E for completeness.
Note that linear functions are also homogeneous and we will show that much like linear functions, the set of homogeneous functions works well with TD learning.
At a high level, the intuition is that the image of a homogeneous mapping from parameters to functions is a set in function space who’s geometry prevents the sort of divergence seen in the spiral example.
When V is homogeneous, then the point V ( θ ) in the space of functions must lie in the span of the columns of ∇ V ( θ ) which deﬁne the tangent space to the manifold of functions.
This prevents examples like the spiral where the tangent space is nearly orthogonal to V ( θ ) for all V ( θ ) in the manifold of functions.
However, since homogeneous functions are a much more general class than linear functions, the following result is not quite as strong as the global convergence in the linear setting.
Theorem 1.
Let V : R d → R n be an h -homogeneous function such that (cid:107) V ( θ ) (cid:107) µ ≤ C (cid:107) θ (cid:107) (cid:96) .
Let
B = (cid:107) ( I − γP ) V ∗ (cid:107) µ
1 − γ
= (cid:107) R (cid:107) µ 1 − γ and choose (cid:15) > 0 .
Then, for any initial conditions θ 0 , if θ follows the dynamics deﬁned by (7) we have
[Equation]
The full proof is found in Appendix A.1.
The main technique is to use homogeneity to see that
[Equation]
This allows us to relate the norm of the value function to the dynamics in parameter space.
We can also extend this result to prove that the limsup of the dynamics attains the same bound if we add a stronger assumption that the approximator is bi-Holder continuous.
This result is in Appendix A.2.
One way to think about the theorem is to say that using a homogeneous approximator does at least as well as a baseline given by the zero function.
This is because B is a potentially tight bound on (cid:107) V ∗ − 0 (cid:107) µ , but cannot be known a priori since we do not know the expected rewards in advance.
Using this intuition, we can change the parametrization of the function to include a stronger baseline.
We can use a linear baseline since we understand how TD behaves with linear approximators.
This gives a parametrization that resembles residual neural networks (He et al., 2016) and which we will call residual-homogeneous when the network is also homogeneous.
Deﬁnition 2 (Residual-homogeneous) .
A function f : R k 1 × R k 2 → R m is residual-homogeneous if f ( x 1 , x 2 ) =
Φ x 1 + g ( x 2 ) where Φ ∈ R m × k
and g is h -homogeneous.
For this function class, we prove the following theorem, which extend the ideas from Theorem 1.
Theorem 2.
Let V : R d 1 × R d 2 → R n be a residual-homogeneous function where Φ is a full rank feature matrix and (cid:107) V ( θ ) (cid:107) µ ≤ C (cid:107) θ (cid:107) (cid:96) .
Let Π Φ be the projection onto the span of Φ and let
B Φ = (cid:107) ( I [Equation]
)( V ∗ − Π Φ
[Equation]
∗ ) (cid:107) µ 1 −
[Equation]
Then for any (cid:15) > 0 and any initial conditions θ 0 , if θ follows the dynamics deﬁned by (7) we have
[Equation]
The proof is in Appendix A.3.
This allows us to bound the quality of the value function found by
TD learning as compared to the linear baseline, but we may want to also bound the actual distance to the true value function.
Using the above bound relative to the baseline in conjunction with the quality of the baseline, we can derive the following corollary, with proof in Appendix A.
Corollary 1.
Under the same assumptions as Theorem 2 we have
[Equation]
Remark 2.
For linear TD, we get that lim t →∞ (cid:107) Φ θ − V ∗ (cid:107) µ ≤ (cid:107) V ∗ − Π Φ V ∗ (cid:107) µ 1 − γ at the ﬁxed point θ ∗ (Tsitsiklis & Van Roy, 1997).
So, our result shows that in terms of the worst case bound, residualhomogeneous approximators perform similarly to linear functions, especially for large γ .
These are the ﬁrst results that characterize the behavior of TD for a broad class of nonlinear functions including neural networks regardless of initialization under the same assumptions on the environment as used in the analysis of linear TD.
Our current results resemble those established in the context of non-convex optimisation using residual networks (Shamir, 2018), also obtained under weak assumptions.
One direction for future work would be to extend the results from the liminf to limsup or even to show that the limit exists.
Another direction for future work is to try to strengthen the assumptions and leverage structure in V ∗ itself to reduce the space of possible solutions and be able to make stronger conclusions.
T HE INTERACTION BETWEEN APPROXIMATOR AND ENVIRONMENT
In the previous section we considered a class of approximators for which we can provide guarantees in all irreducible, aperiodic environments.
Now we consider how the function approximator interacts with the environment during TD learning.
Recall that prior work has shown that in reversible environments, TD learning is performing gradient descent (Ollivier, 2018).
Our insight is that strict reversibility is not necessary to make similar guarantees if we also have more information about the function approximator.
As seen in the spiral example, the geometric problem with TD arises from the combination of “spinning” linear dynamics from an asymmetric A matrix (i.e. a non-reversible environment) with a poorly conditioned function approximator that “kills” some directions of the update towards the true value function in function space.
In this section we will formalize this notion by showing how we can trade off environment reversibility and approximator conditioning and still guarantee convergence.
First we need a way to quantify how reversible an environment is and offer the following deﬁnition.
Deﬁnition 3 (Reversibility coefﬁcient) .
Let S A := 1 / 2( A + A T ) and
R A := 1 / 2( A − A T ) be the symmetric and anti-symmetric parts of A , as deﬁned in equation (5).
Then the reversibility coefﬁcient ρ ( M ) is
[Equation]
Note that when the environment is reversible this coefﬁcient is inﬁnite since in that case A is symmetric and R A is zero.
The antisymmetric part R A captures the spinning behavior of the linear dynamical system in function space (as in the spiral example).
At a high level, more spinning means a less reversible environment and larger R A which lowers the reversibility coefﬁcient.
Now we need a compatible way to quantify the effect of the function approximator.
To do this, we have to examine the matrix ∇ V ( θ ) ∇ V ( θ ) T .
This matrix shapes the dynamics of TD and in the case of neural networks under particular assumptions it is known as the neural tangent kernel (Jacot et al., 2018).
The condition number of this matrix gives us one way to quantify how much the approximator prefers updates along the directions in function space corresponding to maximal eigenvalues over those corresponding to minimal eigenvalues.
This gives us a way to quantify how well-behaved the function approximator is and allows us to prove the following theorem.
Theorem 3.
Let κ ( M ) be the condition number of a matrix M .
Assume that for all
[Equation]
Then if θ evolves according to (7) we have that for all θ
[Equation]
where S A := 1 / 2( A + A T ) .
Thus, V ( θ ) → V ∗ regardless of initial conditions.
The proof is relatively simple and proceeds by using the chain rule to write out the time derivative of the Lyapunov function and applying the Courant-Fischer-Weyl min-max theorem along with the assumption to get the result.
The full details can be found in Appendix B.
This result provides strong global convergence guarantees, albeit under fairly strong assumptions. It nevertheless provides intuition about how the environment interacts with the function approximation.
We show that we can use a nonlinear approximator that generalizes across states by using gradient information so long as the condition number of the tangent kernel of the approximator is bounded by the reversibility coefﬁcient.
Note that for the condition number of the kernel to be ﬁnite, the Jacobian must be full rank which means that the function approximator has more parameters than the number of states.
Such an approximator has more parameters than a tabular approximator, but can be nonlinear and generalize across states using the structure of the input representation.
This opens a few directions for future work to formalize the relationship between the environment and approximator in the regime when there are less parameters than states or when the state space is inﬁnite.
It may be fruitful to connect this to the literature from supervised learning on over-parametrized neural networks (see for example Oymak & Soltanolkotabi (2019) and references therein), especially in the case of value estimation from a ﬁnite dataset (i.e. a replay buffer).
Another direction would be to leverage structure in V ∗ and the input representation so that the approximator is effectively over-parametrized and similar arguments can be made, but it is not clear how to formalize such assumptions.
4. E XTENSION TO K - STEP RETURNS
We now consider how the analysis strategy presented above applies to a classical variation on the TD learning algorithm.
We ﬁnd that k -step returns have better convergence guarantees by increasing the reversibility of the effective environment.
This is not completely surprising since in the limit k
→ ∞ we recover gradient descent in the µ -norm with the Monte Carlo algorithm for value estimation.
However, we show that our sufﬁcient condition to guarantee convergence in the well-conditioned regime weakens exponentially with k .
In Appendix C we show that with k step returns the dynamics of the algorithm become
[Equation]
We can deﬁne a notion of effective reversibility that scales exponentially with k such that we recover the same type of convergence as Theorem 3 whenever
[Equation]
where λ 2 ( P ) is the second largest eigenvalue of P , which is strictly less than 1 under our irreducible and aperiodic assumption.
This result shows how using k -step returns to increase the effective reversibility of the environment can lead to better convergence properties.
See Appendix C for a more precise statement of the result and its proof.
4. N UMERICAL EXPERIMENT ON THE DIVERGENT EXAMPLE
We perform a small set of experiments on the divergent spiral example from Section 2.3 which support our conclusions about reversibility and k -step returns.
We integrate the expected dynamics
ODEs in two settings, one where we introduce reversibility into the environment and the other where we increase the value of k in the algorithm.
The function approximator is always the spiral approximator from the example.
We can introduce reversibility by adding reverse connections to the environment with probability δ ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 23 } as shown in Figure 2.
This effectively reduces the spinning of the linear dynamical system in function space deﬁned by the Bellman operator.
We ﬁnd that increasing reversibility eventually leads to convergence.
We also validate the result that increasing k will lead to convergence by increasing the effective reversibility without changing the
MRP.
Note that the spiral example is outside the assumptions of the theory in this section since the function is not well-conditioned, but we wanted to show that the connection between reversibility and convergence may extend beyond the well-conditioned setting.
[Figure]
A GENERALIZED DIVERGENT EXAMPLE
To motivate the necessity of assumptions similar to the ones that we have made we can look again to the spiral example of (Tsitsiklis & Van Roy, 1997).
Here we generalize this example to arbitrary number of states for most non-reversible MRPs.
Our construction allows for approximators with arbitrary number of parameters, but restricts them to have rank deﬁcient tangent kernels to mimic the spiral in a 2-D subspace of function space.
The construction can be found in Appendix D, and the result can be described formally as follows.
Proposition 1.
If the MRP is not reversible such that A
= D µ ( I − γP ) has at least one non-real eigenvalue, then there exists a function approximator V such that TD learning will diverge.
That is, for any initial parameters θ 0 , as t → ∞ we have (cid:107) V ( θ ) − V ∗ (cid:107) → ∞ .
Moreover, ∇ V ( θ ) can have rank up to n − 1 , where n is the number of states, for all θ .
The construction of the approximator in the counterexample is somewhat pathological, but any convergence proofs have to make assumptions to rule out these divergent examples.
In this work we avoid these by using either smooth homogeneous functions or by using well-conditioned functions in nearly-reversible environments.
While there may be other assumptions that yield convergence, they must also account for this class of divergent examples.
R ELATED WORK 6. C ONNECTIONS TO WORK IN THE LAZY TRAINING REGIME
Concurrent work (Agazzi & Lu, 2019) has proven convergence of expected TD in the nonlinear, non-reversible setting in the so-called “lazy training” regime, in which nonlinear models (including neural networks) with particular parametrization and scaling behave as linear models, with a kernel given by the linear approximation of the function at initialization.
Whereas this kernel captures some structure from the function approximation, the lazy training regime does not account for feature selection, since parameters are conﬁned in a small neighborhood around their initialization (Chizat & Bach, 2018).
Another result in a similar direction is from concurrent work (Cai et al., 2019) which considers two-layer networks (one hidden layer) in the large width regime where only the ﬁrst layer is trained.
They show that this particular type of function with ﬁxed output layer is nearly linear and derive global convergence in the limit of large width with an additional assumption on the regularity of the stationary distribution.
In contrast with these works, our results account for feature selection with more general nonlinear functions.
Our homogeneous results hold for a broad class of approximators much closer to those used in practice and our well-conditioned results hold for general nonlinear parametrization and provide useful intuition about the relationship between approximator and environment.
6. C ONNECTIONS TO WORK ON FITTED VALUE ITERATION
Another line of work provides convergence rates for ﬁtted value iteration or ﬁtted Q iteration under the assumption of small optimization error at each iteration (Munos, 2007; Munos & Szepesv´ari, 2008; Yang et al., 2019b).
These papers give bounds that depend on the maximum difference between the function returned by the Bellman operator applied to the current iterate and its projection into the space of representable functions (which they call the inherent Bellman residual).
This assumption means that a priori the function class has geometry amenable to the MDP being evaluated.
Here we do not rely on any assumptions about successful optimization or an assumption that the projection of the tabular TD update into the space of representable functions is uniformly small.
Instead we ﬁnd scenarios where we can guarantee that the difference between the tabular update and projection cannot be too far so that the optimization procedure succeeds.
6. A LTERNATIVE VALUE ESTIMATION ALGORITHMS
There are several papers that introduce new algorithms inspired by TD learning but modiﬁed so as to have provable convergence with nonlinear approximators.
To our knowledge, all of them use either a two timescale argument where the optimization procedure at the faster timescale views the slower timescale as ﬁxed (Borkar, 1997; 2008) or they attempt to optimize a different objective function (Baird, 1995).
Most of the algorithms have not seen widespread use, potentially because these modiﬁcations make optimization more difﬁcult or decrease the quality of solutions.
More speciﬁcally, Baird (1995) present residual algorithms, which attempt to optimize a different objective which avoids double sampling, but has incorrect value functions as solutions (Sutton & Barto, 2018).
Bhatnagar et al.
(2009) present GTD2/TDC which uses two timescales to perform gradient descent on the norm of the TD(0) dynamics projected onto the image of the nonlinear approximator in function space and thus has the same ﬁxed points as TD(0).
More recently, Dai et al.
and
Chung et al.
(2019) present two timescale algorithms which are provably convergent.
Yang et al.
(2019a) characterize target networks, which have seen widespread use (Mnih et al., 2015), as a two timescale algorithm.
Finally, while they do not provide guaranteed convergence with nonlinear functions, Achiam et al.
(2019) present an algorithm that uses a similar observation to ours about the connection between TD-learning with nonlinear functions and the neural tangent kernel.
Their algorithm then estimates a preconditioning matrix that serves a similar function as the two-timescale argument.
In contrast to this line of work on algorithmic modiﬁcations, our work is a ﬁrst step towards characterizing the behavior of nonlinear TD without two timescales or a modiﬁed objective.
6. E MPIRICAL WORK
Recent empirical work by Fu et al.
(2019) empirically investigates the interaction between nonlinear function approximators (neural networks) with Q-learning, which is a semi-gradient algorithm that is similar to TD.
They ﬁnd that divergence is rare and that more expressive approximators reduce approximation error and reduce the chances of divergence.
Since these more expressive approximators are more likely to be well-conditioned, this gives reason to believe it may be possible to extend our convergence results from the well-conditioned setting for very expressive approximators to more realistic approximators.
D ISCUSSION
We have considered the expected continuous dynamics of the TD algorithm for on policy value estimation from the perspective of the interaction of the geometry of the function approximator and environment.
Using this perspective we derived two positive results and one negative result.
First, we showed attraction to a compact set when homogeneous approximators like ReLU networks.
The worst case solution in this set is comparable to the worst case linear TD solution for a particular parametrization inspired by ResNets.
Second, we showed global convergence when the environment is more reversible than the approximator is poorly-conditioned.
Finally, we provided a generalized counterexample to motivate the assumptions necessary to rule out bad interactions between approximator and environment.
There are several possible directions for future work.
First, while our results extend both the linear and reversible convergence regimes, they do not close the gap between the two.
One direction for future work is thus to provide a unifying analysis that would neatly connect all of the convergent regimes.
Next, it may be possible to ﬁnd a more precise notion of the well-conditioning necessary to get local convergence rather than global convergence which would allow extension to more realistic settings.
It may be possible to leverage assumptions about regularity of the true value function so that the function class is effectively well-conditioned.
Another direction is that, while it was beyond the scope of this paper, it would be instructive to extend the results to ﬁnite sample results as has recently been done for linear TD.
It would also be interesting to extend the results to off-policy and Q-learning settings, but likely would require stronger assumptions.
Finally, we would like to motivate future work by noting that the ultimate goal of this line of work is to put TD on the same solid footing as optimization in supervised learning where we can characterize easy problems (by convexity), can guarantee convergence even in hard problems (to local minima), and have some notions of how to make optimization easier (like over-parametrization). Here we have taken a step towards this kind of analysis, but the precise characterization of what makes a problem easy and whether and where TD converges on hard problems remain incomplete.
A TD WITH HOMOGENEOUS APPROXIMATORS A. P ROOF OF T HEOREM
Theorem 1.
Let V : R d → R n be an h -homogeneous function such that (cid:107) V ( θ ) (cid:107) µ ≤ C (cid:107) θ (cid:107) (cid:96) .
Let
B = (cid:107) ( I − γP ) V ∗ (cid:107) µ
1 − γ
= (cid:107) R (cid:107) µ 1 − γ and choose (cid:15) > 0 .
Then, for any initial conditions θ 0 , if θ follows the dynamics deﬁned by (7) we have
[Equation]
Proof.
Applying the chain rule, (7), and the homogeneity assumption we get that:
[Equation]
Whenever B + (cid:15) =
(cid:107) ( I − γP ) V ∗ (cid:107) µ 1 − γ + (cid:15) <
(cid:107) V ( θ ) (cid:107) µ we have that
[Equation]
µ
[Equation]
µ
[Equation]
for c = (cid:15) (1 − γ )
( B + (cid:15) ) > 0 .
The last inequality follows from an application of Lemma 1 of (Tsitsiklis & Van Roy, 1997) that (cid:107) PV (cid:107) µ ≤ (cid:107) V (cid:107) µ along with Cauchy-Schwarz to show that:
[Equation]
Putting this all together we get that whenever B + (cid:15) < (cid:107) V ( θ ) (cid:107) µ we have that
[Equation]
Put another way, we can deﬁne U
= { θ : d (cid:107) θ (cid:107) 2 dt ≥ − c/h } and W = { V : (cid:107) V (cid:107) µ ≤ B + (cid:15) } .
By the above, we have that V ( U ) ⊆
W where V ( U ) is the image of U under V , i.e. { V : ∃ θ ∈ U s.t.
V =
V ( θ ) } .
Now deﬁne O = { θ : C (cid:107) θ (cid:107) (cid:96) <
B + (cid:15) } which contains an open ball around the origin.
By our Holder continuity assumption, we know that V ( O ) ⊆ W .
Putting it all together, if θ (cid:54)∈ U at time t , the dynamics of (cid:107) θ (cid:107) 2 are contracting faster than c/h so that there exists a ﬁnite T > t such that at time T the parameters θ ( T ) will either be in O or U , either way implying V ( θ ) will be in W , giving the result.
A. B I -H OLDER HOMOGENEOUS STABILITY
Proposition 2.
Let V : R d → R n be h -homogeneous.
Moreover, assume that c (cid:107) θ (cid:107) s ≤ (cid:107) V ( θ ) (cid:107) µ ≤
C (cid:107) θ (cid:107) r for some c, s, r, C > 0 .
Let B = (cid:107) ( I − γP )
V ∗ (cid:107) µ 1 − γ
[Equation]
R
[Equation]
µ 1 − γ .
Then, for any initial conditions θ 0 , if θ follows the dynamics deﬁned by (7) we have
[Equation]
Proof.
Using the same reasoning as above without the additional (cid:15) , we get that whenever B = (cid:107) ( I − γP ) V ∗ (cid:107) µ 1 −
[Equation]
< (cid:107) V ( θ ) (cid:107) µ , we have
[Equation]
So, whenever B < (cid:107) V ( θ ) (cid:107) µ , we have that d (cid:107) θ (cid:107) 2 dt < 0 .
Deﬁne O = { θ : c (cid:107) θ (cid:107) s ≤ B } .
Thus, by the lower Holder bound, d (cid:107) θ (cid:107) 2 dt < 0 for all θ (cid:54)∈ O .
So, we have that as t → ∞ , θ ∈
O .
And, by the upper Holder bound, if θ ∈ O then
[Equation]
V
[Equation]
Since, θ ∈ O in the limit, we get the desired result.
A. P ROOF OF T HEOREM 2 AND C OROLLARY
Theorem 2.
Let V : R d 1 × R d 2 → R n be a residual-homogeneous function where Φ is a full rank feature matrix and (cid:107) V ( θ ) (cid:107) µ ≤ C (cid:107) θ (cid:107) (cid:96) .
Let Π Φ be the projection onto the span of Φ and let
B
[Equation]
Then for any (cid:15) > 0 and any initial conditions θ 0 , if θ follows the dynamics deﬁned by (7) we have
[Equation]
Proof.
Let f be homogeneous and
[Equation]
Let θ ∗
= Π Φ V ∗ be the best linear predictor.
In fact, the proof would go through for any θ ∗ 1 , only the bound gets worse for worse choice of θ ∗ 1 .
Note that
[Equation]
The residual parametrization is necessary here to have the linear part of DV ( θ ) independent from θ so that we can turn the difference of θ 1 − θ ∗ 1 into a difference between functions.
As in the proof of Theorem 1, we get that whenever B Φ + (cid:15) < (cid:107) V ( θ ) − Φ θ ∗ 1 (cid:107) 2 µ we have that |
[Equation]
θ
[Equation]
µ
[Equation]
As a consequence we can conclude that whenever B Φ + (cid:15) < (cid:107) V ( θ ) − Φ θ ∗ 1 (cid:107) 2 µ d (cid:107) θ − ( θ ∗ 1 , 0) (cid:107)dt < − c/h.
To conclude the proof, we can deﬁne U
= { θ : d (cid:107) θ − ( θ ∗ 1 , 0) (cid:107)
dt ≥ − c/h } and W =
{ V : (cid:107) V − Φ θ ∗ 1 (cid:107) µ ≤ B + (cid:15) } .
Then we have that V ( U ) ⊆
W .
Now deﬁne O =
{ ( θ ∗ 1 , 0) + θ : C (cid:107) θ (cid:107) (cid:96) <
B + (cid:15) } which contains an open ball around ( θ ∗ 1 , 0) .
By our Holder continuity assumption and the residual parametrization of V , we know that V ( O ) ⊆ W .
Putting it all together, if θ (cid:54)∈ U at time t , the dynamics of (cid:107) θ (cid:107) 2 are contracting faster than c/h so that there exists a ﬁnite T > t such that at time T the parameters θ ( T ) will either be in O or U , either way implying V ( θ ) will be in W , giving the result.
Corollary 1.
Under the same assumptions as Theorem 2 we have
[Equation]
Proof.
Applying the triangle inequality and the result of the Theorem, we get that
[Equation]
Then, we note that (cid:107) I − γP (cid:107) ≤ 1 + γ since P is a stochastic matrix which lets us bound B Φ by 1+ γ 1 − γ (cid:107) Π Φ V ∗ − V ∗ (cid:107) µ .
Putting this together with the above yields the result.
B C ONVERGENCE IN THE WELL - CONDITIONED SETTING B. P ROOF OF THE THEOREM
Theorem 3.
Let κ ( M ) be the condition number of a matrix M .
Assume that for all θ ,
[Equation]
Then if θ evolves according to (7) we have that for all θ
[Equation]
(16) where S A := 1 / 2( A + A T ) .
Thus, V ( θ ) → V ∗ regardless of initial conditions.
Proof.
To simplify notation, we use S for S A and R for R A .
We deﬁne L : R d → R which will be the Lyapunov function for the dynamical system as
[Equation]
By applying the chain rule and the polarization identity to (7) we have
[Equation]
where R := 1 / 2( A − A T ) .
So using the assumption and then the min-max theorem,
[Equation]
where λ max , λ min are the maximal and minimal eigenvalues.
Combining, we conclude that ˙
L ( θ ) < 0 and the result follows.
B. C ALCULATING THE REVERSIBILITY COEFFICIENT
The reversibility coefﬁcient can be seen as the minimizer of a generalized Rayleigh quotient (Horn & Johnson, 1994).
Such a quotient for Hermitian matrices B, C is deﬁned as R B,C ( u ) =
u T Bu u T Cu .
And when C is full rank, we know that this is maximized by the maximal eigenvalue of C − 1 B .
In our case, this gives us a way to calculate the reversibility coefﬁcient as
[Equation]
C K - STEP RETURNS IN THE WELL - CONDITIONED SETTING C. E XPECTED DYNAMICS DERIVATION
The k -step return variant of TD learning replaces r ( s, s (cid:48) )
+ γV θ ( s (cid:48) ) in the original algorithm (6) by (cid:80) k −t =
γ t r ( s t , s t +1 )
+ γ k V
θ ( s k ) .
To get the continuous matrix version of the dynamics we replace
R with V ∗ − γPV ∗ in the following expression to get a telescoping series so that
[Equation]
Thus, we can deﬁne the TD( k ) dynamics by
[Equation]
C. E FFECTIVE REVERSIBILITY
To draw the comparison to the TD(0) algorithm analyzed in the paper, we deﬁne the matrix
[Equation]
Now we take symmetric and asymmetric parts S k
= 12 ( A k + A Tk ) and R k =
( A k − A Tk ) .
Then we can deﬁne reversibility in this effective environment.
Deﬁnition 4 (Effective reversibility coefﬁcient) .
[Equation]
C ONVERGENCE P ROPOSITION
Proposition 3.
Let λ 2 ( P ) be the modulus of the second largest eigenvalue of P , which is strictly less than 1 by our irreducible, aperiodic assumption.
Then
[Equation]
And if θ follows the ODE deﬁned by (21), then d (cid:107) V ( θ ) − V ∗ (cid:107)
Sk dt < 0 whenever
[Equation]
As a consequence, as long as κ ( ∇ V ( θ ) ∇ V ( θ ) T ) is ﬁnite there exists k sufﬁciently large to guarantee convergence to V ∗ regardless of initial conditions.
Proof.
To extend Theorem 3 to this result, we will show that
[Equation]
First, since R k = 12 (( γP T ) k D µ − D µ ( γP ) k ) we have R k 1 = 0 since D µ is the stationary distribution so that ( P T )
k D µ 1 =
D µ P k 1 = µ .
So we only need to consider V ⊥ 1 .
Now, we have
[Equation]
where λ 2 is the eigenvalue with second largest magnitude (since 1 is the largest eigenvalue, but it has eigenvector 1 for P and µ for P T ).
And we have
[Equation]
V (cid:107) by the Gershgorin circle theorem since P k deﬁnes a distribution with stationary distribution D µ so that subtracting the off diagonal from the diagonal of A k we get λ min ( A k ) ≥ µ
i − (cid:80) j γ k D µ P k ij = µ i − γ k µ i for all i .
Since an analogous argument applies to the columns of A k , we get the above bound on the eigenvalues of S k .
Dividing the bounds we get the ﬁrst result.
Then by Theorem 3, we get the second result that d (cid:107) V ( θ ) − V ∗ (cid:107)
Sk dt < 0 when the condition holds.
Moreover, the global convergence follows for some sufﬁciently large k since the effective reversibility coefﬁcient is increasing exponentially with k .
D G ENERALIZED DIVERGENCE CONSTRUCTION
Proposition 1.
If the MRP is not reversible such that A
= D µ ( I − γP ) has at least one non-real eigenvalue, then there exists a function approximator V such that TD learning will diverge.
That is, for any initial parameters θ 0 , as t → ∞ we have (cid:107) V ( θ ) − V ∗ (cid:107) → ∞ .
Moreover, ∇ V ( θ ) can have rank up to n − 1 , where n is the number of states, for all θ .
Proof.
Since A is real, having one non-real eigenvalue implies that A has a pair of complex eigenvalues a + bi, a − bi corresponding to eigenvectors V 1 , V 2 .
We know that a > 0 since A is a non-singular M-matrix and that b (cid:54) = 0 by assumption.
Deﬁne U
= Re ( V 1 ) and U 2 =
Im ( V 1 ) and
U =
( U
U 2 ) .
Note that U 1 , U 2 are linearly independent since V 1 , V 2 are so that U is full rank and
that AU
= U (cid:18) a b − b a (cid:19) .
To construct a divergent approximator, deﬁne
[Equation]
so that Q is a matrix of rank 2 with range equal to E := span { U 1 , U 2 } .
For any V ∈ E so that V
= U (cid:18) xy (cid:19) we have (cid:107) V (cid:107) 2 ≤ C ( x 2 + y 2 ) with C > 0 .
Then
[Equation]
C
Thus, Q T A is a matrix of rank 2 with eigenvectors V 1 , V 2 and eigenvalues ( a 2 + b 2 ) i, ( a 2 + b 2 ) i .
Choosing V 0 in the span of V 1 , V 2 we deﬁne for θ ∈ R :
[Equation]
Then for V ∗ = 0 , we aply the above to (7) to get
[Equation]
as long as (cid:15) <
( a 2 + b 2 ) /C
and V ( θ ) (cid:54) = 0 .
This gives us divergence since as θ → ∞ we have (cid:107) V ( θ ) (cid:107) → ∞ .
Now we just need to show that we can make ∇ V ( θ ) rank n − 1 .
To do this we will note that we can deﬁne any function ¯ V (¯ θ ) from R d → E ⊥ for any number of parameters and we still get divergence if we use the function approximator V (cid:48)
( θ, ¯ θ )
=
V ( θ )+ ¯ V
(¯ θ ) .
This is because ∂V (cid:48)
( θ, ¯ θ ) ∂θ =
( (cid:15)I + Q ) V ( θ ) ⊥
A ¯ V (¯ θ ) so that ˙ θ remains strictly above 0 as before, implying divergence.
With this construction, we can build a divergent approximator V (cid:48) on n states with ∇ V (cid:48)
( θ, ¯ θ ) of rank n −everywhere.
For example, set d = n − 2 and let ¯ V (¯ θ ) be the identity function.
E N EURAL NETWORKS ARE HOMOGENEOUS
Lemma 1.
(Liang et al., 2017)
Let σ : R → R be h -homogeneous.
Then deﬁne a homogeneous network f : R d → R n by
[Equation]
where θ i is a d i − 1 ×
d
i +1 matrix
and Φ is the m × p matrix of data points and σ is applied componentwise.
Then for all 1 ≤ i ≤ L
[Equation]
and
[Equation]
where vec maps the matrices θ i to vectors while θ ∈ R
d is already a vector.
Proof.
We will consider n = 1 and note that the result applies for arbitrary n by applying the result to each component function of f .
Deﬁne g
i
θ 1 , . .
, θ i )
= σ ( · · · σ ( σ (Φ θ 1 ) θ 2 ) · · · θ i − 1 )
θ i , which we will denote simply g i so that we have f = σ ( g L ) .
Note that g i is a d i +1 dimensional vector and d L +1 = 1 .
Deﬁne f
i ( θ 1 , . . .
, θ i − 1 )
= σ ( · · · σ ( σ (Φ θ 1 ) θ 2 ) · · · θ i − 1 ) , and denote it
f i so that f i
=
σ ( g i ) .
Let f 0 be Φ .
Now we proceed by induction on the gap j − i for j ≥ i from 1 to L − 1 .
As the base case, consider i =
j .
Then we have that ∂f ia
[Equation]
Make the inductive assumption that for j − i = k ≥
we have for a ≤ d i + ∂f ja
[Equation]
= h
[Equation]
f ja
Now consider j − i
= k
+ 1 .
We have
[Equation]
Taking j =
L gives us the ﬁrst result that f ( θ ) =
h L − i +1 ∂f ( θ ) ∂
vec ( θ i ) vec ( θ i ) .
Summing over the L choices of i gives f ( θ ) =
∇ f ( θ ) θ (cid:80)
Li
=1 h L − i +1 /L .