O N THE P ERFORMANCE OF T EMPORAL D IFFERENCE L EARNING W ITH N EURAL N ETWORKS
A BSTRACT
Neural Temporal Difference (TD)
Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation.
Analysis of Neural TD Learning has proven to be challenging.
In this paper we provide a convergence analysis of Neural TD Learning with a projection onto
B ( θ 0 , ω ) , a ball of fixed radius ω around the initial point θ 0 .
We show an approximation bound of O ( ϵ + 1 / √ m ) where ϵ is the approximation quality of the best neural network in B ( θ 0 , ω ) and m is the width of all hidden layers in the network
I NTRODUCTION
Temporal difference (TD) learning is considered to be a major milestone of reinforcement learning (RL).
Proposed by Sutton (1988), TD Learning uses the Bellman error, which is a difference between an agent’s predictions in a Markov Decision Process (MDP) and what it actually observes, to drive the process of learning an estimate of the value of every state.
To deal with large state-spaces, TD learning with linear function approximation was introduced in Tesauro (1995).
A mathematical analysis was given in Tsitsiklis & Van Roy (1996), which shows the process converges under appropriate assumptions on step-size and sampling procedure.
However, with nonlinear function approximation, TD Learning is not guaranteed to converge, as observed in Tsitsiklis & Van Roy (1996) (see also Achiam et al.
for a more recent treatment).
Nevertheless, TD with neural network approximation, referred to as Neural TD, is used in practice despite the lack of strong theoretical guarantees.
To our knowledge, rigorous analysis was only addressed in the three papers Cai et al.
; Xu & Gu (2020); Cayci et al. (2021).
In Cai et al.
(2019), a single hidden layer neural architecture was considered along with projection on a ball around the initial condition; approximate convergence was proved to be an approximate stationary point of a certain function related to the linearization around the initial point.
This result was generalized to multiple hidden layers in Xu & Gu (2020), but this generalization required projection on a ball of radius of ω ∼ m − 1 / 2 around the initial point, where m is the width of the hidden layers.
Because the radius of this projection goes to zero with m , this effectively fixes the neural network to a small distance from its initial condition.
Both Cai et al.
; Xu & Gu (2020) additionally required certain regularity conditions on the policy.
Finally, Cayci et al.
(2021) gave a convergence result for a single hidden layer, also with a projection onto a radius of ω ∼ m − 1 / 2 around the initial point, but with the final objective being the representation error of neural approximation without any kind of linearization.
This result also required a condition on the representability of the value function of the policy in terms of features from random initialization.
In this paper, we analyze Neural TD with a projection onto B ( θ 0 , ω ) , a ball of fixed radius ω around the initial point θ 0 .
We show an approximation bound of O ( ϵ +1 / √ m ) where ϵ is the approximation quality of the best neural network in B ( θ 0 , ω ) .
Our result improves on previous work because it does not require taking the radius ω to decay with m , does not make any regularity or representability assumptions on the policy, applies to any number of hidden layers, and bounds the error associated with the neural approximation without any kind of linearization around the initial condition.
The main technical difference between our paper and previous works is the choice of norm for analysis.
We will describe this at a more technical level in the main body of the paper, but we use a norm introduced by Ollivier (2018), which is a convex combination of the usual l 2 -norm weighted by the stationary distribution of the policy with the so-called Dirichlet semi-norm.
The later has previously been used in the convergence analysis of Markov chains (Diaconis & SaloffCoste (1996);
Levin & Peres (2017)).
It was shown in Ollivier (2018) that Neural TD is exactly gradient descent on this convex combination of norms if the underlying policy is reversible.
In the case where the policy is not reversible, these results were partially generalized in Liu &
Olshevsky (2021), where it was shown that TD Learning with linear function approximation can be viewed as a so-called gradient splitting , a process which is analogous to gradient descent.
We build heavily on that interpretation here.
Our main technical argument is that Neural TD approximates the gradient splitting process at each step so that despite the nonlinearity of the approximation, an improvement in approximation quality can be guaranteed unless the system is already close to the best possible approximation over the projection radius.
Notably, our arguments do not imply that the neural network stays close to its initialization and our empirical simulations show a significant benefit from taking the projection radius ω not to decay with the width m .
P RELIMINARIES 2. M ARKOV D ECISION P ROCESSES
In this section, we present key concepts from MDPs, mostly to introduce our notation.
A finite discounted-reward MDP can be described by a tuple ( S, A, P env , r, γ ) , where S = { s 1 , s 2 , . . .
, s n } is a finite state-space whose elements are vectors; A is a finite action space;
P env =
( P env ( s ′ | s, a ))
s,s ′ ∈ S,a ∈ A is the transition probability matrix, where P env ( s ′ | s, a ) is the probability of transitioning from s to s ′ after taking action a ; r : S ×
A → R is the reward function; and γ ∈ (0 , 1) is the discount factor.
A policy π in an MDP is a mapping π : S ×
A → [0 , 1] such that (cid:80) a ∈
A π ( s, a ) =
for all s ∈ S , where π ( s, a ) is the probability that the agent takes action a in state s .
We will use n for the number of states, i.e., | S | = n .
Given a policy π , we define the corresponding transition probability matrix P =
( P ( s ′ | s )) s,s ′ ∈ S as
[Equation]
We also define the state reward function as
[Equation]
Although P and r ( s ) depend on the policy π , throughout this paper the policy will be fixed and hence we will suppress this dependence in the notation.
The stationary distribution µ corresponding to the policy π is defined to be a nonnegative vector with coordinates summing to one and satisfying µ T = µ T P .
The Perron-Frobenius theorem guarantees that such a µ exists and is unique subject to some conditions on P , e.g., aperiodicity and irreducibility (Gantmacher (1964)).
We use µ ( s ) to denote each entry of µ .
The value function of the policy π is defined as:
[Equation]
where E s,a ∼ π stands for the expectation when the starting state is s and actions are taken according to policy π , and s t is the t ’th state encountered.
Note that this quantity depends on π , but once again we suppress this dependence because the policy will be fixed throughout this paper.
Moreover, note that, despite the star superscript, V ∗ is not the optimal value function but rather the true value function corresponding to policy π .
It is well known that this value function satisfies the Bellman equation,
[Equation]
where V ∗ is a vector whose i ’th element is V ∗ ( s i ) and R is a vector whose i ’th element is r ( s i ) .
We will further assume that rewards are bounded in [ − r max , r max ] as in the following assumption.
Assumption 2.1.
For any s, a ∈ S × A , we have | r ( s, a ) | ≤ r max .
This immediately implies that
[Equation]
There are two standard sampling models where policy evaluation methods are usually considered.
The simplest model involves i.i.d.
sampling of s t at each step from stationary distribution µ .
Alternatively, in the Markov model the states s t are collected from a single path of Markov chain transitioning according to P .
It is still assumed in this case that the initial distribution is µ , so that the distribution of each s t is still µ , with the difference that the successive states are now not independent.
This can always be approximately satisfied by generating a sufficiently long path from
P and ignoring the initial states.
Assuming that the underlying Markov chain mixes with a geometric rate is common in many analyses, like Bhandari et al.
; Liu & Olshevsky (2021).
We will also make this assumption.
Formally, let P t denote the matrix P raised to the t ’th power, P ts, : be the row of P t corresponding to state s , and || · || TV the total variation distance.
Assumption 2.2.
There exists constant C > 0 and β ∈ (0 , 1) such that
[Equation]
This assumption guarantees ”mixing”: no matter the initial distribution, the state will get closer and closer to the stationary distribution µ as t increases.
As pointed out in Levin & Peres (2017), this assumption always holds when the Markov chain is irreducible and aperiodic.
Another useful quantity is the mixing time, τ mix ( ϵ mix ) , defined as the smallest integer t such that
[Equation]
Note that Assumption 2.2 implies
[Equation]
For simplicity, we will use τ mix without specifying its dependence on ϵ mix throughout the paper.
2. D - NORM AND D IRICHLET N ORM IN MDP S
We now introduce the so-called D -norm || · || D and the Dirichlet semi-norm || · || Dir associated with a policy.
While the former has long been used for the analysis of temporal difference learning dating back to Tsitsiklis & Van Roy (1996), the latter has, to our knowledge, been introduced in the context of RL relatively recently in Ollivier (2018).
Let D = diag ( µ ( s )) be the diagonal matrix whose elements are given by the entries of the stationary distribution µ .
Given a function f : S → R , its D -norm is defined as
[Equation]
The D -norm is similar to the Euclidean norm except each entry is weighted proportionally to the stationary distribution.
We also define the Dirichlet semi-norm of f :
[Equation]
A semi-norm satisfies the triangle inequality and absolute homogeneity, as any norm, but it is not a norm as it may be equal to zero at a non-zero vector.
Note that
||
f
|| Dir depends on the policy both through the stationary distribution µ ( s ) as well as through the transition matrix P .
Finally, following Ollivier (2018), the weighted combination of the D -norm and the Dirichlet seminorm is denoted as N ( f ) and defined as
[Equation]
Note that (cid:112) N ( f ) is a valid norm: since N ( f ) is quadratic, we can write N ( f ) =
f T Nf for some symmetric matrix N ; examining the first term in the definition of N ( f ) we see that N ⪰ (1 − γ )diag( π 1 , . . .
, π n ) ≻ 0 by irreducibility and aperiodicity.
2. N EURAL N ETWORK B ASED A PPROXIMATION
In this section we closely follow the notation from the previous works in Cai et al.
Allen-Zhu et al.
Liu et al.
(2020); Ollivier (2018) on neural approximations.
We define a multi-layer fully connected neural network by the following recursion:
[Equation]
where σ is an activation function and the input is state of the MDP: x (0) ∈ S .
Next, we define
[Equation]
to be the output with no activation function on the output.
We assume that each entry of θ ( k ) are initialized from N (0 , 1) and each entry of the vector b satisfies | b r | ≤ 1 , ∀ r .
We further assume that all the hidden layers have the same width which we denote by m , i.e., all the matrices θ ( k ) have first dimension of m .
Note that the total number of layers in the neural network is denoted by K .
We will stack up the weights of different layers into a column vector θ consisting of the entries of the matrices θ (1) , . . .
, θ ( K ) , with norm defined by
[Equation]
where || · || F is the Frobenius norm.
During the training process, only the weights θ will be updated while the weights b will be left to their initial value.
This particular definition of a neural network, as well as the decision to leave b fixed, is used by many papers on both TD Learning (i.e., Xu & Gu (2020)) and Deep Neural Network (i.e., Liu et al.
(2020)) analysis.
Although there is not an explicit bias term above, this definition does allow each layer to have different bias.
This can be reached by setting the last entry of x (0) to be 1 and last row of each θ ( k ) to be (0 , . . .
We can view this neural network as mapping the parameters θ to a vector with as many entries as the number of states.
Specifically, we can define the vector V ( θ ) whose i ’th entry is
[Equation]
Note that n in the above equation denotes the number of states in the MDP, i.e., | S | .
We remark that this vector will never be actually used in the execution of any algorithms we discuss due to its large size, but its use is still useful conceptually.
The Jacobian of V ( θ ) is then the matrix
[Equation]
where, abusing standard notation, ∇ θ V ( s, θ ) are defined to be row vectors.
A standard assumption made to simplify the analysis is the following.
Assumption 2.3.
Suppose for any i ∈ { 1 , 2 , . . . , n } ,
|| s i || ≤ 1 where || · || stands for the l 2 -norm.
Assumption 2.3 can always be satisfied by scaling, as we typically have control of how we choose to represent the states of the MDP.
It is also a common assumption that appears in many previous works (e.g., Cai et al.
Allen-Zhu et al.
A function f is called L -Lipschitz if
[Equation]
And a differentiable function f : R → R is c 0 -smooth
if
[Equation]
where f ′ stands for the derivative of f .
We find it helpful to make the following assumption:
Assumption 2.4.
The activation function σ is l -Lipschitz and c 0 -smooth.
The smoothness condition implies that our results below are not directly applicable to popular functions like ReLU.
However, many activation functions are twice differentiable (e.g., sigmoid, tanh, arctan, softplus) and one could always use a smooth approximation to a ReLU activation (e.g., GeLU or ELU).
We do need the smoothness of the input-to-output map as we will use the result from Liu et al. (2020), which claims the neural network is O ( 1 √ m ) -smoothness with respect to its parameters. The following assumption is also required to implement their result: Assumption 2.5.
For any k ∈ { 1 , 2 , . . . , K } , given i ∈ { 1 , 2 , . . , m } , there exists some constant c ( k ) > 0 , such that | x ( k ) i | = ˜ O (1) at initialization.
Here, x ( k ) i means the i ’th entry of x ( k ) .
2. N EURAL TD
In this section, we introduce (projected) neural TD learning.
At each time step t , this algorithm samples state s from the stationary distribution from either of two sampling models discussed earlier, generates the next state s ′ in the MDP, and computes the temporal difference error, defined as
[Equation]
Defining g
( θ t ) as
[Equation]
projected neural TD updates the weights as
[Equation]
where the projection is onto a ball of radius ω around the initial condition:
[Equation]
Projection is a common tool in neural TD and neural Q-learning to try to stabilize the iterates, since divergence can occur (Achiam et al.
; Van Hasselt et al.
Most analyses of TD learning proceed by comparing the evolution of TD to the mean-path update, defined as
[Equation]
where ¯
[Equation]
It is convenient to rewrite ¯ g ( θ t ) in terms of the difference between V ( θ t ) and V ∗ , which can be done by subtracting Eq.(1) with the result being
[Equation]
For simplicity [Equation] for the rest of paper we will define Θ to be the set onto which we are projecting:
[Equation]
Finally [Equation] we will say that ˆ V ∗ is an ϵ -approximation to the true value function V ∗ if
[Equation]
We will denote ˆ θ ∗ to be the point where the function V ( ˆ θ ∗ )
= ˆ V ∗ is the ϵ -approximation of V ∗ .
O UR MAIN R ESULT
We can now state the main contribution of this paper, which is a performance result for Neural TD.
We will require an assumption to the effect that the initialization is not too large.
Assumption 3.1.
For all k ∈ { 1 , 2 , . . .
, K } , || θ ( k )
|| ≤ O ( √ m ) .
This is a common assumption: theoretical justification can be found in Liu et al.
In particular, this assumption holds with high probability for a random Gaussian initialization if, for example, K (the depth) grows slower than exponentially in m (the width).
Now we are ready to state our result.
Theorem 3.1.
Suppose Assumption 2.1, 2.2, 2.3, 2.4, 2.5, 3.1 hold, θ t is generated by projected
Neural TD with the constant step-size α t = α , and C, β come from the Assumption 2.2.
(a) Under i.i.d.
sampling, we have
[Equation]
In particular, if α = T − 1 / 2 and ϵ ≤ 1 , we have
[Equation]
(b) Under Markov sampling, we have
[Equation]
In particular, if α = T − 1 / 2 and ϵ ≤ 1 , we have
[Equation]
In all O ( · ) notations above, we treat factors that do not depend on T, ϵ, m, α, β, θ 0 as constants.
As mentioned earlier, the key distinguishing feature of this theorem is the choice of norm.
The left-hand side of all the equations measures the difference between V ( θ t ) and the best possible V (ˆ θ ∗ ) within B ( θ 0 , ω ) by taking the N ( · ) norm.
We note that since, trivially, ||
f
|| 2 D ≤ N ( f ) / (1 − γ ) , where γ is the discount factor, one can simply replace the left-hand sides of all the equations by ||
f
|| 2 D
to obtain results that look more similar to the previous literature on TD, which usually proceeds based on an analysis in the D -norm (e.g., Tsitsiklis & Van Roy (1996)).
As is common in analyses of SGD, the performance measure is the average of performance measures from 1 to T .
If a particular θ t ′ is sought that satisfies the bounds obtained, a standard trick is to choose t ′ to be uniform from 1 , . . .
, T .
In that case, the expectation E [ N ( V ( θ t ′ ) − V (ˆ θ ∗ )] is exactly the left-hand side of all the bounds in the theorem.
Looking at the right-hand sides of all the equations, the theorem guarantees a final error of O ( ϵ + 1 / √ m ) , with a convergence rate that scales as ˜ O
(1 / √ T ) .
The difference between the two cases is that that the Markov sampling case contains an additional term containing the mixing time.
As a result of this extra term, the convergence time in the Markov case is worse by a factor of O (log √ T ) .
[Figure]
We now revisit the discussion of the novelty of this paper.
First, in contrast to previous work, we do not assume the projection radius ω has to decay with m , nor do we restrict our analysis to a single hidden layer case.
In our proof, the projection radius appears as a constant in the O ( · ) notation, which is why we need to assume it is a constant.
Second, the left-hand side of the equations is a measure of the error V ( θ t ) −
V (ˆ θ ∗ ) , which can be thought of the difference between the error of the neural network and the best possible error in B ( θ 0 , ω ) .
Moreover, as already discussed, the left-hand side of the equations above is greater than the quantity
[Equation]
which is a natural measure of the average error.
The point is that the network is not being linearized around the initial condition in any sense.
Finally, no additional assumptions on the policy are being made here; in particular, no assumptions that the policy is regular or that it is representable are necessary.
As outlined in the Introduction, out of the four potentially undesirable elements discussed here (small projection radius, linearization around the initial condition, assumptions on policy, restriction to single layer case) all previous papers suffered from at least three.
D ISTINGUISHING FEATURE OF OUR ANALYSIS
The main technical difference between our work and the previous papers is the use of the function
N ( · ) to measure the approximation error.
Here, we follow Ollivier (2018); Liu & Olshevsky (2021) which explained why this function is the “right” function to analyze policy evaluation.
In Ollivier (2018) it was shown that if the matrix P corresponds to a reversible Markov chain, then
E [¯ g ( θ t )] = ∇ θ N ( f ) for some f .
This makes neural TD very easy to analyze, as it can be viewed as gradient descent.
Unfortunately, in practice policies are almost never reversible.
In Liu & Olshevsky (2021), it was shown how to further use the function N ( · ) to analyze TD with linear approximation when the policy is not necessarily reversible.
The key idea was the notion of a gradient splitting: a linear function h ( θ ) is said to be a gradient splitting of a convex quadratic f ( θ ) minimized at θ
= a if
[Equation]
[Figure]
[Figure]
In other words, h ( θ ) has exactly the same inner product with the “direction to the optimal solution” as the true gradient of f ( θ ) (up to the factor of 1 / 2) .
The significance of this is that it allows many analyses of gradient descent to be modified to analyze TD with linear approximation, since the key step in analyses of gradient descent is usually to argue that the left-hand side of Eq.
(9) is negative, signifying that gradient descent “makes progress” towards the optimal solution.
In Liu &
Olshevsky (2021) it was shown that for TD with linear approximation, ¯ g ( θ ) is exactly the gradient splitting of N ( f ) .
We build on that idea in this paper as follows.
We use recent NTK-style bounds from Liu et al.
(2020) to argue that with increasing width, the neural approximation gets more linear.
For finite m , we can then modify existing techniques for analyzing gradient descent with errors in the gradient evaluations to analyze the neural TD update, which we view as gradient splitting with error in the evaluations.
It should be stressed that the most interesting “regime” to which we expect these results to be applicable is when m is only moderately large.
Intuitively, when m is too large, the network will be very close to linear, and the benefit over taking random linear features will be small.
In terms of our bounds, in this case we would expect the ϵ term in Theorem 3.1 to be large.
On the other hand, when m is too small, the error bound scaling with width of m − 1 / 2 will not be very attractive.
On the other hand, our results can be applied to the “middle range” – say m ∼ 100 – where, depending on the context of the problem, an ∼ m − 1 / 2 error is acceptable.
Informally, in this regime the network will be sufficiently linear to converge well, but not so linear to lose approximation quality.
Our simulations confirm this, as we verify good approximations for networks of approximately this width on several open AI benchmarks.
S IMULATIONS
It might be objected that many analysis of neural network training in the large-width regime proceed by arguing that the neural network stays around its initial point (e.g., Chizat et al.
; Telgarsky (2020)).
If so, then the part of our results dealing with avoiding a projection radius that goes to zero with m would not make much of a difference in practice.
Here, we show empirically that, setting the projection radius ω to be a constant rather than ω ∼ m − 1 / 2 does make a substantial difference.
We produce simulations on Open AI Gym tasks: Mountaincar, Cartpole and Acrobot.
In each task, a trained policy is used to sample data to train a fully connected neural network for policy evaluation using Projected Neural TD learning.
The policy in Mountaincar and Cartpole is trained by Proximal Policy Optimization (PPO) (Schulman et al.
while in Acrobot it is trained by
Deep Q-Learning (DQN) (Mnih et al.
Both algorithms are implemented through the Stable
Baselines package Raffin et al.
[Figure]
nonlinearity of the neural network. We run a total of ( in Acrobot task) steps on each task, computing the y value during the training process every 20 ( 50 in Acrobot task) step.
Different widths m are chosen for each task.
In each figure, the x -axis plots time steps while the y -axis plots the averaged Bellman error, the distance to initialization (given by || θ t − θ 0 || ω ), and the difference between gradients (given by ||∇
θ V ( s t , θ t ) − ∇ θ
V ( s t , θ 0 ) || ), respectively.
Each subtitle of each figure suggests which task is performed on and how many hidden layers are used.
It is clear from the figures that the networks we use are nonlinear.
It is also clear that networks with a decaying projection radius are significantly outperformed by networks with a constant projection radius.
C ONCLUSIONS
We have provided an analysis of Projected Neural TD for policy evaluation.
We have shown that if the projection set is a ball of constant radius B ( θ 0 , ω ) around the initial point θ 0 , the final approximation error is O ( ϵ + 1 / √ m ) , where ϵ is the approximation quality of the best neural network in
B ( θ 0 , ω ) and m is the width of all hidden layers in the network.
Our result improves on previous works because it does not require taking the radius ω to decay with m , does not make any regularity or representability assumptions on the policy, applies to any number of hidden layers, and bounds the error associated with the neural approximation without any kind of linearization around the initial condition.
We conjecture that unprojected neural TD converges to the optimal solution with high probability provided it begins within a radius of O ( √ m ) away from a point which exactly describes the true value function.
This conjecture is proved for single hidden layer networks.
Finally, we have demonstrated empirically that these changes can make a substantial difference in performance.
A P ROOF FOR T HE M AIN R ESULT A. U SEFUL LEMMAS
Before we go into details
[Equation]
we first introduce the following lemmas.
Recall we have defined N ( f ) as [Equation]
The first lemma implies an important property of N ( · )
[Equation]
Lemma A.1. For any function f defined on the state space S , the following equation holds:
[Equation]
Proof.
We can perform the following sequence of manipulations:
[Equation]
where the first equation uses Eq.(5) and the forth equation uses Eq
[Equation] [Equation]
Thus,
[Equation]
The following lemmas are some variations of the mean-value theorem.
In this lemma and below, we adopt the notation that gradients are row vectors.
Lemma A.2.
(a). Let h : R → R be any differentiable function.
For any x, y ∈ R , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
(b). Let ξ : R a → R be any differentiable function.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
(c). Let f : R a → R b be any differentiable function and e ∈ R b be any vector.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx [Equation] (1 − λ )
y such that
[Equation]
where f ′ ( z ) is the Jacobian at z .
Proof.
(a).
This is a direct result of the well-known mean value theorem (Theorem 5.10 in Rudin (1976)).
(b). Define h : R → R such that h ( w ) =
ξ ( x + w ( y − x )) .
Using the above fact, for any u, v ∈ R , there exists λ ∈ (0 , 1) such that
[Equation]
By letting v = 1 and u = 0 , ξ ( y
[Equation]
− ξ ( x
[Equation]
= ξ ′
[Equation]
λx +
[Equation]
1 − λ
[Equation]
y
[Equation] ( y − x ) . (c). Define ξ : R a → R such that ξ = e T f .
Using the above fact, for any vectors x, y ∈ R a ,there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
Notice that in this case, ξ ( y ) − ξ ( x ) =
e T ( f ( y ) − f ( x )) and ξ ′ ( z ) =
e T f ′ ( z ) .
So for any vectors x, y , there exists z = λx + (1 − λ )
y such that
[Equation]
which is exactly what needs to be proved.
Lemma A.3.
Let f : R a → R b be any differentiable function.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
Proof.
Let us take e = f ( y ) − f ( x ) in Lemma A.2. We thus have [Equation] We now apply Cauchy-Schwarz inequality on the right hand side to obtain,
[Equation]
and finally using the definition of matrix norm,
[Equation] The next lemma shows how the mean-value theorem can help to analyze Neural TD Learning [Equation] Lemma A.4.
The Projected Neural TD Learning with a mean-path update can be rewritten as
[Equation]
with ¯ g 1 ( θ t ) , ¯ g 2 ( θ t ) , ¯ g 3 ( θ t ) defined as follows:
[Equation]
where λ ∈ [0 , 1] is a scalar and θ mid
= λθ t + (1 − λ )
ˆ θ ∗ is a vector such that
[Equation]
Proof.
By Eq.
[Equation]
t )
[Equation]
θ
[Equation]
Now let D ( γP − I )
( V ( θ t ) − V
(ˆ θ ∗ )) be the vector e in Lemma A.2.
There exists a scalar λ ∈ (0 , 1) and a vector θ mid1 =
λθ t + (1 − λ )
ˆ θ ∗ such that
[Equation]
This gives the reason to divide Eq.(14) into two parts as follows:
[Equation]
The following lemma builds the relationship between x T Dy and expectation.
Lemma A.5.
Let x, y be two vectors and x ( i ) , y ( i ) denote their i ’th entries, respectively.
Let ¯ x, ¯ y be two scalars such that | x ( i ) |
≤ ¯ x and | y
( i ) | ≤ ¯ y hold for all i .
The following results hold:
[Equation]
Proof.
We expand x T Dy , x T DPy and y T DPx as follows:
[Equation]
The following lemmas implies two important properties of neural network approximation:
Lipschitzness and smoothness.
Lemma A.6.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
Proof.
[Equation]
where the second inequality is by projection and the last inequality uses Assumption 3.1 and the fact ω is constant to m .
Lemma A.7.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
k )
[Equation]
O ( √ m
[Equation]
Proof.
From Assumption 2.3, || x (0) || ≤ 1 .
By Lemma A.6,
[Equation]
By induction, suppose || x ( k ) ||
2 ≤ O ( m ) .
By Lemma A.6,
[Equation]
Lemma A.8.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
x
[Equation]
O
[Equation]
[Equation]
Proof.
[Equation]
which implies
[Equation]
where Σ ′ is a diagonal matrix with Σ ′
( i, i ) =
σ ′ ( (cid:80) j θ ( k ) ( i, j ) x
( k − 1) ( j )) .
The last inequality holds because of Lemma A.6.
Lemma A.9.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
Here, ∇ θ ( k ) x
( k ) is defined to be a matrix whose ( i, ( j − 1)
m
+ h ) ’
th entry ∇ θ ( k ) x
( k ) ( i, j, h ) is given by
[Equation]
Proof.
[Equation]
We can write this as
[Equation]
which implies
[Equation]
The last inequality holds because of Lemma A.7.
Lemma A.10.
For all s ∈ S and θ , ||∇ θ V ( s, θ ) || ≤ O (1) .
with respect to m .
Proof.
Since each entry of b satisfies | b r | ≤ 1 , it is easy to see that
[Equation]
By Lemma A.8, Lemma A.9, and the chain rule,
[Equation]
k ) x
[Equation]
k +1)
[Equation]
θ
[Equation]
[Equation]
[Equation]
≤ O
[Equation]
[Equation]
It follows:
[Equation]
Lemma A.11.
The Hessian matrix, which is ∇ 2 θ V ( s, θ ) , has a norm of O ( m − 0 . 5 ) .
In other words,
[Equation]
|∇
[Equation]
This is a direct result of Theorem 3.2 in Liu et al.
Notice that since we assume Assumption 3.1 and2.5 (which correspond to Lemma G.1 and Lemma G.4 in Liu et al.
(2020)) holds with probability 1 , Lemma A.11 also holds with probability 1 .
The following lemma implies that the update during each step will not be too large.
Lemma A.12. We have the following bound for || g ( θ t )
|| :
[Equation]
Proof.
Recall that Eq.
(1) tell us the property V ∗ satisfies
[Equation]
and it can be rewritten as
[Equation]
Moreover
[Equation]
recall that g ( θ t ) is defined to be
[Equation]
This immediately implies that g ( θ t ) is actually a random variable and implicitly relies on the state s .
This allows g ( θ t ) to be rewritten as
[Equation]
where, for simplicity, we denote ˆ f ( s )
=
V ( s, ˆ θ ∗ ) − V
( s, θ t ) .
Now, || g ( θ t )
|| 2 can be bounded as
[Equation]
There are five different terms, and now we will bound them respectively.
By Lemma A.10 which says ˆ f ( s ) is O (1) -Lipschitz and the fact that || θ t − ˆ θ ∗ || ≤ ω
[Equation]
ˆ f
[Equation]
Using the above fact,
[Equation]
Using Eq.(8) , we can derive
[Equation]
Using the above fact,
[Equation]
Using Eq.(2) and Jensen’s inequality
[Equation]
Combine the above five facts, and we have the bound for ||
g ( θ t ) || 2 : [Equation] where the last inequality uses Lemma A.10.
Since 0 ≤ γ ≤ [Equation] we can simplify the bound as
[-|| 
 = ||-][Equation]
A. P ROOF FOR T HE M AIN R ESULT
[-− ˆ θ ∗ ||≤ || θ t − ˆ θ ∗ + α t g ( θ t ) ||= || θ t − ˆ θ ∗ || 
 + 2 α t ( θ t − ˆ θ ∗ ) T g ( θ t ) + α 2 t-]Proof.
Consider Projected Neural TD Learning, it is easy to see ||[-g ( θ t ) ||= || θ t − ˆ θ ∗ || 
 + 2 α t (-] θ t
[Equation]
First, we consider 2 α t ( θ t − ˆ θ ∗ )
T ¯ g ( θ t ) .
Lemma A.4 allows us to divide it into several parts and bound them respectively as follows:
[Equation]
To bound ( θ t − ˆ θ ∗ ) T ¯ g 1 ( θ
[Equation]
where the first equality is by Eq.(10), the second equality is by Eq.
(13), and the third equality is by setting f = V ( θ t ) − V (ˆ θ ∗ ) in Lemma A.1.
To bound ( θ t − ˆ θ ∗ ) T ¯ g 2 ( θ t ) , we first notice that Lemma A.11 means V ( s, θ ) is O ( m − 0 . 5 ) -smoothness with respect to θ . Hence,
[Equation]
, where the inequality is by Lemma A.11 and the equality is because both θ t and θ midare in B ( θ 0 , ω ) . Similarly, Lemma A.10 tells us that V ( s, θ ) is O (1) -Lipschitz.
This means
[Equation]
where the inequality is by Lemma A.10 and the equality is because both θ t and ˆ θ ∗ lie in B ( θ 0 .ω ) .
These two facts imply that each entry of ( ∇ θ V ( θ t ) − ∇ θ V ( θ mid))
( θ t − ˆ θ ∗ ) is upper-bounded by
O ( m − 0 . 5 ) and each entry of V ( θ t ) − V
(ˆ θ ∗ ) is upper-bounded by O (1) .
With this fact,
[Equation]
where the equality is by Eq.(11) and the first inequality is by setting x to be ( ∇ θ V ( θ t ) − ∇ θ V ( θ mid1 ))
( θ t − ˆ θ ∗ ) , y to be V ( θ t ) − V
(ˆ θ ∗ ) in Lemma A.5.
To bound ( θ t − ˆ θ ∗ ) T ¯ g 3 ( θ t [Equation]
[Equation]
where the equality is by Eq.(12) and the first inequality is by setting x to be ∇ θ V ( θ t )( θ t − ˆ θ ∗ ) , y to be ˆ V ∗ − V ∗ in Lemma A.5, as each entry of ∇ θ V ( θ t )( θ t − ˆ θ ∗ ) is bounded by O (1) using Lemma A.10 and each entry of ˆ V ∗ − V ∗ is bounded by ϵ using Assumption 8.
Combining the above facts,
[Equation]
which is the bound of the first part in Eq.(16).
Second, we consider α 2 t
|| g
( θ t )
|| 2 .
By Lemma A.12,
[Equation]
The above facts are all we need to establish the result when each s t is sampled i.i
[Equation]
d. from µ .
In summary,
[Equation]
which just simply combines the bound for 2 α t ( θ t − ˆ θ ∗ )
T ¯ g ( θ t ) and α 2 t
|| g
( θ t )
|| 2 we obtained earlier.
Given θ t , taking expectation on both sides of Eq.(16):
[Equation]
where the equality uses the condition that given s t are sampled i.i.d.
from µ , which will lead to E [ g ( θ t ) − ¯ g
( θ t )]
= 0 .
From now on we fix α t = α , and this leads to E [ || θ
[Equation]
Telescoping a sum from 1 to T and dividing both sides by T , we establish the result in i.i.d. case.
To continue with the non i.i.d.
case, we need to consider 2 α ( θ t − ˆ θ ∗ ) T ( g ( θ t ) − ¯ g ( θ t )) in Eq.(16) which is no longer 0 .
We will use τ mix , the mixing time defined in Eq.(3), to split it into two terms and bound them separately.
This idea is from Sun et al.
Our first step is to divide it into two terms:
[Equation]
[Equation]
To bound 2 α ( θ t − θ t − τ mix )( g ( θ t ) − ¯ g ( θ t )) [Equation]
notice that Lemma A.12 implies || g ( θ t ) || ≤ O ( ϵ ) + 11 − γ O (1) .
As a consequence of Eq.(6), we have
[Equation]
Further [Equation] the same upper bound that g ( θ t ) has holds for ¯ g ( θ t ) since ¯ g ( θ t ) =
E [ g ( θ t )] .
These observations imply that
[Equation]
To bound 2 α ( θ t − τ mix − ˆ θ ∗ )( g ( θ t ) − ¯ g ( θ t )) , observe that, conditioned on θ t − τ mix , the quantity θ t − [Equation] − ˆ [Equation]
∗ is deterministic, and E [ g ( θ t ) − ¯ g
( θ t ) | θ t − τ mix ] can be bounded by
[Equation]
The first inequality follows because, conditional on s t − τ mix , the random variable s t has the distribution of one row of P τ mix .
Thus, the second term in Eq.(17) can be bounded as
[Equation]
Thus, coming back to Eq.(17), we obtain that
[Equation]
Now let ϵ mix =
α , by the definition of τ mix , τ mix = log αC log β .
Using the fact that log x ≤ x − 1 , ∀ x > 0 , we derive
[Equation]
where the first inequality is because log β ≤ β − 1 < 0 and log αC ≤ 0 .
The bound can be rewritten as
[Equation]
α [Equation] Now we are ready to consider Eq.(16).
Taking expectation on both sides,
[Equation]
Rewrite this inequality as
[Equation]
Telescoping a sum from 1 to T and dividing both sides by T , we establish the result in non the i.i.d. case.
B A NALYSIS OF U NPROJECTED TD L EARNING – P ROOF OF OUR C ONJECTURE FOR THE S INGLE H IDDEN L AYER C ASE B. R ESULT WITHOUT P ROJECTION
Here we prove that when the distance from the optimal solution is O ( √ m ) , unprojected TD learning converges with high probability.
We begin by standardizing notation.
Recall g ( θ t ) is defined by
[Equation]
where
[Equation]
Without projection, there is only one step in the algorithm, which is
[Equation]
We call this algorithm Non Projected Neural TD Learning.
Similarly to the way we proceeded earlier in this paper, Non Projected Neural TD Learning with mean-path update is given by
[Equation]
We will need to make the following assumption on exact approximation, which is stronger than what we assumed in the projected case.
Assumption B.1.
There exists some θ ∗ such that V ( θ ∗ )
=
V ∗ .
Now, for simplicity of notations, we also introduce the following notation.
Remember the smallest eigenvalue λ min of some matrix A is defined as
[Equation]
A similar way can be applied to define the smallest 2 , D eigenvalue σ 2 ,D min of a matrix A :
[Equation]
For simplicity, we will use σ 2 ,D min to denote the 2 , D eigenvalue of ∇ θ V ( θ ∗ ) .
We will always suppose each s t is sampled i.i.d.
from µ .
We now introduce the following result.
Theorem B.1.
In the Non Projected Neural TD learning using a one hidden layer neural network, smooth and Lipschitz activation function, suppose each s t are sampled i.i.d from µ , and stepsize is chosen to be α t = 1 λ ( t +1) .
Further, let A be the event A
[Equation]
sup
[Equation]
and assume
E
[Equation]
Then, A happens with probability at least 1 − δ and the sequence (cid:8) E (cid:2)
||
θ t − θ ∗ || 2 | A (cid:3)(cid:9) converges to 0 .
B. U SEFUL L EMMAS
Lemma B.1.
( θ t − θ ∗ ) T ¯ g ( θ t ) can be rewritten as
[Equation]
where ¯ g 1 ( θ t ) , ¯ g 2 ( θ t ) , ¯ g 3 ( θ t ) are defined as follows:
[Equation]
[Equation] Here, λ 2 ∈ [0 , 1] is a scalar and θ mid2 =
λ 2 θ t + (1 − λ 2 ) θ ∗ is a vector such that
[Equation]
Proof.
First
[Equation] we divide ( θ t − θ ∗ ) T ¯ g ( θ t ) into two parts,
[Equation]
where the first equality is by the definition of ¯ g ( θ t ) in Eq.(7).
Now let ( θ t − θ ∗ ) T ∇ θ V ( θ ∗ ) T D ( γP − I ) be the vector e in Lemma A.2.
There exists a scalar λ 2 ∈ [0 , 1] and a vector θ mid2 =
λ 2 θ t + (1 − λ 2 ) θ ∗ such that
[Equation]
Using this fact [Equation] ( θ t − θ ∗ ) T ∇ θ V ( θ ∗ ) T D ( γP − I )
( V ( θ t ) − V ( θ ∗ )) can be divided by
[Equation]
Lemma B.2.
If the activation function is l -Lipschitz and c 0 -smooth, for any s , θ 1 and θ 2 , the inequalities
[Equation]
hold where c 1 is a scalar that is independent of m and θ .
Proof.
Using the definition of neural network in Section 2.
[Equation]
one hidden layer neural network would be simplified as [Equation] It is easy to see that
[Equation]
Suppose the activation function σ is c [-
 || x − y || . This means ||∇ θ-]0 -smooth.
That is
[Equation]
for any x and y ,
[Equation]
This means
[Equation]
which proves the first part of the lemma.
Now let us move on to the second part of the lemma.
By
Lipschitzness,
[Equation]
Now we finish the second part of the lemma.
Lemma B.3.
The following inequality holds:
[Equation]
Proof.
To begin with, the definition of N is
[Equation]
Using the definition of σ 2 ,D min ,
[Equation]
which establishes the result.
Lemma B.4.
If a non negative sequence { X t } satisfies
[Equation]
for some b, c > 0 , then the sequence { X t } converges to 0 .
Proof.
Recursively applying the relation between X t +1 and X t , we can derive the following:
[Equation]
The first term in Eq.(21) definitely goes to 0 as t goes to infinity.
Now let us consider the term (cid:81) tj = i +1 (1 − cj ) .
A logarithm usually helps us to convert it into a sum, so we perform the following manipulations:
[Equation]
where the first inequality uses the fact ln(1 + x ) ≤ x and the second inequality uses (cid:80) tj = i +1 1 j ≥
ln
( t + 1) − ln
( i + 1) .
So,
[Equation]
By setting i = 0 , this means the third term in Eq.(21) goes to 0 as t goes to infinity.
Now consider (cid:80) t −i =1 bi 2 (cid:81) tj
= i +1 (1 − cj ) , we obtain
[Equation]
where the second inequality uses the fact ( i +1) 2 i≤ 4 for all positive integer i , and the third inequality combine the two facts (cid:80) t −
[Equation]
+ [Equation]
≤
[Equation]
di
[Equation]
+ [Equation] c − 2 di given c ≥ 2 and
[Equation]
[-2 ( i + 1) c − 2 di given c ≥ 2 and (cid:80) t − 1 i =
-]=
[Equation] i + 1) [Equation]
≤ (cid:82) [Equation] [Equation] di ≤
[Equation]
1 ( i + 1) c − 2 di given c ≤ 2 . If c ̸ = 1 , then
[Equation]
If c = 1 , then
[Equation]
Under both cases, we can easily argue that the second term in Eq.
(21) goes to 0 .
Now, we have proved that all three terms in the right hand side of Eq.(21) go to 0 as t goes to infinity.
This directly implies { X t } converges to 0 .
Lemma B.5.
(Optional Stopping Theorem) Suppose { X t } is a super martingale and T is a stopping time.
If there exists a constant c such that | X τ ∧ T | ≤ c holds for all τ , then we have
[Equation]
This is also called Doob’s Optional Stopping Theorem.
See Theorem 10.10 of Williams (1991).
B. P ROOF OF L EMMA B.
Consider Non Projected Neural TD Learning,
[Equation]
Given θ t , the only randomness is from s t .
Taking expectation on both side and using the fact that E [ g ( θ t )] = ¯ g ( θ t ) (since we assume s t are sampled i.i.d. from µ ), we obtain
[Equation]
First, we consider 2 α t ( θ t − θ ∗ )
T ¯ g ( θ t ) .
Lemma B.1 allows us to divide it into several parts and thus we can bound them respectively:
[Equation]
g 3 ( θ t )) . To bound ¯ g 1 ( θ t ) ,
[Equation]
where the first equality is because the definition of ¯ g 1 ( θ t ) in Eq.(18), the second equality is by setting f = ∇ θ V ( θ ∗ )( θ t − θ ∗ ) in Lemma A.1, and the inequality is by Lemma B.3.
To bound ¯ g 2 ( θ t ) ,
[Equation]
where the equality is because the definition of ¯ g 2 ( θ t ) in Eq.(19), the first inequality is by triangle inequality which says | x + y | ≤ | x | + | y | , and the second inequality is by setting x to be ( ∇ θ V ( θ t ) − ∇ θ V ( θ ∗ ))
( θ t − θ ∗ ) , y to be V ( θ t ) − V
(ˆ θ ∗ ) in Lemma A.5 with each entry of x , y bounded by c 0 m − 0 .
5 ||
θ t − θ ∗ || 2 , l || θ t − θ ∗ || respectively using Lemma B.2.
To bound ¯ g 3 ( θ t ) ,
[Equation]
where the equality is because of the definition of ¯ g 3 ( θ t ) in Eq.(20), the first inequality is because of the triangle inequality, and the second inequality is by setting x to be ( ∇ θ V ( θ mid2 ) −∇ θ V ( θ ∗ ))
( θ t − θ ∗ ) , y to be ∇ θ V ( θ ∗ )( θ t − θ ∗ ) in Lemma A.5 with each entry of x , y bounded by c 0 m − 0 . 5 || θ t − θ ∗ || 2 , l || θ t − θ ∗ || respectively using Lemma B.2.
Combine the above facts and we now have the bound for the second term in the right hand side of Eq.(22),
[Equation]
Second, we consider E [ α 2 t
|| g
( θ t ) ||
2 | θ t ] in Eq.(22).
For simplicity, define f ( s ) =
V ( s, θ ∗ ) − V
( s, θ t ) .
Since we are using one hidden layer neural network to approximate V ( s, θ ) ,
[Equation]
Recall that V ( s, θ ∗ ) satisfies Eq.(1), which is
[Equation]
and g ( θ t ) is defined to be g ( θ t ) =
∇ θ V ( s, θ t )
[ r ( s ) + γV ( s ′ , θ t ) − V ( s, θ t )] .
This latter immediately implies that g ( θ t ) is actually a random variable and implicitly relies on the state s .
Using these facts
[Equation]
we obtain [Equation]
By Eq.(2) we can obtain the following quick result:
[Equation]
And this leads to the following:
[Equation]
[Equation] where the first inequality uses the fact that ( a + b + c ) 2 ≤ 3( a 2 + b 2 + c 2 ) , the second inequality is by Jensen’s inequality [Equation] and the third inequality simply uses Lemma B.2 to bound ||∇ θ V ( s, θ t ) || .
Further,
[Equation]
Now let us go back to Eq.(22)
[Equation]
where the second and third terms on the right hand side can be bounded by
[Equation]
This means
[Equation]
[Equation]
For simplicity, we define the following notations:
[Equation]
(while d 0 is defined to be d 0 = [-
 α t (1 − γ )( σ-]2 ) . We can rewrite Eq.(24) as
[Equation]
while the condition E (cid:2) || θ 0 − θ ∗
[Equation] | 2 (cid:3) < ( 2(
[Equation]
( σ [Equation] ) 2 λ − [Equation]
l [Equation] [Equation]
+ γ [Equation] 2 64(1+ γ ) 2 c 20 l 2 λmδ − 2 is just [Equation]
Under such notations, an important fact is that [Equation] This can be showed easily since
[Equation]
If we define the stopping time T = inf t { X t ≥ a 0 2 b 0 } and the sequence { Y t } to be
[Equation]
Now we first claim that the sequence { Y t } is a super martingale. We show this as follows: Suppose T = 0 , and we find that Y t
=
Y 0 , ∀ t .
In this trivial case, obviously, { Y t [-such that Y 
-]} is a super martingale. Now we assume T > 0 .
When t = 0 , we have
[Equation]
which means E [ Y 1 | Y 0 ] ≤ Y 0 .
For t ≤ T , by the definition of T we know that X t ≤ a 0 2 b 0 , which implies that X t ≤ a t 2 b t (this is because by the definition of a t and b t , h ( t ) =
a t 2 b t is increasing with t ).
Hence,
[Equation]
which means E [ Y t +1 | Y t ] ≤ Y t . For t > [-1536(
 + γ 2 ) γ-]T , by the definition of { Y t } we know that Y t =
Y T .
Hence, combine all the above facts and we conclude the sequence { Y t } is a super martingale.
Next, we claim that X t < a 0 2 b 0 , ∀ t holds with probability at least 1 − δ .
This can be shown as follows: Let A be the event (cid:110) sup t X
[Equation]
<
[Equation] , X 1 < a [Equation] b 0 , · · · (cid:111) .
To compute P ( A ) , notice that
[Equation]
We can easily check that T is also a stopping time for { Y t } .
In order to use Lemma B.5, we need to check the conditions that | Y T | ≤ c for some constant c .
We split it into two cases.
First, if τ < T , then | Y τ ∧ T | = | Y τ | .
Because of the definition of T we know that
[Equation]
where we use the fact that { d t } is a decreasing sequence and d 0 = 2 .
Second, if τ ≥ T , then | Y τ ∧ T | = | Y T | .
Recall that update of the algorithm is
[Equation]
which implies [Equation] Now we let t = T − 1 and use X t notations.
The above fact can be rewritten as
[Equation]
Because of the definition of T , we know that | X T − 1 | ≤ a 0 2 b 0 .
Moreover, Eq.(23) give us a bound for ||
g
( θ T − 1 ) || , which is || g
{+( θ T − 1 ) || ≤ (cid:113) 3(1 + γ 2 ) l 4 X 2 T − 1 + 
+}
γ 2 l 2 r 2max (1 − γ ) 2 .
Finally, α T − 1 is set to be 1 λ ( t +1)
so it is obviously bounded. All these facts lead to the result that | Y T | is bounded. Combining the above two cases, we are now eligible to use Lemma B.5. Hence, we obtain
[Equation]
On the other hand, E [ Y τ ∧ T ] can be expanded as [Equation]
[Equation]
Combining these two facts,
[Equation]
where the second inequality using the fact that Y t ≥ a 20 4 b 20 + d T .
Hence,
[Equation]
where the equality is because E [ Y 0 ] = E [ X 20 ]+ d 0 , the second inequality is because d t is nonnegtive, and the third inequality is because of the condition E [ X 20 ] ≤ a 20 4 b 20 δ − d 0 .
Next, it is easy to see
[Equation]
This result holds for all τ , so we can let τ → ∞ and obtain
P ( A ) ≥ 1 − δ.
Actually, using h ( t ) = a t 2 b t is monotonically increase with t again, event A implies X t ≤ a t 2 b t , ∀ t .
With this fact,
[Equation]
Plugging in
[-= 
 α t (1 − γ )( σ 2 ,D min ) 2 − 3-][Equation] and α [Equation]
, we can derive [Equation] Given that X t ≤ a
[Equation]
min [Equation] [Equation] Hence
[Equation]
we conclude
[Equation]
Take expectation on both sides and we derive
[Equation]
By Lemma B.4, we conclude that the sequence (cid:8) E (cid:2)
X 2 t | A (cid:3)(cid:9) converges to 0 .