O N THE P ERFORMANCE OF T EMPORAL D IFFERENCE L EARNING W ITH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural Temporal Difference (TD)
Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation.
Analysis of Neural TD Learning has proven to be challenging.
In this paper we provide a convergence analysis of Neural TD Learning with a projection onto
B ( θ 0 , ω ) , a ball of fixed radius ω around the initial point θ 0 .
We show an approximation bound of O ( ϵ + 1 / √ m ) where ϵ is the approximation quality of the best neural network in B ( θ 0 , ω ) and m is the width of all hidden layers in the network
I NTRODUCTION
Temporal difference (TD) learning is considered to be a major milestone of reinforcement learning (RL).
Proposed by Sutton (1988), TD Learning uses the Bellman error, which is a difference between an agent’s predictions in a Markov Decision Process (MDP) and what it actually observes, to drive the process of learning an estimate of the value of every state.
To deal with large state-spaces, TD learning with linear function approximation was introduced in (Tesauro (1995)).
A mathematical analysis was given in (Tsitsiklis & Van Roy (1996)), which shows the process converges under appropriate assumptions on step-size and sampling procedure.
However, with nonlinear function approximation, TD Learning is not guaranteed to converge, as observed in Tsitsiklis & Van Roy (1996) (see also (Achiam et al.
for a more recent treatment).
Nevertheless, TD with neural network approximation, referred to as Neural TD, is used in practice despite the lack of strong theoretical guarantees.
To our knowledge, rigorous analysis of the convergence of Neural TD was only addressed in the three papers Cai et al.
; Xu & Gu (2020);
Cayci et al.
In Cai et al.
(2019), a single hidden layer neural architecture was considered along with projection on a ball around the initial condition; approximate convergence was proved to an approximate stationary point of a certain function related to the linearization around the initial point.
This result was generalized to multiple hidden layers in Xu & Gu (2020), but this generalization required projection on a ball of radius of ω ∼ m − 1 / 2 around the initial point, where m is the width of the hidden layers.
Because the radius of of this projection goes to zero with m , this effectively fixes the neural network to a small distance from its initial condition.
Both Cai et al.
; Xu & Gu (2020) additionally required certain regularity conditions on the policy.
Finally, Cayci et al.
(2021) gave a convergence result for a single hidden layer, also with a projection onto a radius of ω ∼ m − 1 / 2 around the initial point, but with the final objective being the representation error of neural approximation without any kind of linearization.
This result also required a condition on the representability of the value function of the policy in terms of features from a random initialization.
In this paper, we analyze Neural TD with a projection onto B ( θ 0 , ω ) , a ball of fixed radius ω around the initial point θ 0 .
We show an approximation bound of O ( ϵ +1 / √ m ) where ϵ is the approximation quality of the best neural network in B ( θ 0 , ω ) .
Our result improves on previous work because it does not require taking the radius ω to decay with m , does not make any regularity or representability assumptions on the policy, applies to any number of hidden layers, and bounds the error associated with the neural approximation without any kind of linearization around the initial condition.
The main technical difference between our paper and previous works is the choice of norm for analysis.
We will describe this at a more technical level in main body of the paper, but we use a norm introduced by Ollivier (2018), which is a convex combination of the usual l 2 -norm weighted by the stationary distribution of the policy with the so-called Dirichlet seminorm.
The Dirichlet seminorm has previously been used in the convergence analysis of Markov chains (Diaconis & Saloff-Coste (1996);
Levin & Peres (2017)).
Remarkably, it was shown in Ollivier (2018) that neural TD is exactly gradient descent on this convex combination of norms if the underlying policy is reversible.
In the case where the policy is not reversible, these results were partially generalized in Liu &
Olshevsky (2021), where it was shown that TD with linear function approximation can be viewed as a so-called gradient splitting , a process which is analogous to gradient descent.
We build heavily on that interpretation here.
Our main technical argument is that neural TD approximates the gradient splitting process at each step, so that despite the nonlinearity of the approximation, an improvement in approximation quality can be guaranteed unless the system is already close to the best possible approximation over the projection radius.
Notably, our arguments do not imply that the neural network stays close to its initialization and our empirical simulations show a significant benefit from taking the projection radius ω not to decay with the width m .
P RELIMINARIES 2. M ARKOV D ECISION P ROCESSES
In this section, we present key concepts from MDPs, mostly to introduce our notation.
A finite discounted-reward MDP can be described by a tuple ( S, A, P env , r, γ ) , where S = { s 1 , s 2 , . . .
, s n } is a finite state-space whose elements are vectors; A is a finite action space;
P env =
( P env ( s ′ | s, a ))
s,s ′ ∈ S,a ∈ A is the transition probability matrix, where P env ( s ′ | s, a ) is the probability of transitioning from s to s ′ after taking action a ; r : S ×
A → R is the reward function; and γ ∈ (0 , 1) is the discount factor.
A policy π in an MDP is a mapping π : S ×
A → [0 , 1] such that (cid:80) a ∈
A π ( s, a ) =
for all s ∈ S , where π ( s, a ) is the probability that the agent takes action a in state s . We will use n for the number of states, i.e., | S | = n .
Given a policy π , we define the corresponding transition probability matrix P =
( P ( s ′ | s )) s,s ′ ∈ S where
[Equation]
We also define the state reward function as
[Equation]
P and r ( s ) naturally depend on the policy π but throughout this paper the policy will be fixed and hence we will suppress this dependence in the notation.
The stationary distribution µ corresponding to the policy π is defined to be a non negative vector with coordinates summing to one and satisfying µ T = µ T P .
The Perron-Frobenius theorem guarantees that such a µ exists and is unique subject to some conditions on P , e.g., aperiodicity and irreducibility (Gantmacher (1964)).
We use µ ( s ) to denote each entry of µ .
The value function of the policy π is defined as:
[Equation]
where E s,a ∼ π stands for the expectation when the starting state is s and actions are taken according to policy π , and s t is the t ’th state encountered.
Note that this quantity depends on π , but once again we suppress this dependence because the policy will be fixed throughout this paper.
Moreover, note that, despite the star superscript, V ∗ is not the optimal value function but rather the true value function corresponding to policy π .
This value function satisfies satisfies the Bellman equation,
[Equation]
where V ∗ is a vector whose i ’th element is V ∗ ( s i ) and R is a vector whose i ’th element R ( s i )
=
r ( s i ) corresponds to state s i .
Throughout the paper, we will assume the following:
Assumption 2.1.
Suppose for any s, a ∈ S × A , the function | r ( s, a ) | ≤ r max .
This immediately implies that
[Equation]
2. M ARKOV C HAIN N OISE M ODEL
There are two standard sampling models.
The simplest model involves i.i.d.
sampling of the states s t at each step from the stationary distribution µ .
Alternatively, in the Markov model the states s t are collected from a single path of Markov chain transitioning according to P .
It is still assumed in this case that the initial distribution is µ , so that the distribution of each s t is still µ (with the difference that the successive states are now correlated).
This can always be approximately satisfied by generating a sufficiently long path from
P and ignoring the initial states.
Assuming that the underlying Markov chain mixes with a geometric rate is common in many analyses, like Bhandari et al.
; Liu & Olshevsky (2021).
We will make the following assumption.
Let P t denote the matrix P raised to the t -th power, P ts, : be the row of P t corresponding to state s , and || · || TV the total variation distance.
Assumption 2.2.
There exists constant C > 0 and β ∈ (0 , 1) such that
[Equation]
This assumption guarantees “mixing”: no matter the initial distribution, the state will get closer and closer to the stationary distribution µ as t increases.
As pointed out in Levin & Peres (2017), this assumption always holds when the Markov chain is irreducible and aperiodic.
Another useful quantity is the mixing time, τ mix ( ϵ mix ) , defined as the smallest integer t such that
[Equation]
Note that Assumption 2.2 implies
[Equation]
For simplicity, we will use τ mix without specifying its dependence on ϵ mix throughout the paper.
2. D - NORM AND D IRICHLET N ORM IN MDP S
We now introduce the so-called D -norm || · || D and the Dirichlet semi-norm || · || Dir associated with a policy.
While the former has long been used for analysis of temporal difference learning dating back to Tsitsiklis & Van Roy (1996), the latter has, to our knowledge, been introduced in the context of RL relatively recently in Ollivier (2018).
Let D = diag ( µ ( s )) be the diagonal matrix whose elements are given by the entries of the stationary distribution µ .
Given a function f : S → R , its D -norm is defined as
[Equation]
The D -norm is similar to the Euclidean norm except each entry is weighted proportionally to the stationary distribution.
We also define the Dirichlet semi-norm of f :
[Equation]
A semi-norm satisfies the triangle inequality and absolute homogeneity, as any norm, but it is not a norm as it may be equal to zero at a non-zero vector.
Note that ||
f
|| Dir depends on the policy both through the stationary distribution µ ( s ) as well as through the transition matrix P .
Finally, following Ollivier (2018), the weighted combination of the D -norm and the Dirichlet seminorm is denoted as N ( f ) and defined as
[Equation]
Note that (cid:112) N ( f ) is a valid norm: since N ( f ) is quadratic, we can write N ( f ) = f T Nf for some symmetric matrix N ; examining the first term in the definition of N ( f ) we see that N ⪰ (1 − γ )diag( π 1 , . . . , π n ) ≻ 0 .
2. N EURAL N ETWORK B ASED A PPROXIMATION
In this section we closely follow the notation from the previous works in Cai et al.
Allen-Zhu et al.
Liu et al.
(2020); Ollivier (2018) on neural approximations in reinforcement learning.
We define a multi-layer fully connected neural network by the following recursion:
[Equation]
where σ is an activation function and the input is one of the states of the MDP: x (0)
= s ∈ S .
We then define
[Equation]
to be the output with no activation function on the output.
Each entry of b satisfies | b r | ≤ 1 , ∀ r .
We assume that all the hidden layers have the same width which we denote by m , i.e., all the matrices θ ( k ) have first dimension of m .
Note that the total number of layers in the neural network is denoted by K .
We will stack up the weights of different layers into a column vector θ consisting of the entries of the matrices θ (1) , . . .
, θ ( K ) , with norm defined by
[Equation]
where || · || F is the Frobenius norm.
During the training process, only the weights θ will be updated while the final weights b will be left to their initial value.
This particular definition of a neural network, as well as the decision to leave b fixed, is used by many papers on both TD Learning (i.e., Xu & Gu (2020)) and Deep Neural Network (i.e., Liu et al. (2020)) analysis. Although there is not an explicit bias term above, this definition does allow each layer to have different bias.
This can be reached by setting the last entry of x (0) to be 1 and last row of each θ ( k ) to be (0 , ..., 0 , 1) .
We can view this neural network as mapping the parameters θ to a vector with as many entries as the number of states.
Specifically, we can define the vector V ( θ ) whose i ’th entry is
[Equation]
Note that n in the above equation denotes the number of states in the MDP, i.e., | S | .
The Jacobian of V ( θ ) is then the matrix
[Equation]
where, abusing standard notation, ∇ θ V ( s, θ ) are defined to be row vectors.
A standard assumption made to simplify the analysis is the following.
Assumption 2.3.
Suppose for any i ∈ { 1 , 2 , . . . , n } ,
|| s i || ≤ 1 where || · || stands for the l 2 -norm.
Assumption 2.3 can always be satisfied by scaling, as we typically have control of how we choose to represent the states of the MDP.
It is also a common assumption which appears in many previous works (e.g., Cai et al.
Allen-Zhu et al.
We will assume the activation function σ satisfies the following properties. A function f is called L -Lipschitz if [Equation]
And a differentiable function f : R → R is c 0 -smooth if
[Equation]
where f ′ stands for the derivative of f .
We find it is helpful to make the following assumption:
Assumption 2.4.
The activation function σ is l -Lipschitz and twice differentiable.
The twice differentiability condition implies that our results below are not directly applicable to popular functions like ReLU.
However, many activation functions are twice differentiable (e.g., sigmoid, tanh, arctan, softplus) and one could always use a smooth approximation to a ReLU activation (e.g., GeLU or ELU).
In what follows, we do need the smoothness of the input-to-output map, so the assumption of twice differentiability cannot be dropped.
It is, however, possible to use non-smooth activation functions such as ReLU if we additionally assume that the state under the stationary distribution has a smooth distribution, similarly to Cai et al.
2. N EURAL TD
In this section, we introduce (projected) neural TD learning.
At each time step t , this algorithm samples a state s from the stationary distribution from either of two sampling models discussed earlier, generates the next state s ′ in the MDP, and computes the temporal difference error, defined as
[Equation]
Defining g
( θ t ) as
[Equation]
neural TD updates the weights as:
[Equation]
where the projection is onto a ball of radius ω around the initial condition:
[Equation]
Projection is a common tool in neural TD and neural Q-learning to try to stabilize the iterates, since divergence can occur Achiam et al.
; Van Hasselt et al.
Most analyses of TD learning proceed by comparing the evolution to the mean-path update, defined as
[Equation]
where
[Equation]
It is convenient to rewrite ¯ g ( θ t ) in terms of the difference between V ( θ t ) and V ∗ , which can be done by subtracting Eq. (1) with the result being
[Equation]
For simplicity, for the rest of paper we will define Θ to be the set onto which we are projecting:
[Equation]
Finally, we will say that V ( ˆ θ ∗ ) or ˆ V ∗ is an ϵ -approximation to V ∗ if
[Equation] [Equation]
Using Eq. (2), this definition immediately implies that
[Equation]
O UR MAIN R ESULT
We can now state the main contribution of this paper, which is a performance result for Neural TD.
We will require an assumption to the effect that the initialization is not too large.
Recall that m is the width of each layer.
Assumption 3.1.
Assume
that || θ ( k )
|| ≤ O ( √ m ) holds for all k ∈ { 1 , 2 , . .
. , K } .
This is a common assumption; theoretical justification can be found in Liu et al.
In particular, this assumption holds with high probability for a random Gaussian initialization if, for example, K (the depth) grows slower than exponentially in m (the width).
Now we are ready to state our result.
Theorem [Equation]
Suppose Assumption 2.1, 2.2, 2.3, 2.4, 3.1 hold [Equation] and θ t is generated by projected Neural TD with the constant step-size α t = α .
We further assume there exists some ˆ θ ∗ ∈ B ( θ 0 , ω ) which is an ϵ -approximation to V ∗ .
(a) Under i.i
[Equation]
d. sampling, we have
[Equation]
In particular, if α = T − 1 / 2 and ϵ ≤ 1 , we have
[Equation]
(b) Under Markov sampling, we have
[Equation]
ϵ
[Equation] α [Equation] where C, β come from the Assumption 2.2 on the mixing time.
In particular, if α = T − 1 / 2 and ϵ ≤ 1 , we have
[Equation]
As mentioned earlier, the key distinguishing feature of this theorem is the choice of norm.
The lefthand side of all the equations measures the difference between V ( θ t ) and the best possible V (ˆ θ ∗ ) within B ( θ 0 , ω ) by taking the N ( · ) norm.
We note that since, trivially, || f
|| 2 D ≤ N ( f ) / (1 − γ ) , where γ is the discount factor, one can simply replace the left-hand sides of all the equations by ||
f
|| 2 D
to obtain results that look more similar to the previous literature on TD, which usually proceeds based on an analysis in the D -norm (e.g., Tsitsiklis & Van Roy (1996)).
As is common in analyses of SGD, the performance measure is the average of performance measures from 1 to T .
If a particular θ t ′ is sought that satisfies the bounds obtained, a standard trick is to choose t ′ to be uniform from 1 , . . .
, T .
In that case, E [ N ( V ( θ t ′ ) − V ( ˆ θ ∗ )] is exactly the left-hand side of all the bounds in the theorem.
Looking at the right-hand sides of all the equations, the theorem guarantees a final error of O ( ϵ + 1 / √ m ) , with a convergence rate that scales as ˜ O
(1 / √ T ) .
The difference between the two cases is that that the Markov sampling case contains an additional term containing the mixing time.
As a result of this extra term, the convergence time in the Markov case is worse by a factor of log √ T .
property of gradient splitting: h ( θ ) has the same inner product with a − θ as (1 / 2) ∇ f ( θ ) .
We now revisit the discussion of novelty of this paper.
First, in contrast to previous work, we do not assume the projection radius ω has to decay with m , nor do we restrict our analysis to a single hidden layer case.
In our proof, the projection radius appears as a constant in the O ( · ) notation, which is why we need to assume it is a constant.
Second, the left-hand side of the equations is a measure of the error V ( θ t ) − V
(ˆ θ ∗ ) , the difference between the error of the neural network and the best possible error in B ( θ 0 , ω ) .
Moreover, as already discussed, the left-hand side of the equations above is greater than the quantity
[Equation]
which is a natural measure of the average error.
The point is that the network is not being linearized around the initial condition in any sense.
Finally, no additional assumptions on the policy are being made here; in particular, no assumptions that the policy is regular or that it is representable are necessary.
As outlined in the Introduction, out of the four potentially undesirable elements discussed here (small projection radius, linearization around the initial condition, assumptions on policy, restriction to single layer case) all previous papers suffered from at least three.
D ISTINGUISHING FEATURE OF OUR ANALYSIS
[Figure] The main technical difference between our work and the previous papers is the use of the function N ( · ) to measure the approximation error.
Here, we follow Ollivier (2018);
Liu &
Olshevsky (2021) which explained why this function is the “right” function to analyze policy evaluation.
In Ollivier (2018) it was shown that if the matrix P corresponds to a reversible
Markov chain, then E
[¯ g ( θ t )] = ∇ θ N ( f ) .
This makes neural TD very easy to analyze, as it can be viewed as gradient descent.
Unfortunately, in practice policies obtained in reinforcement learning are almost never reversible.
In Liu & Olshevsky (2021), it was shown how to further use the function N ( · ) to analyze TD with linear approximation when the policy is not necessarily reversible.
The key idea was the notion of a gradient splitting: a linear function h ( θ ) is said to be a gradient splitting of a convex quadratic f ( θ ) minimized at θ
= a if
[Equation] [Figure]
In other words, h ( θ ) has exactly the same inner product with the “direction to the optimal solution” as the true gradient of f ( θ ) (up to the factor of 1 / 2) .
The significance of this is that it allows many analyses of gradient descent to be modified to analyze TD with linear approximation, since the key step in analyses of gradient descent is usually to argue that the left-hand side of Eq. (11) is negative, signifying that gradient descent “makes progress” towards the optimal solution.
In Liu & Olshevsky (2021) it was shown that for TD with linear approximation, ¯ g ( θ ) is exactly the gradient splitting of N ( f ) .
We build on that idea in this paper as follows.
We use recent NTK-style bounds from Liu et al.
(2020) to argue that with increasing width, the neural approximation gets more linear.
For finite m , we can then modify existing techniques for analyzing gradient descent with errors in the gradient evaluations to analyze the neural TD update, which we view as gradient splitting with error in the evaluations.
It should be stressed that the most interesting “regime” to which we expect these results to be applicable is when m is only moderately large.
Intuitively, when m is too large, the network will be very close to linear, and the benefit over taking random linear features will be small.
In terms of our bounds, we would expect the ϵ term in Theorem 3.1 to be large.
On the other hand, when m is too small, the error bound scaling with width of m − 1 / 2 will not be very attractive.
On the other hand, our results can be applied to the “middle range” – say m ∼ 100 – where, depending on the context of the problem, an ∼ m − 1 / 2 error is acceptable.
Informally, in this regime the network will be sufficiently linear to converge well, but not so linear to lose approximation quality.
Our simulations confirm this, as we verify good approximations for networks of approximately this width on several open AI benchmarks.
S IMULATIONS
It might be objected that many analysis of neural network training in the large-width regime proceed by arguing that the neural network stays around its initial point (e.g., Chizat et al.
; Telgarsky (2020)).
If so, then the part of our results dealing with avoiding a projection radius that goes to zero with m would not make much of a difference in practice.
In this section, we show empirically that, in fact, setting the projection radius ω to be a constant rather than ω ∼ m − 1 / 2 does make a substantial difference. [Figure]
We produce simulations on Open AI Gym tasks: Mountaincar,
Cartpole and
Acrobot. In each task, a trained policy is used to sample data to train a fully connected neural network for policy evaluation using Projected Neural TD learning.
The policy in Mountaincar and
Cartpole is trained by Proximal Policy Optimization (PPO) (Schulman et al.
(2017)) while in Acrobot it is trained by Deep Q-Learning (DQN) (Mnih et al.
Both algorithms are implemented through the Stable Baselines package Raffin et al.
We run a total of 2000 (5000 in Acrobot task) steps on each task, computing the y value during the training process every 20 ( 50 in Acrobot task) steps.
Different widths m are chosen for each task.
In each figure, the x -axis plots time steps while the y -axis plots the averaged Bellman error, the distance to initialization (given by || θ t − θ ∗ || ω ), and the difference between gradients (given by ||∇ θ V ( s t , θ t ) − ∇ θ V ( s t , θ 0 ) || ), respectively. Each subtitle of each figure suggests which task is performed on and how many hidden layers are used.
It is clear from the figures that the networks we use are nonlinear. It is also clear that networks with a decaying projection radius are significantly outperformed by networks with a constant projection radius.
C ONCLUSIONS
We have provided an analysis of Projected Neural TD for policy evaluation.
We have shown that if the projection set is a ball of constant radius B ( θ 0 , ω ) around the initial point θ 0 , the final approximation error is O ( ϵ + 1 / √ m ) , where ϵ is the approximation quality of the best neural network in
B ( θ 0 , ω ) and m is the width of all hidden layers in the network.
Our result improves on previous works because it does not require taking the radius ω to decay with m , does not make any regularity or representability assumptions on the policy, applies to any number of hidden layers, and bounds the error associated with the neural approximation without any kind of linearization around the initial condition.
We conjecture that unprojected neural TD converges to the optimal solution with high probability provided it begins within a radius of O ( √ m ) away from a point which exactly describes the true value function.
This conjecture is proved for single hidden layer networks.
Finally, we have demonstrated empirically that these changes can make a substantial difference in performance.
A P ROOF FOR T HE M AIN R ESULT A. U SEFUL LEMMAS
Before we go into details
[Equation]
we first introduce the following lemmas.
Lemma A.1.
Recall we have defined N ( f ) as N ( f ) =
(1 − γ ) ||
f
|| 2 D + γ
|| f
|| 2 Dir .
For any function f defined on the state space S , the following equation holds.
[Equation]
Proof.
We can perform the following sequence of manipulations:
[Equation]
where the first equation uses Eq. (5) and the forth equation uses Eq
[Equation] [Equation]
Thus,
[Equation]
Lemma A.2.
(a) Let h : R → R be any differentiable function.
For any x, y ∈ R , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
(b) Let ξ : R a → R be any differentiable function.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
(c) Let f : R a → R b be any differentiable function and e ∈ R b be any vector.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
Proof.
(a)
This is a directly result of the well-known mean value theorem (Theorem 5.10 in Rudin (1976)).
(b) Define h : R → R such that h ( w ) =
ξ ( x + w ( y − x )) .
Using the above fact, for any u, v ∈ R , there exists λ ∈ (0 , 1) such that
[Equation]
By letting v = 1 and u = 0 ,
[Equation]
(c) Define ξ : R a → R such that ξ = e T f .
Using the above fact, for any vectors x, y ∈ R a ,there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
Notice that in this case, ξ ( y ) − ξ ( x ) =
e T ( f ( y ) − f ( x )) and ξ ′ ( z ) =
e T f ′ ( z ) .
So for any vectors x, y , there exists z = λx + (1 − λ )
y such that
[Equation]
which is exactly what needs to be proved.
Lemma A.3.
Let f : R a → R b be any differentiable function.
For any x, y ∈ R a , there exists λ ∈ (0 , 1) and z = λx + (1 − λ )
y such that
[Equation]
Proof.
This is a direct result from Lemma A.2 by letting e = f ( y ) − f ( x ) , taking l 2 -norm on both sides, and using the Cauchy-Schwarz inequality on the right hand side.
Lemma A.4.
The Projected Neural TD Learning with a mean-path update can be rewritten as
[Equation]
where ¯ g 1 ( θ t ) , · · · , ¯ g 5 ( θ t ) are defined as follows:
[Equation]
where λ ∈ [0 , 1] is a scalar and θ mid
= λθ t + (1 − λ )
ˆ θ ∗ is a vector such that
[Equation]
Proof.
By Eq.
[Equation]
Now let D ( γP − I )
( V ( θ t ) − V
(ˆ θ ∗ )) be the vector e in Lemma A.2.
There exists a scalar λ ∈ (0 , 1) and a vector θ mid1 =
λθ t + (1 − λ )
ˆ θ ∗ such that
[Equation]
So the term ∇ θ V ( θ t ) T D ( γP − I )
( V ( θ t ) − V
(ˆ θ ∗ )) can also be divided into two parts,
[Equation]
Lemma A.5.
Let x, y be two vectors and x ( i ) , y ( i ) denote their i ’th entry, respectively.
Let ¯ x, ¯ y be two scalars such that | x ( i ) |
≤ ¯ x and | y
( i ) | ≤ ¯ y hold for all i .
The following results hold:
[Equation]
Proof.
We expand x T Dy , x T DPy and y T DPx as follows:
[Equation]
Lemma A.6.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
Proof.
[Equation]
where the last inequality uses Assumption 3.1.
Lemma A.7.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
Proof.
From Assumption 2.3, || x (0) || ≤ 1 .
By Lemma A.6,
[Equation]
By induction, suppose || x ( k ) ||
2 ≤ O ( m ) .
By Lemma A.6,
[Equation]
Lemma A.8.
For all k ∈ { 1 , 2 , . . . , K } , [Equation]
Proof.
[Equation]
which implies
[Equation]
where Σ ′ is a diagonal matrix with Σ ′
( i, i ) =
σ ′ ( (cid:80) j θ ( k ) ( i, j ) x
( k − 1) ( j )) .
The last inequality holds because of Lemma A.6.
Lemma A.9.
For all k ∈ { 1 , 2 , . . .
, K } ,
[Equation]
Here, ∇ θ ( k ) x
( k ) is defined to be a matrix whose ( i, ( j − 1)
m
+ h ) ’
th entry ∇ θ ( k ) x
( k ) ( i, j, h ) is given by
[Equation]
Proof.
[Equation]
which implies
[Equation]
The last inequality holds because of Lemma A.7.
Lemma A.10. For all s ∈ S and θ , ||∇ θ V ( s, θ ) || ≤ O (1) .
with respect to m . Proof. Since each entry of b satisfies | b r | ≤ 1 , it is easy to see that
[Equation] By Lemma A.8 and Lemma A.9, [Equation]
k ) [Equation] θ [Equation]
| ≤ O
[Equation]
It follows:
[Equation]
Lemma A.11.
The Hessian matrix, which is ∇ 2 θ V ( s, θ ) , has a norm of O ( m − 0 . 5 ) . In other words, ||∇
[Equation]
≤ O ( m − 0 .
[Equation]
This is a direct result from Theorem 3.1 in Liu et al. A. P ROOF FOR T HE M AIN R ESULT Proof. Consider Projected Neural TD Learning, it is easy to see || θ t +1 − ˆ θ ∗
{+|| 
 = ||+}
{+Proj ( θ t + α t g θ t )) − ˆ θ ∗ ||≤ || θ t − ˆ θ ∗ + α t g ( θ t ) ||= || θ t − ˆ θ ∗ || 
+}
+ 2 α t ( θ t − ˆ θ ∗ ) T g ( θ t ) + α 2 t || g
( θ t ) ||= || θ t − ˆ θ {+∗ || 
 + 2 α t+} ( θ t − ˆ θ ∗ ) T ¯ g ( θ t ) + α 2 t
|| g ( θ t ) || + 2 α t ( θ t − ˆ θ ∗ ) T ( g ( θ t ) − ¯ g ( θ t )) . First, we consider 2 α t ( θ t − ˆ θ ∗ )
T ¯ g ( θ t ) .
Lemma A.4 allows us to divide it into several parts and bound them respectively as follows:
[Equation]
To bound ( θ t − ˆ θ ∗ ) T ¯ g 1 ( θ t ) ,
[Equation]
where the first equality is by Eq. (12), the second equality is by Eq.
(15), and the third equality is by setting f = V ( θ t ) − V (ˆ θ ∗ ) in Lemma A.1. To bound ( θ t − ˆ θ ∗ ) T ¯ g 2 ( θ t ) , we first notice that
[Equation]
where the first inequality is by Lemma A.3 for some λ 2 ∈ [0 , 1] and θ mid2 = λ 2 θ t + (1 − λ 2 ) θ mid1 , and second inequality is by Lemma A.11 and the projection restriction. We also notice that
[Equation]
where the first inequality is by Lemma A.2 for some λ 3 ∈ [0 , 1] and θ mid3 = λ 3 θ t + (1 − λ 3 )ˆ θ ∗ , and the second inequality is by Lemma A.10 and the projection restriction. These two facts imply that each entry of ( ∇ θ V ( θ t ) − ∇ θ V ( θ mid1 ))
( θ t − ˆ θ ∗ ) is upper-bounded by O ( m − 0 . and each entry of V ( θ t ) − V (ˆ θ ∗ ) is upper-bounded by O (1) .
With this fact, [Equation] where the equality is by Eq. (13) and inequality is by setting x to be ( ∇ θ V ( θ t ) − ∇ θ V ( θ mid1 ))
( θ t − ˆ θ ∗ ) , y to be V ( θ t ) − V
(ˆ θ ∗ ) in Lemma A.5.
To bound ( θ t − ˆ θ ∗ ) T ¯ g 3 ( θ t ) ,
[Equation]
where the equality is by Eq.
(14) and inequality is by setting x to be ∇ θ V ( θ t )( θ t − ˆ θ ∗ ) , y to be ˆ V ∗ − V ∗ in Lemma A.5, as each entry of ∇ θ V ( θ t )( θ t − ˆ θ ∗ ) is bounded by O (1) using Lemma A. and each entry of ˆ V ∗ − V ∗ is bounded by ϵ using Assumption 9.
Combining the above facts,
[Equation] O ( α t ( m − 0 . [Equation]
ϵ )) , which is the bound of first part in Eq.16.
Second, we consider α 2 t || g ( θ t ) || 2 .
For simplicity, define ˆ f ( s )
=
V ( s, ˆ θ ∗ ) − V
( s, θ t ) . By Lemma A.10, || ˆ f
[Equation]
Using the above fact and Jensen’s inequality,
[Equation]
Using Eq.
(10) and Jensen’s inequality again,
[Equation]
On the other hand
[Equation]
Eq.
(1) can be rewritten as
[Equation]
where g ( θ t ) is actually a random variable and implicitly relied on the state s .
This is easy to see from the definition of g ( θ t ) [Equation] which is [Equation]
Hence
[Equation]
g ( θ t ) can be rewritten as
[Equation]
Now, || g ( θ t )
|| 2 can be bounded as
[Equation]
where the last inequality is by Lemma A.10.
Further,
[Equation]
The above facts are all we need to establish the result when each s t is sampled i
[Equation]
i
[Equation]
d. from µ .
In summary,
[Equation]
which just simply combines the bound for 2 α t ( θ t − ˆ θ ∗ )
T ¯ g ( θ t ) and α 2 t || g ( θ t ) || 2 we obtained earlier.
Given θ t , taking expectation on both sides of Eq.
[Equation]
where the equality uses the condition that given s t are sampled i
[Equation]
i
[Equation]
d. from µ , which will lead to E [ g ( θ t ) − ¯
g ( θ t )] = 0 . If α t = α are constant over t ,
[Equation]
Telescoping a sum from 1 to T and dividing both sides by T , we establish the result in i.i
[Equation]
d. case.
From now on, we fix α t = α .
To continue with the non i.i
[Equation]
d. case, we need to consider 2 α ( θ t − ˆ θ ∗ ) T ( g ( θ t ) − ¯ g ( θ t )) in Eq.
Our first step is to divide it into two terms:
[Equation]
where τ mix is defined in Eq.
We will bound each of these two terms separately.
To bound 2 α ( θ t − θ t − τ mix )( g ( θ t ) − ¯ g ( θ t )) , notice that Eq.
(18) implies || g ( θ t ) || ≤ O (1 + ϵ ) .
As a consequence of Eq.
(6), we have
[Equation]
Further, since || g ( θ t ) || ≤ O (1 + ϵ ) , the same upper bound holds for ¯ g ( θ t ) since ¯ g ( θ t ) =
E [ g ( θ t )] .
These observations imply that
[Equation]
To bound 2 α ( θ t − τ mix − ˆ θ ∗ )( g ( θ t ) − ¯ g ( θ t )) , observe that, conditioned on θ t − τ mix , the quantity θ t − τ mix − ˆ θ ∗ is deterministic, and E [ g ( θ t ) − ¯ g
( θ t ) | θ t − τ mix ] can be bounded by
[Equation]
The first inequality follows because, conditional on s t − τ mix , the random variable s t has the distribution of one row of P τ mix (and we have already mentioned that
|| g ( θ t ) − ¯ g ( θ t )
|| ≤ O (1 + ϵ ) ).
Thus, the second term in Eq.
(19) can be bounded as
[Equation]
Thus, coming back to Eq.
[Equation]
we obtain that
[Equation]
Now let ϵ mix =
α , by the definition of τ mix , τ mix = log αC log β .
Using the fact that log x ≤ x − 1 , ∀ x > 0 , we derive
[Equation]
where the first inequality is because log β ≤ β − 1 < 0 and log αC ≤ 0 .
The bound can be rewritten as
[Equation]
Now we are ready to consider Eq.
Taking expectation on both sides,
[Equation]
Rewrite this inequality as
[Equation]
Telescoping a sum from 1 to T and dividing both sides by T , we establish the result in non the i.i.d. case.
B A NALYSIS OF U NPROJECTED TD L EARNING – P ROOF OF OUR C ONJECTURE FOR THE S INGLE H IDDEN L AYER C ASE B. R ESULT WITHOUT P ROJECTION
Here we prove that when the distance from the optimal solution is O ( √ m ) , unpojected TD learning converges with high probability.
We beging by standardizing notation.
Recall g ( θ t ) is defined by
[Equation]
where δ t = r
( s )+ γV ( s ′ , θ t ) − V ( s, θ t ) .
Without projection, there is only one step in the algorithm, which is
[Equation]
We call this algorithm Non Projected Neural TD Learning.
Similarly to the way we proceeded earlier in this paper, Non Projected Neural TD Learning with mean-path update is given by
[Equation]
We will need to make the following assumption on exact approximation, which is stronger than what we assumed in the projected case.
Assumption B.1.
There exists some θ ∗ such that V ( θ ∗ )
=
V ∗ .
Now, for simplicity of notations, we also introduce the following notation.
Remember the smallest eigenvalue λ min of some matrix A is defined as
[Equation]
A similar way can be applied to define the smallest 2 , D eigenvalue σ 2 ,D min of a matrix A :
[Equation]
For simplicity, we will use σ 2 ,D min to denote the 2 , D eigenvalue of ∇ θ V ( θ ∗ ) .
We will always suppose each s t are sampled i.i.d.
from µ .
We now introduce the following result.
Theorem B.1.
In the Non Projected Neural TD learning using a one hidden layer neural network, smooth and Lipschitz activation function, suppose each s t are sampled i.i.d from µ , and stepsize is chosen to be α t = 1 λ ( t +1) .
Further, let A be the event
[Equation]
and assume
[Equation]
Then, A happens with probability at least 1 − δ and the sequence (cid:8) E (cid:2)
||
θ t − θ ∗ || 2 | A (cid:3)(cid:9) converges to 0 .
B. U SEFUL L EMMAS
Lemma B.1.
Non Projected Neural TD Learning with mean-path update can be rewritten as
[Equation]
where ¯ g 1 ( θ t ) , ¯ g 2 ( θ t ) , ¯ g 3 ( θ t ) are defined as follows:
[Equation]
Here, λ 4 ∈ [0 , 1] is a scalar and θ mid4 =
λ 4 θ t
+ (1 − λ 4 ) θ ∗ is a vector such that
[Equation]
Proof.
¯ g ( θ t ) in Eq.
(8) can be divided into two parts,
[Equation]
Now let ∇ θ V ( θ ∗ ) T D ( γP − I ) be the vector e in Lemma A.2.
There exists a scalar λ 4 ∈ [0 , 1] and a vector θ mid4 =
λ 4 θ t
+ (1 − λ 4 ) θ ∗ such that
[Equation]
Using this fact, ∇ θ V ( θ ∗ ) T D ( γP − I )
( V ( θ t ) − V ( θ ∗ )) can be divided by
[Equation]
Lemma B.2.
If the activation function is l -Lipschitz and c 0 -smooth, for any s , θ 1 and θ 2 , the inequalities
[Equation]
hold where c 1 is a scalar that is independent of m and θ .
Proof.
It is easy to see that
[Equation]
Suppose the activation function σ is c 0 -smooth.
That is, for any x and y , ||
σ ′ ( x )
[Equation]
y )
{+|| ≤ c 
+}
|| x − y || .
This means ||∇ θ
[Equation]
( s
[Equation]
θ 1 )
[Equation]
∇ θ V ( s, θ 2 ) ||
[Equation]
1 m || [ b 1 ( σ ′ ( θ 1 T 1 s ) − σ ′
( θ 1 T 2 s ))
s T , · · · , b m ( σ ′ ( θ mT 1 s ) − σ ′
( θ mT 2 s ))
s T ] ||
[Equation]
which proves the first part of the lemma.
Now let us move on to the second part of the lemma.
By
Lipschitzness,
[Equation]
Now we finish the second part of the lemma.
Lemma B.3.
The following inequality holds:
[Equation]
Proof.
To begin with, the definition of N is
[Equation]
Using the definition of σ 2 ,D min ,
[Equation]
which establishes the result.
Lemma B.4.
If a non negative sequence { x t } satisfies
[Equation]
for some b, c > 0 , then the sequence { X t } converges to 0 .
Proof.
Recursively applying the relation between X t +1 and X t , we can derive the following:
[Equation]
The first term in Eq.
(B.2) definitely goes to 0 as t goes to infinity.
Now let us consider the term (cid:81) tj = i +1 (1 − cj ) .
A logarithm usually help us to convert it into a sum, so we perform the following manipulations:
[Equation]
where the first inequality uses the fact ln(1 + x ) ≤ x and the second inequality uses (cid:80) tj = i +1 1 j ≥
ln
( t + 1) − ln
( i + 1) .
So,
[Equation]
This means the third term in Eq.
(B.2) goes to 0 . Now consider (cid:80) t − 1 i =1 bi 2 (cid:81) tj = i +1 (1 − cj ) , we obtain [Equation]
1 (cid:88)
[Equation]
where the second inequality uses the fact ( i +1)
2 i≤ 4 , and the third inequality combine the two facts
[Equation]
[Equation]
(cid:82)
[Equation]
{+2 ( i + 1) c − 2 di given c ≥ 2 and (cid:80) t − 1 i =
+}
( i + 1) c − 2 ≤ (cid:82) t − 1( i + 1) c − 2 di given c ≤ 2 .
If c ̸ = 1 , then t − 1 (cid:88)
[Equation]
=
[Equation]
If c = 1 , then
[Equation]
Under both cases, we can easily argue that the second term in Eq.
(B.2) goes to 0 .
[Equation]
P ROOF OF L EMMA B.
Consider Non Projected Neural TD Learning,
[Equation]
Given θ t , the only randomness is from s t .
Taking expectation on both side, we obtain
[Equation]
First, we consider 2 α t ( θ t − θ ∗ )
T ¯ g ( θ t ) .
Lemma B.1 allows us to divide it into several parts and bound them respectively as follows: [Equation]
To bound ( θ t − θ ∗ ) T ¯ g 1 ( θ t ) , ( θ [Equation] − θ ∗ [Equation]
T
[Equation]
g 1 (
[Equation]
where the first equality is by Eq.
(20), the second equality is by setting f = ∇ θ
V ( θ ∗ )( θ t − θ ∗ ) in
Lemma A.1, and the inequality is by Lemma B.3.
To bound ( θ t − θ ∗ ) T ¯ g 2 ( θ t ) , ( θ t − θ ∗ ) T ¯ g 2 ( θ t ) =( θ t − θ ∗ ) T ( ∇ θ V ( θ t ) − ∇ θ V ( θ ∗ ))
T D ( γP − I )
( V ( θ t ) − V ( θ ∗ ))
≤ γ ( θ t − θ ∗ ) T ( ∇ θ V ( θ t ) − ∇ θ V ( θ ∗ ))
T DP ( V ( θ t ) − V ( θ ∗ ))
+ ( θ t − θ ∗ ) T ( ∇ θ V ( θ t ) − ∇ θ V ( θ ∗ ))
T D ( V ( θ t ) − V ( θ ∗ )) ≤ (1 + γ ) c 0 lm − 0 .
5 ||
θ t − θ ∗ || 3 , where the equality is by Eq.
(21), and second inequality is by setting x to be ( ∇ θ V ( θ t ) − ∇ θ V ( θ ∗ ))
( θ t − θ ∗ ) , y to be V ( θ t ) − V (ˆ θ ∗ ) in Lemma A.5 and bounding each entry of x , y using Lemma B.2. To bound ( θ t − θ ∗ ) T ¯ g 3 ( θ t ) ,
[Equation]
where the equality is by Eq. (22), and second inequality is by setting x to be ( ∇ θ V ( θ mid4 )
− ∇ θ V ( θ ∗ ))
( θ t − θ ∗ ) , y to be ∇ θ V ( θ ∗ )( θ t − θ ∗ ) in Lemma A.5 and bounding each entry of x , y using Lemma B.2.
Combining the above facts,
[Equation]
Second, we consider E [ α 2 t
|| g
( θ t ) ||
2 | θ t ] in Eq.
For simplicity, define f ( s ) =
V ( s, θ ∗ ) −
V ( s, θ t ) .
Since we are using one hidden layer neural network to approximate V ( s, θ ) ,
[Equation]
Recall that V ( s, θ ∗ ) is
[Equation]
and ¯ g ( θ t ) can be written as
[Equation]
By Eq.
(10) we can obtain the following quick result:
[Equation]
And this leads to the following:
[Equation]
where the second inequality is by Jensen’s inequality. Further,
[Equation]
Now let us go back to Eq.
[Equation]
which can be bounded as
[Equation]
This means
[Equation]
{+For simplicity [Equation] denote a t = 
 α t (1 − γ )( σ+}
[Equation]
, X t = || θ t − θ ∗ || . We can rewrite the above equation as:
[Equation]
A common choice of stepsize is α t = 1 λ ( t +1) , ∀ t ≤ 0 . For this specific choice, we claim that
X t < a 1 2 b 1 holds with probability at least 1 − δ , given that E [ X 0 ] < a 1 2 b 1 δ . This can be shown as follows: First, notice that the condition E [ || θ
[Equation] − θ [Equation] || ] <
[Equation]
δm 0 . 5 actually implies that E [ X 0 ] < a 1 2 b 1 δ .
Next, define a new random variable sequence Y t {+such that Y 
 = X 0 and for+} t ≥
[Equation]
Consider t 0 = inf t { X t > a 1 2 b 1 } . It is easy to conclude that
[Equation]
> t
[Equation]
The definition of t 0 tells us X t ≤ a 1 2 b 1 for all t < t 0 , which leads to X t < a t 2 b t since h ( t ) =
a t 2 b t is monotonically decrease.
This furthermore implies
[Equation]
{+where the last inequality follows by m > 1536(
+}
+ γ 2 ) γ
2 c 20 l 4 r 2max λ(1 − γ ) 2 (2(1 − γ )( σ 2 ,D min ) 2 λ 3 − 3(1+ γ 2 ) l 4 ) 3 and further c t < a 3 t 8 b 2 t , ∀ t . With the above fact we conclude the sequence { Y t } is a super martingale. Now, let A be the event (cid:110) sup t X t < a 1 2 b 1 (cid:111) =
(cid:110) X 0 < a 1 2 b 1 , X 1 < a 1 2 b 1 , · · · (cid:111) . To compute P ( A ) , notice that
[Equation]
Define T = inf t { Y t ≥ a 1 2 b 1 } be a stopping time for { Y t } . On one hand, using Doob’s optional sampling theorem,
[Equation] On the other hand, E [ Y τ ∧ T ] can be expanded as [Equation]
Combining these two facts,
[Equation]
Hence, [Equation] sup
[Equation]
It is easy to see P
[Equation]
This result holds for all τ , so we can let τ → ∞ and obtain
P ( A ) ≥ 1 − δ. Actually, using h ( t ) = a t 2 b t is monotonically decrease again, event A implies X t ≤ a t 2 b t , ∀ t . With this fact,
[Equation]
{+Plugging in a t = 
 α t (1 − γ )( σ 2 ,D min ) 2+} − 3 α 2 t 1 + γ 2 ) l 4 , c t = 12 α 2 t γ 2 l 2 r 2max
[Equation]
1 −
[Equation]
and α t = 1 λ ( t +1) , we can derive
[Equation]
Take expectation on both sides and we derive [Equation] By applying Lemma B.4, we conclude that the sequence (cid:8) E (cid:2)
X 2 t | A (cid:3)(cid:9) converges to 0 .