Original: On the role of overparameterization in  Temporal Difference learning with linear function approximation
Revised: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation

Original: Anonymous Author s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Much of the recent successes of deep learning can be attributed to scaling up thesize of the networks to the point where they often are vastly overparameterized.
Revised: Much of the recent successes of deep learning can be attributed to scaling up thesize of the networks to the point where they often are vastly overparameterized.

Original: Thus, understanding the role of overparameterization is of increasing importance.
Revised: Thus, understanding the role of overparameterization is of increasing importance.

Original: While predictive theories have been developed for supervised learning, little isknown about the Reinforcement Learning case.
Revised: While predictive theories have been developed for supervised learning, little isknown about the Reinforcement Learning case.

Original: In this work, we take a theoretical approach and study the role of overparameterization for  Temporal Difference(TD) learning in the linear setting.
Revised: In this work, we take a theoreticalapproach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.

Original: We leverage tools from random matrix theoryand random graph theory to obtain a characterization of the spectrum of the TDoperator.
Revised: We leverage tools from random matrixtheory and random graph theory to obtain a characterization of the spectrum of the TD operator.

Original: We use this result to study the stability and optimization dynamics of TD learning as a function of the number of parameters
Revised: We use this result to study the stability and optimization dynamicsof TD learning as a function of the number of parameters

Original:  Introduction
Revised:  Introduction

Original: Occam’s razor principle states that among many plausible explanations, the simplest one is themost likely to be true.
Revised: Occam’s razor principle states that among many plausible explanations, the simplest one is themost likely to be true.

Original: In statistics and machine learning, this is often interpreted as models witha restricted number of parameters should be privileged as they generalize better [1].
Revised: In statistics and machine learning, this is often interpreted as models witha restricted number of parameters should be privileged as they generalize better [1].

Original: This view issupported by the classical bias-variance trade-off which implies that models that are too flexibleare bound to overfit [2].
Revised: This view issupported by the classical bias-variance trade-off which implies that models that are too flexibleare bound to overfit [2].

Original: Yet, deep neural networks with more parameters than training data pointsachieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.
Revised: Yet, deep neural networks with more parameters than training data pointsachieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.

Original: A significant understanding of this apparent tension has been reached in recent years.
Revised: A significant understanding of this apparent tension has been reached in recent years.

Original: A largebody of work using tools from random matrix theory has shown that, in models with randomfeatures, overparameterization does not lead to overfitting but instead better generalization [6].
Revised: A largebody of work using tools from random matrix theory has shown that, in models with randomfeatures, overparameterization does not lead to overfitting but instead better generalization [6].

Original: Thisphenomenon is known as double descent and unifies the traditional and modern understanding ofbias and variance.
Revised: Thisphenomenon is known as double descent and unifies the traditional and modern understanding ofbias and variance.

Original: Importantly, [7] has shown that the double descent phenomenon is in large partexplained by purely optimization rather than label noise.
Revised: Importantly, [7] has shown that the double descent phenomenon is in large partexplained by purely optimization rather than label noise.

Original: Following their insight, we mainly take anoptimization approach in the rest of our paper.
Revised: Following their insight, we mainly take anoptimization approach in the rest of our paper.

Original: While we now have a theory of overparameterization in supervised learning, we are not aware ofany theoretical work on overparameterization in the reinforcement learning (RL) setting.
Revised: While we now have a theory of overparameterization in supervised learning, we are not aware ofany theoretical work on overparameterization in the reinforcement learning (RL) setting.

Original: Indeed,
Revised: Indeed,

Original: RL poses several new additional challenges.
Revised: RL poses several new additional challenges.

Original: First, the agent has to interact with an environmentwhose structure may be complex.
Revised: First, the agent has to interact with an environmentwhose structure may be complex.

Original: Second, each state may not sampled with equal probabilitywhile each datapoint is usually sampled uniformly in supervised learning.
Revised: Second, each state may not sampled with equal probabilitywhile each datapoint is usually sampled uniformly in supervised learning.

Original: In this work, we focusmainly on policy evaluation , i.e evaluating the expected return of a fixed policy.
Revised: In this work, we focusmainly on policy evaluation , i.e evaluating the expected return of a fixed policy.

Original: More precisely,we study the behavior of Temporal Difference learning [8] which has become ubiquitous for policyevaluation in modern Deep RL [9, 10].
Revised: More precisely,we study the behavior of Temporal Difference learning [8] which has become ubiquitous for policy evaluation in modern Deep RL [9, 10].

Original: More precisely, our contributions are, in the limit of large number of states and parameters: 1) We introduce a class of random graphs for which we can compute analytically the spectrum ofthe TD operator when using uniform off-policy sampling, 2) we derive approximations for thelargest and smallest eigenvalue in the non-uniform case.
Revised: Our contributions are, in the limit of large number of states and parameters: 1) We introduce aclass of random graphs for which we can compute analytically the spectrum of the TD operatorwhen using uniform off-policy sampling, 2) we derive approximations for the largest and smallesteigenvalue in the non-uniform case.

Original: Furthermore, we analyze the behavior ofthe traditional Mean Square Bellman Error (MSBE) when learning with TD(0) and show that 3)it exhibits a double descent phenomenon similar to supervised learning but with a multiplicativefactor depending on the discount factor in the overparameterized regime and 4) the asymptoticresidual MSBE displays a peaking behavior near the interpolation threshold (when the number ofparameters approaches the number of states) that is specific to RL.
Revised: Furthermore, we analyze the behavior of the traditional Mean Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a doubledescent phenomenon similar to supervised learning but with a multiplicative factor depending onthe discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displaysa peaking behavior near the interpolation threshold (when the number of parameters approachesthe number of states) that is specific to RL.

Original: Finally, 5) we showcase howthenon-stationarity of the policy can make the optimization of the value function unstable.
Revised: Finally, 5) we showcase how the non-stationarity of thepolicy can make the optimization of the value function unstable.

Original: All of thesepoints are motivated theoretically and empirically validated on small gridworlds MDPs.
Revised: All of these points are motivatedtheoretically and empirically validated on small gridworlds MDPs.

Original:  Background2.1 Value estimationDefinition:
Revised:  Background2.1 Value estimationDefinition:

Original: We place ourselves in the Markov Decision Process (MDP) setting [11] definedby {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈ 47 R |S|·|A|×|A| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.
Revised: We place ourselves in the Markov Decision Process (MDP) setting [11] definedby {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈ 47 R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.

Original: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.
Revised: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.

Original: Fora given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be ableto estimate the expected discounted return q π ( s, a ) =
Revised: Fora given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be ableto estimate the expected discounted return q π ( s, a ) =

Original: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s 

Revised: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s

Original: =
Revised: =

Original: s, a 0 = a ] .
Revised: s, a 0 = a ] .

Original: This quantity is also known as the state-action value or the Q-function.
Revised: This quantity is also known as the state-action value or the Q-function.

Original: Furthermore, it can be written as the solution of the Bellman equation [12] 1 : q π ( s, a ) =
Revised: Furthermore, it can be written as the solution of the Bellman equation [12] 1 : q π ( s, a ) =

Original: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .
Revised: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .

Original: Bydefining P π ∈ R |S|·|A|×|S|·|A|
Revised: Bydefining P π ∈ R |S|·|A|×|S|·|A|

Original: the state-action to state-action transition matrix, we can write it inmatrix form
Revised: the state-action to state-action transition matrix, we can write it inmatrix form

Original: [Equation]
Revised: [Equation]

Original: Linear function approximation: While Equation (1) can be solve efficiently when r , P π are knownand when n = |S| · |A| is not too large, in many cases this cannot be computed.
Revised: Linear function approximation: While Equation (1) can be solved efficiently when r , P π areknown and when n = |S|·|A| is not too large, in many cases this cannot be computed.

Original: One solution is tointroduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximateq π as q π ≈ Φ θ .
Revised: One solutionis to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .

Original: This is referred to as the Linear Function Approximation (LFA) setting.
Revised: This is referred to as the Linear Function Approximation (LFA) setting.

Original: Temporal Difference with LFA:
Revised: Temporal Difference with LFA:

Original: Estimating the Q-function in the Linear Function Approximationsetting, can be done using the well-known algorithm TD(
Revised: Estimating the Q-function in the Linear Function Approximationsetting can be done using the well-known algorithm TD(0)

Original: [Equation]
Revised: [8] whose expected update is

Original: [Equation]
Revised: [Equation]

Original: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summingto 1 , weighting each state-action by their probability of being sampled.
Revised: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summingto 1 , weighting each state-action by their probability of being sampled.

Original: Contrary to gradient descentwith an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φmight have a negative eigenvalue.
Revised: Contrary to gradient descentwith an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φmight have a negative eigenvalue.

Original: This phenomenon is known as the deadly triad as it can onlyhappen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on thenext value as TD(0) does [13, 14, 15].
Revised: This phenomenon is known as the deadly triad as it can onlyhappen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on thenext value as TD(0) does [13, 14, 15].

Original: 2.2 Random Matrix Theory
Revised: 2.2 Random Matrix Theory

Original: Wishart matrices:
Revised: Wishart matrices:

Original: Extremely relevant in the study of least squares with random features arematrices from the Wishart distribution [16].
Revised: Extremely relevant in the study of least squares with random features arematrices from the Wishart distribution [16].

Original: These matrices can be written as 1 n Φ ⊤ Φ for Φ an × p matrix with entries of 0 mean, unit variance and bounded 4 th moment sampled independently and identically distributed.
Revised: These matrices can be written as 1 n Φ ⊤ Φ for Φ an × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.

Original: In that case, when p, n → ∞ and the ratio converges to a finitelimit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [17] (MP) distribution whose density can be written as
Revised: In that case, when p, n → ∞ and the ratio converges to a finitelimit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [17] (MP) distribution whose density can be written as

Original: [Equation]
Revised: [Equation]

Original: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x )are the 0 eigenvalues, which happens when p ≥ n , λ max
Revised: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x )are the 0 eigenvalues, which happens when p ≥ n , λ max

Original: =
Revised: =

Original: (1 + √ ρ ) 2 , the largest, and λ +min =(1 − √ ρ ) 2 , the smallest non-zero eigenvalue.
Revised: (1 + √ ρ ) 2 , the largest, and λ +min =(1 − √ ρ ) 2 , the smallest non-zero eigenvalue.

Original: Wigner-type matrices: [18] historically introduced and studied symmetric n × n matrices ofthe form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance fori ≥ j 2 .
Revised: Wigner-type matrices: [18] historically introduced and studied symmetric n × n matrices ofthe form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance fori ≥ j 2 .

Original: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) =12 π √ 4 − x 2 1 [ − 2 , 2] dx .
Revised: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) =12 π √ 4 − x 2 1 [ − 2 , 2] dx .

Original: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [20].
Revised: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [20].

Original: 2.3 Random graphs
Revised: 2.3 Random graphs

Original: Random graphs are useful models to infer structural or spectral properties of typical graphs.
Revised: Random graphs are useful models to infer structural or spectral properties of typical graphs.

Original: Theearliest theoretical analyses of random graphs that we are aware of were done independently by [21]and [22].
Revised: Theearliest theoretical analyses of random graphs that we are aware of were done independently by [21]and [22].

Original: The model introduced in [21] is a random graph where each of the n nodes is connectedat random to d others.
Revised: The model introduced in [21] is a random graph where each of the n nodes is connectedat random to d others.

Original: The second model [22] is slightly different as each node is independentlyconnected to any other with a probability µ ∈ ]0 , 1[ .
Revised: The second model [22] is slightly different as each node is independentlyconnected to any other with a probability µ ∈ ]0 , 1[ .

Original: We refer to the first one as the d -regular Erd˝osR ´ enyi model G ( n, d ) and the second one as the Erd ˝ os-R ´ enyi-Gilbert G ( n, µ ) model.
Revised: We refer to the first one as the d -regular Erd˝osR ´ enyi model G ( n, d ) and the second one as the Erd ˝ os-R ´ enyi-Gilbert G ( n, µ ) model.

Original: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct amatrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rowssumming to 1 ).
Revised: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct amatrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rowssumming to 1 ).

Original: P corresponds to the Markov transition matrix of a random walk on the graph and isconnected to the (left)-normalized Laplacian L by L =
Revised: P corresponds to the Markov transition matrix of a random walk on the graph and isconnected to the (left)-normalized Laplacian L by L =

Original: I n − P .
Revised: I n − P .

Original: For many classes of random graphs,the spectrum of these quantities is known [23, 24].
Revised: For many classes of random graphs,the spectrum of these quantities is known [23, 24].

Original:  Spectrum of TD: from random graphs to spiked models
Revised:  Spectrum of TD: from random graphs to spiked models

Original: 3.1 From Ordinary Least Squares to TD(0)
Revised: 3.1 From Ordinary Least Squares to TD(0)

Original: Random matrix models have been used in supervised learning, especially in the simple least squaresmodel [25, 26, 27], to explain phenomena arising in deep learning.
Revised: Random matrix models have been used in supervised learning, especially in the simple least squaresmodel [25, 26, 27], to explain phenomena arising in deep learning.

Original: Indeed, while much simpler thanlarge neural networks, these models can shed light on the optimization and generalization behaviorof vastly overparameterized models [28, 29], something that most traditional complexity measurescould not [30, 31].
Revised: Indeed, while much simpler thanlarge neural networks, these models can shed light on the optimization and generalization behaviorof vastly overparameterized models [28, 29], something that most traditional complexity measurescould not [30, 31].

Original: Using gradient descent on a least square problem, our parameter update can be written as
Revised: Using gradient descent on a least square problem, our parameter update can be written as

Original: [Equation]
Revised: [Equation]

Original: (4) for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.
Revised: (4) for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.

Original: When choosing thelearning rate η optimally, the convergence rate is expressed as λ
Revised: When choosing thelearning rate η optimally, the convergence rate is expressed as λ

Original: [Equation]
Revised: [Equation]

Original: for λ +min , respectivelyλ max , the smallest, resp.
Revised: for λ +min , respectivelyλ max , the smallest, resp.

Original: largest, non-zero eigenvalue of 1 n Φ ⊤ Φ [32].
Revised: largest, non-zero eigenvalue of 1 n Φ ⊤ Φ [32].

Original: When Φ has i.i.d entries with 0 mean, unit variance and bounded 4th moment, as we willconsider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.
Revised: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we willconsider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.

Original: In particular, we haveanalytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).
Revised: In particular, we haveanalytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).

Original: Whenlim p,n →∞ p / n = ρ , we have λ max / λ +min
Revised: Whenlim p,n →∞ p / n = ρ , we have λ max / λ +min

Original: →
Revised: →

Original: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .
Revised: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .

Original: This function has asymptotes at 1 , forp
Revised: This function has asymptotes at 1 , forp

Original: = o ( n ) or n = o ( p ) and diverges when n = p .
Revised: = o ( n ) or n = o ( p ) and diverges when n = p .

Original: Note that this peak-shaped function can also beobserved empirically in more complex models such as kernel methods [33] and neural networks inthe limit of infinite width as they are in the Neural Tangent Kernel regime [34, 35].
Revised: Note that this peak-shaped function can also beobserved empirically in more complex models such as kernel methods [33] and neural networks inthe limit of infinite width as they are in the Neural Tangent Kernel regime [34, 35].

Original: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would bereplaced by Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would bereplaced by Φ ⊤ Ξ( I n − γ P π )Φ .

Original: While we can model Φ as an i.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allow us to characterize the spectrum of the
Revised: While we can model Φ as an i.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allow us to characterize the spectrum of the

Original: TD operator  Ξ( I n − γ P π ) .
Revised: TD operator Φ ⊤ Ξ( I n − γ P π )Φ .

Original: We believe this is one of the key reasons why overparameterization in Temporal Difference learning has not been studied extensively.
Revised: We believe this is one of the key reasons why overparameterizationin Temporal Difference learning has not been studied extensively.

Original: Finding good typical models for P πis therefore of utmost importance, and the next section will introduce a class of models for whichthe spectrum of TD can be computed.
Revised: Finding good typical modelsfor P π is therefore of utmost importance, and the next section will introduce a class of models forwhich the spectrum of TD can be computed.

Original: 3.2 P π as the Markov transition matrix of a random graph
Revised: 3.2 P π as the Markov transition matrix of a random graph

Original: In RL, P π is the Markov transition matrix of a random walk when sampling from π .
Revised: In RL, P π is the Markov transition matrix of a random walk when sampling from π .

Original: As randomgraphs are used to model properties of typical graphs, a candidate for a model of P π would be to takethe Markov transition matrix of a random graph.
Revised: As randomgraphs are used to model properties of typical graphs, a candidate for a model of P π would be to takethe Markov transition matrix of a random graph.

Original: However, random graphs, at least the Erd˝os-R´enyi(-
Revised: However, random graphs, at least the Erd˝os-R´enyi(-

Original: Gilbert) type ones, are not able natively to take into account the impact of π , most importantly thefact that some state-actions are visited more often than others under d π , the stationary distributionof π .
Revised: Gilbert) type ones, are not able natively to take into account the impact of π , most importantly thefact that some state-actions are visited more often than others under d π , the stationary distributionof π .

Original: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markovtransition matrix of a G ( n, µ ) graph deformed by d π .
Revised: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markovtransition matrix of a G ( n, µ ) graph deformed by d π .

Original: Let us start with an undirected graph of the type G ( n, µ ) .
Revised: Let us start with an undirected graph of the type G ( n, µ ) .

Original: To take into account the state-actionvisitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .
Revised: To take into account the state-actionvisitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .

Original: Now, we define thedeformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.
Revised: Now, we define thedeformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.

Original: Notehere that when d π is uniform, we revert back to the original case A π = A .
Revised: Notehere that when d π is uniform, we revert back to the original case A π = A .

Original: This construction keepsthe same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.
Revised: This construction keepsthe same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.

Original: We can informally write the Markov transition matrix associated to this deformed graph as 4ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n Wis a Wigner-type matrix with 0 mean and unit variance.
Revised: We can informally write the Markov transition matrix associated to this deformed graph as 4ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n Wis a Wigner-type matrix with zero mean and unit variance.

Original: It can be tempting to understand underwhich conditions the stochastic part becomes negligible and what it entails.
Revised: It can be tempting to understand underwhich conditions the stochastic part becomes negligible and what it entails.

Original: As we will see shortly,this setting will reveal itself to be of importance when studying the spectrum of TD.
Revised: As we will see shortly,this setting will reveal itself to be of importance when studying the spectrum of TD.

Original: Definition 1 (Asymptotically well-connected graphs) .
Revised: Definition 1 (Asymptotically well-connected graphs) .

Original: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its
Revised: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its

Original: Markov transition matrix.
Revised: Markov transition matrix.

Original: The choice for the name of Definition 1 comes from the fact that | λ 2 | is related to the presence ofbottlenecks in the graph through Cheeger’s inequality [36].
Revised: The choice for the name of Definition 1 comes from the fact that | λ 2 | is related to the presence ofbottlenecks in the graph through Cheeger’s inequality [36].

Original: Example.
Revised: Example.

Original: Graphs of the type G ( n, µ ) (resp.
Revised: Graphs of the type G ( n, µ ) (resp.

Original: G ( n, d ) ) are asymptotically well-connected if theirexpected degree d
Revised: G ( n, d ) ) are asymptotically well-connected if theirexpected degree d

Original: = nµ (resp.
Revised: = nµ (resp.

Original: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [23, 24].
Revised: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [23, 24].

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: Note here that this again generalizes the uniform case as, when d π is uniform, the conditionbecomes equivalent to 1 ∈ o ( d ) as in the example.
Revised: Note here that this again generalizes the uniform case as, when d π is uniform, the conditionbecomes equivalent to 1 ∈ o ( d ) as in the example.

Original: In the non-uniform case, this condition entailsthat either d π spreads efficiently across states, or the degree d rises fast enough that those states arestill visited sufficiently often.
Revised: In the non-uniform case, this condition entailsthat either d π spreads efficiently across states, or the degree d rises fast enough that those states arestill visited sufficiently often.

Original: In that case, we have
Revised: In that case, we have

Original: [Equation]
Revised: [Equation]

Original: This approximation is particularly interesting as it asymptotically conserves the left and righteigenvectors, d π and 1 n , of the original P π associated to the eigenvalue 1 .
Revised: This approximation is particularly interesting as it asymptotically conserves the left and righteigenvectors, d π and 1 n , of the original P π associated to the eigenvalue 1 .

Original: Furthermore, if allstate-actions are reachable, the sequence of matrix power iterates of P π , (cid:0) P π (cid:1) k converges to 1 n d ⊤ π .
Revised: Furthermore, if allstate-actions are reachable, the sequence of matrix power iterates of P π , (cid:0) P π (cid:1) k converges to 1 n d ⊤ π .

Original: While we are not the first to use tools from graph theory in the context of RL, related work weare aware of [37, 38, 39, 40] takes a different approach and leverages eigenvectors of the Markovtransition matrix (or Laplacian) to construct better representations for the value function or usethem as options.
Revised: While we are not the first to use tools from graph theory in the context of RL, related work weare aware of [37, 38, 39, 40] takes a different approach and leverages eigenvectors of the Markovtransition matrix (or Laplacian) to construct better representations for the value function or usethem as options.

Original: In contrast, we use spectral properties of P π to study the convergence of TemporalDifference learning.
Revised: In contrast, we use spectral properties of P π to study the convergence of Temporal learning.

Original: 3.3 Spiked Wishart model for TD with uniform sampling
Revised: 3.3 Spiked Wishart model for TD with uniform sampling

Original: Now that we have a simple random graph model for P π , we focus on estimating the spectrumof Φ ⊤ Ξ( I n − γ ˆ P π )Φ .
Revised: Now that we have a simple random graph model for P π , we focus on estimating the spectrumof Φ ⊤ Ξ( I n − γ ˆ P π )Φ .

Original: For simplicity, we assume the off-policy sampling is done uniformly, i.eΞ = 1 n I n .
Revised: For simplicity, we assume the off-policy sampling is done uniformly, i.eΞ = 1 n I n .

Original: While this assumption is not realistic in most RL settings, it is still of interest in thecontext of dynamic programming where we have access to all the states.
Revised: While this assumption is not realistic in most RL settings, it is still of interest in thecontext of dynamic programming where we have access to all the states.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition [41] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to
Revised: When ργ 2 ≥ 1 , there is a phase transition [41] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation]
Revised: [Equation]

Original: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =
Revised: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =

Original: ( (cid:112) 1 / γ 2 − 1) 2 .
Revised: ( (cid:112) 1 / γ 2 − 1) 2 .

Original: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .
Revised: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .

Original: In thatcase, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , theeffective horizon of the problem, as lim
Revised: In thatcase, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , theeffective horizon of the problem, as lim

Original: ρ →∞ λ max / λ +min
Revised: ρ →∞ λ max / λ +min

Original: =
Revised: =

Original: 11 − γ .
Revised: 11 − γ .

Original: In the tabular regime, Φ = I n .
Revised: In the tabular regime, Φ = I n .

Original: As
Revised: As

Original: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .
Revised: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .

Original: This would give aworst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterizedcase above.
Revised: This would give aworst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterizedcase above.

Original: As these eigenvalues govern the optimization speed of TD, the vastly overparameterizedcase and the tabular can be thought of as comparable.
Revised: As these eigenvalues govern the optimization speed of TD, the vastly overparameterizedcase and the tabular can be thought of as comparable.

Original: 3.4 Non-uniform off-policy sampling
Revised: 3.4 Non-uniform off-policy sampling

Original: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assumingΞ = 1 n I n .
Revised: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assumingΞ = 1 n I n .

Original: In this subsection we analyze the more general case where Ξ is not uniform.
Revised: In this subsection we analyze the more general case where Ξ is not uniform.

Original: However bydoing so, we lose guarantees and must resort to approximations.
Revised: However bydoing so, we lose guarantees and must resort to approximations.

Original: First, we analyze the eigenvaluesof Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .
Revised: First, we analyze the eigenvaluesof Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy asecular equation [42]
Revised: γ 1 n d ⊤ π ) satisfy asecular equation [42]

Original: [Equation]
Revised: [Equation]

Original: This expression is another form of the characteristic polynomial of a matrix whose roots are itseigenvalues.
Revised: This expression is another form of the characteristic polynomial of a matrix whose roots are itseigenvalues.

Original: However, this form enables us to derive approximations more easily.
Revised: However, this form enables us to derive approximations more easily.

Original: In particular,for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸
Revised: In particular,for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸

Original: = m , we have for λ ≈ ξ mthat 1 ξ i − λ ≈ 1 ξ i .
Revised: = m , we have for λ ≈ ξ mthat 1 ξ i − λ ≈ [Equation]

Original:  From this assumption, we derive an approximation for the minimum eigenvalue ofΞ( [Equation]
Revised: ξ i if i ̸ = m . From this assumption, we derive an approximation for the minimumeigenvalue of Ξ( I n − γ ˆ P π )

Original: [Equation]
Revised: [Equation]

Original: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution andthe discount factor.
Revised: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution andthe discount factor.

Original: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which isthe product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .
Revised: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which isthe product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .

Original: Note that a similar expression can bederived for the maximum eigenvalue under similar assumptions.
Revised: Note that a similar expression can bederived for the maximum eigenvalue under similar assumptions.

Original: Now, again reasoning approximately, if that eigenvalue was very small compared to the other oneswho would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similarspiking behavior as before.
Revised: Now, again reasoning approximately, if that eigenvalue was very small compared to the other oneswho would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similarspiking behavior as before.

Original: However, in that case, the spiking eigenvalue would be
Revised: However, in that case, the spiking eigenvalue would be

Original: [Equation]
Revised: [Equation]

Original: While less simple than the estimators developed in the last subsection, we will see in Section 5 thatit still predicts accurately the smallest non-zero eigenvalue.
Revised: While less simple than the estimators developed in the last subsection, we will see in Section 5 thatit still predicts accurately the smallest non-zero eigenvalue.

Original:  Optimizing with Temporal Difference learning
Revised:  Optimizing with Temporal Difference learning

Original: 4.1 Decomposition of the error
Revised: 4.1 Decomposition of the error

Original: Using results about the spectrum of the TD operator shown in the last section, we analyze thebehavior of the Mean Square Bellman Error L MSBE ( q t ) =
Revised: Using results about the spectrum of the TD operator shown in the last section, we analyze thebehavior of the Mean Square Bellman Error L MSBE ( q t ) =

Original: 12 ∥
Revised: 12 ∥

Original: r + γ P π q t − q t ∥
Revised: r + γ P π q t − q t ∥

Original: 2Ξ during theoptimization.
Revised: 2Ξ during theoptimization.

Original: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n
Revised: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n

Original: [Equation]
Revised: [Equation]

Original: the q -value asymptotically reached by TD(0), we can decompose this error into two components.
Revised: the q -value asymptotically reached by TD(0), we can decompose this error into two components.

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ),we have the following decomposition
Revised: (2) ),we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the - norm in metric Ξ .
Revised: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Following [43], we refer to ∥ q t − q TD ∥ Ξ as the approximation error and ∥
Revised: Following [43], we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥

Original: r + γ P π q TD − q TD
Revised: r + γ P π q TD − q TD

Original: ∥ Ξ as theestimation error.
Revised: ∥ Ξ as theestimation error.

Original: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π )in metric Ξ .
Revised: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π )in metric Ξ .

Original: Note that, beyond this factor, we are not exactly in the same setting as [43] as theasymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, asknown in the RL community [44].
Revised: Note that, beyond this factor, we are not exactly in the same setting as [43] as theasymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, asknown in the RL community [44].

Original: Indeed, TD(0) finds the minimum of the Mean Square Projected
Revised: Indeed, TD(0) finds the minimum of the Mean Square Projected

Original: Bellman Error (MSPBE) but not of the MSBE [45].
Revised: Bellman Error (MSPBE) but not of the MSBE [45].

Original: However in practice it is common to minimizethe MSBE as it does not require an expensive projection step.
Revised: However in practice it is common to minimizethe MSBE as it does not require an expensive projection step.

Original: 4.2 The ordinary behavior of the approximation error
Revised: 4.2 The ordinary behavior of the approximation error

Original: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .
Revised: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η = [Equation] max
Revised: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η = 2 λ +min + λ max

Original: [Equation]
Revised: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).
Revised: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).

Original: This is a slightly more general version of the linear convergence of gradient descent, however, as the
Revised: This is a slightly more general version of the linear convergence of gradient descent, however, as the

Original: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .
Revised: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .

Original: 4.3 The singular behavior of the estimation error
Revised: 4.3 The singular behavior of the estimation error

Original: Now we turn our attention to the second term; the estimation error which is the residual error after
Revised: Now we turn our attention to the second term; the estimation error which is the residual error after

Original: T → + ∞ steps.
Revised: T → + ∞ steps.

Original: In supervised learning, we expect this error to be the usual bias of our algorithmand thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possible tointerpolate the data.
Revised: In supervised learning, we expect this error to be the usual bias of our algorithmand thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possibleto interpolate the data.

Original: However, inthe reinforcement learning, the behavior of this term can be quitedifferent, in particular it can be non-monotonous.
Revised: However, in reinforcement learning, the behavior of this term can be quitedifferent, in particular it can be non-monotonous.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptoticallyrank one
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptoticallyrank one

Original: [Equation]
Revised: [Equation]

Original: When γ = 0 , i.e we just perform reward regression, this is equal to r ⊤ Π ⊥ r  which is exactly theresidual error in r that our representation cannot capture.
Revised: When γ = 0 , i.e we just perform reward regression, this is equal to ∥ Π ⊥ r ∥ 2Ξ which is exactly theresidual error in r that our representation cannot capture.

Original: When ρ > 1 , Π ⊥ = 

Revised: When ρ > 1 , Π ⊥ =

Original: so the limit is zeroas we expect from a model able to interpolate the data.
Revised: so the limit is zeroas we expect from a model able to interpolate the data.

Original: However, there is a behavior here not presentin supervised learning.
Revised: However, there is a behavior here not presentin supervised learning.

Original: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p closeto but less than n , a peaking behavior of the estimation error can happen.
Revised: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p closeto but less than n , a peaking behavior of the estimation error can happen.

Original: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased
Revised: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased

Original: [Equation]
Revised: [Equation]

Original: This phenomenon might appear counter-intuitive as we would expected our error to decrease withp .
Revised: This phenomenon might appear counter-intuitive as we would expected our error to decrease withp .

Original: We provide a few insights to make sense of this.
Revised: We provide a few insights to make sense of this.

Original: First, when γ increases, so does the scale of the
Revised: First, when γ increases, so does the scale of the

Original: Q-function, so we can expect our errors to get bigger.
Revised: Q-function, so we can expect our errors to get bigger.

Original: Second, using random features Φ informallyhas a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from11 − γ to 1 1 − γρ .
Revised: Second, using random features Φ informallyhas a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from11 − γ to 1 1 − γρ .

Original: This can be observed in the proof in appendix.
Revised: This can be observed in the proof in appendix.

Original: Finally, peaking behaviors of TD suchas this one have been observed in the community [44] but not theoretically explained in a generalsetting as far as we know.
Revised: Finally, peaking behaviors of TD suchas this one have been observed in the community [44] but not theoretically explained in a generalsetting as far as we know.

Original: 4.4 Beyond stationary sampling
Revised: 4.4 Beyond stationary sampling

Original: Until now, we always mentioned n as the number of states with the assumption that Ξ was nonsingular, i.e that our off-policy sampling visited all the states .
Revised: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξwas non-singular, i.e that our off-policy sampling visited all the states .

Original: Here, we take a closer look at whathappens when this is not the case.
Revised: Here, we take a closer lookat what happens when this is not the case.

Original: In particular, we study a simple toy model where we assume Ξonly visits n vis ( t ) states at time t with uniform probability.
Revised: In particular, we study a simple toy model where weassume Ξ only visits n vis ( t ) states at time t with uniform probability.

Original: This simply changes all the n by n vis ( t ) inthe previous results.
Revised: This simply changes all then by n vis ( t ) in the previous results.

Original: As often observed during training, at t = 0 , π might start as a more exploratorypolicy ( n vis ( t = 0) high) and over time converges on a more deterministic solution ( n vis ( t ) low).
Revised: As often observed during training, at t = 0 , π might start as amore exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution( n vis ( t ) low).

Original: This can be problematic when performing TD learning with a fixed stepsize η .
Revised: This can be problematic when performing TD learning with a fixed stepsize η .

Original: In that case, underthe spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis [Equation] t ) (cid:1) 2  or equivalently[Equation]
Revised: In thatcase, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 →2 / λ max or equivalently

Original: vis

Original: [Equation]
Revised: [Equation]

Original: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .
Revised: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .

Original: We conjecture this phenomenon may  related to the popularity of adaptative step size methods in RLover fixed step size ones.
Revised: We conjecture this phenomenon may be related to the popularity of adaptative step size methods in RL over fixed step size ones.

Original:  Experiments
Revised:  Experiments

Original: In all the experiments we use Φ i.i.d with normal entries.
Revised: In all the experiments we use Φ i.i.d with normal entries.

Original: Per the universality property, our theoreticalresults hold for all distributions satisfying the hypotheses in Section 2.2.
Revised: Per the universality property, our theoreticalresults hold for all distributions satisfying the hypotheses in Section 2.2.

Original: Study of thespectral distribution
Revised: Study of the

Revised: spectral distribution:

Original: On Figure 1, we validate the theory and approximationsof Section 3.
Revised: On Figure 1, we validate the theory and approximationsof Section 3.

Original: For an asymmetric random graph n = 5000 , µ = 1 / 2 , the spectrum of P (a) hasan eigenvalue at 1 and the rest of order 1 √ n [47].
Revised: For an asymmetric random graph n = 5000 , µ = 1 / 2 , the spectrum of P (a) hasan eigenvalue at 1 and the rest of order 1 √ n [47].

Original: As predicted, the spectrum of the TD operatormatches closely a spiked MP distribution (b).
Revised: As predicted, the spectrum of the TD operatormatches closely a spiked MP distribution (b).

Original: When the graph does not satisfy the well-connectedproperty, for instance on a 4 rooms domain ( n = 1600 , π stochastic 6 ) (c) the spectral distributionis more complex (d).
Revised: When the graph does not satisfy the well-connectedproperty, for instance on a 4 rooms domain ( n = 1600 , π stochastic 6 ) (c) the spectral distributionis more complex (d).

Original: Despite this, we show that the empirical and theoretical (eq.
Revised: Despite this, we show that the empirical and theoretical (eq.

Original: values forλ +min and λ max on the 4 rooms domain ( n = 256 ) closely match, both for Ξ uniform or highlyill-conditioned with evenly spaced ξ i so that max
Revised: values forλ +min and λ max on the 4 rooms domain ( n = 256 ) closely match, both for Ξ uniform or highlyill-conditioned with evenly spaced ξ i so that max

Original: i ξ i min i ξ i = 2000 .
Revised: i ξ i min i ξ i = 2000 .

Original: While (b) and (d) show how violatingthe asymptotically well-connected property breaks our spiked MP distribution result, we show on(e) and (f) that the extreme eigenvalues, which are the ones that matter for our optimization results,are still well predicted, even in more complex settings.
Revised: While (b) and (d) show how violatingthe asymptotically well-connected property breaks our spiked MP distribution result, we show on(e) and (f) that the extreme eigenvalues, which are the ones that matter for our optimization results,are still well predicted, even in more complex settings.

Original:  of TD : On Figure 2 we discuss the results of Section 4.
Revised: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.

Original: We use againa 4 rooms domain with a fixed stochastic policy and study the Bellman error during training.
Revised: We use again a4 rooms domain with a fixed stochastic policy and study the Bellman error during training.

Original: On 
Revised: On all

Original: [Figure]
Revised: [Figure]

Original: b), the distribution bleeds out of the MP support, yet the minimum eigenvalue is still close to the spike (d).
Revised: b), the distribution bleeds out of the MP support, yet the minimum eigenvalue is still close to the spike (d).

Original: On (e) and (f) we compare our prediction of the extreme eigenvalues λ +min and λ max of the TD operator on the 4 rooms domain [46] (b) as a function of p / n .
Revised: On (e) and (f) we compare our prediction of the extreme eigenvalues λ +min and λ max of the TD operator on the 4 rooms domain [46] (b) as a function of p / n .

Original: We plot the empirical extreme eigenvalues (dotted) vs (e) MP theoretical ones (eq. (3)) for Ξ uniform and (f) approximation in eq.
Revised: We plot the empirical extreme eigenvalues (dotted) vs (e) MP theoretical ones (eqs. 3, 6) for Ξ uniform and (f) approximation in eq.

Original: (9) for Ξ non-uniform.
Revised: for Ξ non-uniform.

Original: Shaded areas are a 95% confidence interval when randomizing Φ and π (10 seeds).
Revised: Shaded areas are a 95% confidence interval when randomizing Φ and π (10 seeds)

Original: [Figure]
Revised: [Figure]

Original: We all subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as thecurves for low T are much closer to the asymptotic solution T = + ∞ (computed analytically) thanfor ρ ≈ 1 .
Revised: subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curvesfor low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than forρ ≈ 1 .

Original: This is in line with the result of section 4.2 for a spike model: the ratio of eigenvalueswhich governs the optimization speed is λ max / λ +min
Revised: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalueswhich governs the optimization speed is λ max / λ +min

Original: → 1 for ρ ≪ 1 and to 1 1 − γ for ρ ≫ 1 while itdiverges for ρ → 1 .
Revised: → 1 for ρ ≪ 1 and to 1 1 − γ for ρ ≫ 1 while itdiverges for ρ → 1 .

Original: This leads to a classical double descent phenomenon on (a) and (b) purely dueto optimization as predicted by [7].
Revised: This leads to a classical double descent phenomenon on (a) and (b) purely dueto optimization as predicted by [7].

Original: On (a) and (b) because either γ or d ⊤ π r is small, the terms oforder 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.
Revised: On (a) and (b) because either γ or d ⊤ π r is small, the terms oforder 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.

Original: On (c), the term 1 1 − γ ˜ ρ 279 dominates and we observe the peaking behavior of the estimation error (Section 4.3).
Revised: On (c), the term 1 1 − γ ˜ ρ 280 dominates and we observe the peaking behavior of the estimation error (Section 4.3).

Original: Furthermore
Revised: Furthermore

Original: [Figure]
Revised: [Figure]

Original: on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match theobserved peaking behavior.
Revised: on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match theobserved peaking behavior.

Original: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD( p = 200 ).
Revised: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD( p = 200 ).

Original: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .
Revised: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .

Original: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze whichcombination of step size η and ϵ would lead to convergence.
Revised: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze whichcombination of step size η and ϵ would lead to convergence.

Original: On (a), we show the discounted statedistribution for various ϵ where larger ϵ lead to more exploration.
Revised: On (a), we show the discounted statedistribution for various ϵ where larger ϵ lead to more exploration.

Original: While our theoretical argumentin Section 4.4 was about the number of visited states, we take here a more general and realistic approach where the state visitation is controlled by the greediness of our policy.
Revised: While our theoretical argumentin Section 4.4 was about the number of visited states, we take here a more general and realisticapproach where the state visitation is controlled by the greediness of our policy.

Original: Typically, during theoptimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.
Revised: Typically, duringthe optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.

Original: On (b) we show  probability of diverging (averaged over 50 samples of Φ ) measured by computingthe spectral radius of our iteration matrix.
Revised: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured bycomputing the spectral radius of our iteration matrix.

Original: As predicted, for a fixed η , when ϵ decreases, we mightgo through a phase transition where the TD iteration, stable at the beginning, becomes unstable.
Revised: As predicted, for a fixed η , when ϵ decreases,we might go through a phase transition where the TD iteration, stable at the beginning, becomesunstable.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have analyzed the spectral distribution of the Temporal Difference operator whenusing random features and modeling the Markov transition matrix as the one of a random graph.
Revised: In this work, we have analyzed the spectral distribution of the Temporal Difference operator whenusing random features and modeling the Markov transition matrix as the one of a random graph.

Original: This characterization allowed us to make predictive theories for the optimization error and stabilityof TD.
Revised: Thischaracterization allowed us to make predictive theories for the optimization error and stability of TD.

Original: Notably we are able to predictthe how fast we converge and whether the solution foundis accurate as a function of the ratio between the number of parameters and the number of stateaction pairs.
Revised: Notably we are able to predict how fast we converge and whether the solution found is accurate asa function of the ratio between the number of parameters and the number of state-action pairs.

Original: In particular, we highlighted and theoretically explained several phenomena specific toreinforcement learning such as the peaking behavior of the estimation error of TD or how changingthe policy might cause divergence when using a fixed step, even in the on-policy setting.
Revised: Inparticular, we highlighted and theoretically explained several phenomena specific to reinforcementlearning such as the peaking behavior of the estimation error of TD or how changing the policymight cause divergence when using a fixed step size, even in the on-policy setting.

Original: We believe our work paves to way toward an average-case theory of reinforcement learning algorithms in the same way this was done for supervised learning and used to understand deep learning.
Revised: We believe our work paves to way toward an average-case theory of reinforcement learning algorithms in the same way this was done for supervised learning and used to understand deep learning.

Original: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations ofthe extreme eigenvalues of the MP law [41] (iii) extend our results to neural networks in the Neural
Revised: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations ofthe extreme eigenvalues of the MP law [41] (iii) extend our results to neural networks in the Neural

Original: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms. 
Revised: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms. Checklist

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Revised: contributions and scope?

Revised: [Yes](b) Did you describe the limitations of your work?

Revised: [Yes](c)

Revised: Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] Thiswork is a theoretical model for TD(0) in the linear setting.

Revised: As such we do not think

Revised: there are any applications with a societal impact to be foreseen.(d)

Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] Some finer

Revised: details can be found in the proofs in appendices.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] Statements such

Revised: as Lemmas or propositions have a full proof in appendix.

Revised: Informal statements andapproximations have a derivation in appendix where we make clear what the approxi-

Revised: mation/asssumption is.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Revised: perimental results (either in the supplemental material or as a URL)?

Revised: [Yes] Colab

Revised: notebooks are supplied to reproduce the figures.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Revised: [Yes]

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Revised: [N/A]

Revised: Used public Colab notebook

Revised: compute.4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Revised: (a)

Revised: If your work uses existing assets, did you cite the creators?

Revised: [N/A]

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data

Revised: you’re using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifi-

Revised: able information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with

Revised: human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]