Original: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation
Revised: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation

Original: Abstract
Revised: Abstract

Original: Much of the recent successes of deep learning can be attributed to scaling up thesize of the networks to the point where they often are vastly overparameterized.
Revised: Much of the recent successes of deep learning can be attributed to scaling up the size of the networks to the point where they often are vastly overparameterized.

Original: Thus, understanding the role of overparameterization is of increasing importance.
Revised: Thus, understanding the role of overparameterization is of increasing importance.

Original: While predictive theories have been developed for supervised learning, little isknown about the Reinforcement Learning case.
Revised: While predictive theories have been developed for supervised learning, little is known about the Reinforcement Learning case.

Original: In this work, we take a theoreticalapproach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.
Revised: In this work, we take a theoretical approach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.

Original: We leverage tools from random matrixtheory and random graph theory to obtain a characterization of the spectrum of the
Revised: We leverage tools from random matrix theory and random graph theory to obtain a characterization of the spectrum of the

Original: TD operator.
Revised: TD operator.

Original: We use this result to study the stability and optimization dynamicsof TD learning as a function of the number of parameters
Revised: We use this result to study the stability and optimization dynamics of TD learning as a function of the number of parameters.

Original:  Introduction
Revised:  Introduction

Original: Occam’s razor principle states that among many plausible explanations, the simplest one is themost likely to be true.
Revised: Occam’s razor principle states that among many plausible explanations, the simplest one is the most likely to be true.

Original: In statistics and machine learning, this is often interpreted as models witha restricted number of parameters should be privileged as they generalize better [1].
Revised: In statistics and machine learning, this is often interpreted as models with a restricted number of parameters should be privileged as they generalize better [1].

Original: This view issupported by the classical bias-variance trade-off which implies that models that are too flexibleare bound to overfit [2].
Revised: This view is supported by the classical bias-variance trade-off which implies that models that are too flexible are bound to overfit [2].

Original: Yet, deep neural networks with more parameters than training data pointsachieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.
Revised: Yet, deep neural networks with more parameters than training data points achieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.

Original: A significant understanding of this apparent tension has been reached in recent years.
Revised: A significant understanding of this apparent tension has been reached in recent years.

Original: A largebody of work using tools from random matrix theory has shown that, in models with randomfeatures, overparameterization does not lead to overfitting but instead better generalization [6].
Revised: A large body of work using tools from random matrix theory has shown that, in models with random features, overparameterization does not lead to overfitting but instead better generalization [6].

Original: Thisphenomenon is known as double descent and unifies the traditional and modern understanding ofbias and variance.
Revised: This phenomenon is known as double descent and unifies the traditional and modern understanding of bias and variance.

Original: Importantly, [7] has shown that the double descent phenomenon is in large partexplained by purely optimization rather than label noise.
Revised: Importantly, [7] has shown that the double descent phenomenon is in large part explained by purely optimization rather than label noise.

Original: Following their insight, we mainly take anoptimization approach in the rest of our paper.
Revised: Following their insight, we mainly take an optimization approach in the rest of our paper.

Original: While we now have a theory of overparameterization in supervised learning, the theoretical analysisof overparameterization in the reinforcement learning (RL) setting is largely unexplored [8].
Revised: While we now have a theory of overparameterization in supervised learning, the theoretical analysis of overparameterization in the reinforcement learning (RL) setting is largely unexplored [8].

Original: Indeed,
Revised: Indeed,

Original: RL poses several new additional challenges.
Revised: RL poses several new additional challenges.

Original: First, the agent has to interact with an environment whose structure may be complex.
Revised: First, the agent has to interact with an environment whose structure may be complex.

Original: Second, each state may not  sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.
Revised: Second, each state may not be sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.

Original: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.
Revised: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.

Original: More precisely, we study the behavior of Temporal Difference learning [9] which has become ubiquitous for policy evaluation in modern Deep RL [10, 11]
Revised: More precisely, we study the behavior of Temporal Difference learning [9] which has become ubiquitous for policy evaluation in modern Deep RL [10, 11].

Original: Our contributions are, in the limit of large number of states and parameters: 1) We introduce aclass of random graphs for which we can compute analytically the spectrum of the TD operatorwhen using uniform off-policy sampling, 2) we derive approximations for the largest and smallesteigenvalue in the non-uniform case.
Revised: Our contributions are, in the limit of large number of states and parameters: 1) We introduce a class of random graphs for which we can compute analytically the spectrum of the TD operator when using uniform off-policy sampling, 2) we derive approximations for the largest and smallest eigenvalue in the non-uniform case.

Original: Furthermore, we analyze the behavior of the traditional Mean
Revised: Furthermore, we analyze the behavior of the traditional Mean

Original: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a doubledescent phenomenon similar to supervised learning but with a multiplicative factor depending onthe discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displaysa peaking behavior near the interpolation threshold (when the number of parameters approachesthe number of states) that is specific to RL.
Revised: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a double descent phenomenon similar to supervised learning but with a multiplicative factor depending on the discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displays a peaking behavior near the interpolation threshold (when the number of parameters approaches the number of states) that is specific to RL.

Original: Finally, 5) we showcase how the non-stationarity of thepolicy can make the optimization of the value function unstable.
Revised: Finally, 5) we showcase how the non-stationarity of the policy can make the optimization of the value function unstable.

Original: All of these points are motivatedtheoretically and empirically validated on small gridworlds MDPs.
Revised: All of these points are motivated theoretically and empirically validated on small gridworlds MDPs.

Original:  Background2.1 Value estimationDefinition:
Revised:  Background 2.1 Value estimation

Revised: Definition:

Original: We place ourselves in the Markov Decision Process (MDP) setting [12] definedby {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈47 R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.
Revised: We place ourselves in the Markov Decision Process (MDP) setting [12] defined by {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈ R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.

Original: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.
Revised: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.

Original: Fora given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be ableto estimate the expected discounted return q π ( s, a) =
Revised: For a given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be able to estimate the expected discounted return q π ( s, a

Original: [-E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s 
-]
Revised: [-E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s 
-][Equation]

Original: =
Revised: =

Original: s, a 0 = a ] .
Revised: [Equation]

Original: This quantity is also known as the state-action value or the Q-function.
Revised: This quantity is also known as the state-action value or the Q-function.

Original: Furthermore, it can be written as the solution of the Bellman equation [13] 1 : q π ( s, a ) =
Revised: Furthermore, it can be written as the solution of the Bellman equation [13] 1 : q π ( s, a ) =

Original: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .
Revised: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .

Original: Bydefining P π ∈ R |S|·|A|×|S|·|A|
Revised: By defining P π ∈ R |S|·|A|×|S|·|A|

Original: the state-action to state-action transition matrix, we can write it inmatrix form 
Revised: the state-action to state-action transition matrix, we can write it in matrix form q

Original: [Equation]
Revised: [Equation]

Original: Linear function approximation: While Equation (1) can be solved efficiently when r , P π areknown and when n = |S|·|A| is not too large, in many cases this cannot be computed.
Revised: Linear function approximation: While Equation (1) can be solved efficiently when r , P π are known and when n = |S|·|A| is not too large, in many cases this cannot be computed.

Original: One solutionis to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .
Revised: One solution is to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .

Original: This is referred to as the Linear Function Approximation (LFA) setting.
Revised: This is referred to as the Linear Function Approximation (LFA) setting.

Original: Temporal Difference with LFA:
Revised: Temporal Difference with LFA:

Original: Estimating the Q-function in the Linear Function Approximationsetting can be done using the well-known algorithm TD(0)
Revised: Estimating the Q-function in the Linear Function Approximation setting can be done using the well-known algorithm TD(0)

Original: [9] whose expected update is
Revised: [9] whose expected update is

Original: [Equation]
Revised: [Equation]

Original: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summingto 1 , weighting each state-action by their probability of being sampled.
Revised: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summing to 1 , weighting each state-action by their probability of being sampled.

Original: Contrary to gradient descentwith an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φmight have a negative eigenvalue.
Revised: Contrary to gradient descent with an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φ might have a negative eigenvalue.

Original: This phenomenon is known as the deadly triad as it can onlyhappen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on thenext value as TD(0) does [14, 15, 16].
Revised: This phenomenon is known as the deadly triad as it can only happen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on the next value as TD(0) does [14, 15, 16].

Original: 2.2 Random Matrix Theory
Revised: 2.2 Random Matrix Theory

Original: Wishart matrices:
Revised: Wishart matrices:

Original: Extremely relevant in the study of least squares with random features arematrices from the Wishart distribution [17].
Revised: Extremely relevant in the study of least squares with random features are matrices from the Wishart distribution [17].

Original: These matrices can be written as 1 n Φ ⊤ Φ for Φ an × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.
Revised: These matrices can be written as 1 n Φ ⊤ Φ for Φ a n × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.

Original: In that case, when p, n → ∞ and the ratio converges to a finitelimit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [18] (MP) distribution whose density can be written as
Revised: In that case, when p, n → ∞ and the ratio converges to a finite limit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [18] (MP) distribution whose density can be written as

Original: [Equation]
Revised: [Equation]

Original: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x )are the 0 eigenvalues, which happens when p ≥ n , λ max
Revised: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x ) are the 0 eigenvalues, which exist when p ≥ n , λ max

Original: =
Revised: =

Original: (1 + √ ρ ) 2 , the largest, and λ +min =(1 − √ ρ ) 2 , the smallest non-zero eigenvalue.
Revised: (1 + √ ρ ) 2 , the largest, and λ +min = (1 − √ ρ ) 2 , the smallest non-zero eigenvalue.

Original: Wigner-type matrices: [19] historically introduced and studied symmetric n × n matrices ofthe form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance fori ≥ j 2 .
Revised: Wigner-type matrices: [19] historically introduced and studied symmetric n × n matrices of the form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance for i ≥ j 2 .

Original: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) =12 π √ 4 − x 2 1 [ − 2 , 2] dx .
Revised: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) = 12 π √ 4 − x 2 1 [ − 2 , 2] dx .

Original: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [21].
Revised: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [21].

Original: 2.3 Random graphs
Revised: 2.3 Random graphs

Original: Random graphs are useful models to infer structural or spectral properties of typical graphs.
Revised: Random graphs are useful models to infer structural or spectral properties of typical graphs.

Original: Theearliest theoretical analyses of random graphs that we are aware of were done independently by [22]and [23].
Revised: The earliest theoretical analyses of random graphs that we are aware of were done independently by [22] and [23].

Original: The model introduced in [22] is a random graph where each of the n nodes is connectedat random to d others.
Revised: The model introduced in [22] is a random graph where each of the n nodes is connected at random to d others.

Original: The second model [23] is slightly different as each node is independentlyconnected to any other with a probability µ ∈ ]0 , 1[ .
Revised: The second model [23] is slightly different as each node is independently connected to any other with a probability µ ∈ ]0 , 1[ .

Original: We refer to the first one as the d -regular Erd˝osR ´ enyi model G ( n, d ) and the second one as the Erd ˝ os-R ´ enyi-Gilbert G ( n, µ ) model.
Revised: We refer to the first one as the d -regular Erd˝osR´enyi model G ( n, d ) and the second one as the Erd˝os-R´enyi-Gilbert G ( n, µ ) model.

Original: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct amatrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rowssumming to 1 ).
Revised: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct a matrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rows summing to 1 ).

Original: P corresponds to the Markov transition matrix of a random walk on the graph and isconnected to the (left)-normalized Laplacian L by L =
Revised: P corresponds to the Markov transition matrix of a random walk on the graph and is connected to the (left)-normalized Laplacian L by L =

Original: I n − P .
Revised: I n − P .

Original: For many classes of random graphs,the spectrum of these quantities is known [24, 25].
Revised: For many classes of random graphs, the spectrum of these quantities is known [24, 25].

Original:  Spectrum of TD: from random graphs to spiked models 
Revised:  Spectrum of TD: from random graphs to spiked models 3.1 From Ordinary Least Squares to TD(0)

Original: 3.1 From Ordinary Least Squares to TD(0)

Original: Random matrix models have been used in supervised learning, especially in the simple least squaresmodel [26, 27, 28], to explain phenomena arising in deep learning.
Revised: Random matrix models have been used in supervised learning, especially in the simple least squares model [26, 27, 28], to explain phenomena arising in deep learning.

Original: Indeed, while much simpler thanlarge neural networks, these models can shed light on the optimization and generalization behaviorof vastly overparameterized models [29, 30], something that most traditional complexity measurescould not [31, 32].
Revised: Indeed, while much simpler than large neural networks, these models can shed light on the optimization and generalization behavior of vastly overparameterized models [29, 30], something that most traditional complexity measures could not [31, 32].

Original: Using gradient descent on a least square problem, our parameter update can be written as
Revised: Using gradient descent on a least square problem, our parameter update can be written as

Original: [Equation]
Revised: [Equation]

Original: (4) for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.
Revised: for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.

Original: When choosing thelearning rate η optimally, the convergence rate is expressed as λ [Equation] for λ +min , respectivelyλ max , the smallest, resp.
Revised: When choosing the learning rate η optimally, the convergence rate is expressed as λ max / λ +min − 1 λ max / λ +min +1 for λ +min , respectively λ max , the smallest, resp.

Original: largest, non-zero eigenvalue of Φ ⊤ Φ [33].
Revised: largest, non-zero eigenvalue of Φ ⊤ Φ [33].

Original: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we willconsider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.
Revised: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we will consider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.

Original: In particular, we haveanalytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).
Revised: In particular, we have analytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).

Original: Whenlim p,n →∞ p / n = ρ , we have λ max / λ +min
Revised: When lim p,n →∞ p / n = ρ , we have λ max / λ +min

Original: →
Revised: →

Original: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .
Revised: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .

Original: This function has asymptotes at 1 , forp
Revised: This function has asymptotes at 1 , for p

Original: = o ( n ) or n = o ( p ) and diverges when n = p .
Revised: = o ( n ) or n = o ( p ) and diverges when n = p .

Original: Note that this peak-shaped function can also beobserved empirically in more complex models such as kernel methods [34] and neural networks inthe limit of infinite width as they are in the Neural Tangent Kernel regime [35, 36].
Revised: This peak-shaped function can also be observed empirically in more complex models such as kernel methods [34] and neural networks in the limit of infinite width as they are in the Neural Tangent Kernel regime [35, 36].

Original: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would bereplaced by Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would be replaced by Φ ⊤ Ξ( I n − γ P π )Φ .

Original: While [8] derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.
Revised: While [8] derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.

Original: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.
Revised: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.

Original: While we can model Φ as ani.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allowus to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ . 
Revised: While we can model Φ as an i.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allow us to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ . For general Ξ and P π no closed form solution for the spectrum exists.

Original: We believe this is one ofthe key reasons why overparameterization in Temporal Difference learning has not been studiedextensively.
Revised: We believe this is one of the key reasons why overparameterization in Temporal Difference learning has not been studied extensively.

Original: Thus, finding good typical models for P π is of utmost importance, and the next sectionwill introduce a class of models for which the spectrum of TD can be computed.
Revised: Thus, finding good typical models for P π is of utmost importance, and the next section will introduce a class of models for which the spectrum of TD can be computed.

Original: 3.2 P π as the Markov transition matrix of a random graph
Revised: 3.2 P π as the Markov transition matrix of a random graph

Original: In RL, P π is the Markov transition matrix of a random walk when sampling from π .
Revised: In RL, P π is the Markov transition matrix of a random walk when sampling from π .

Revised: We will assume in the rest of the paper that the Markov chain is irreducible and aperiodic so that P π has a unique stationary distribution denoted d π .

Revised: As we wish to understand the expected behavior of the TD algorithm, we propose in this subsection a simple yet expressive model of P π that will allow us to do so.

Original: As randomgraphs are used to model properties of typical graphs, a candidate for a model of P π would be to takethe Markov transition matrix of a random graph.
Revised: As random graphs are used to model properties of typical graphs, a candidate for a model of P π would be to take the Markov transition matrix of a random graph.

Original: However, random graphs, at least the Erd˝os-R´enyi(- Gilbert) type ones, are not able natively to take into account the impact of π , most importantly thefact that some state-actions are visited more often than others under d π , the stationary distributionof π .
Revised: However, random graphs, at least the Erd˝os-R´enyi(-Gilbert) type ones, are not able natively to take into account the impact of π , most importantly the fact that some state-actions are visited more often than others under d π , the stationary distribution of π .

Original: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markovtransition matrix of a G ( n, µ ) graph deformed by d π .
Revised: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markov transition matrix of a G ( n, µ ) graph deformed by d π .

Original: Let us start with an undirected graph of the type G ( n, µ ) .
Revised: Let us start with an undirected graph of the type G ( n, µ ) .

Original: To take into account the state-actionvisitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .
Revised: To take into account the state-action visitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .

Original: Now, we define thedeformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.
Revised: Now, we define the deformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.

Original: Notehere that when d π is uniform, we revert back to the original case A π = A .
Revised: When d π is uniform, we revert back to the original case A π = A .

Original: This construction keepsthe same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.
Revised: This construction keeps the same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.

Original: We can informally write the Markov transition matrix associated to this deformed graph as 4ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n Wis a Wigner-type matrix with zero mean and unit variance.
Revised: We can informally write the Markov transition matrix associated with this deformed graph as 4 ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1 as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n W is a Wigner-type matrix with zero mean and unit variance.

Original: It can be tempting to understand underwhich conditions the stochastic part becomes negligible and what it entails.
Revised: It can be tempting to understand under which conditions the stochastic part becomes negligible and what it entails.

Original: As we will see shortly,this setting will reveal itself to be of importance when studying the spectrum of TD.
Revised: As we will see shortly, this setting will reveal itself to be of importance when studying the spectrum of TD.

Original: Definition 1 (Asymptotically well-connected graphs) .
Revised: Definition 1 (Asymptotically well-connected graphs) .

Original: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its
Revised: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its

Original: Markov transition matrix.
Revised: Markov transition matrix.

Original: The choice for the name of Definition 1 comes from the fact that | λ 2 | is related to the presence ofbottlenecks in the graph through Cheeger’s inequality [37].
Revised: The choice for the name of Definition 1 comes from the fact that the second smallest eigenvalue of the Laplacian is related to the presence of bottlenecks in the graph through Cheeger’s inequality [37].

Original: Example.
Revised: Example.

Original: Graphs of the type G ( n, µ ) (resp.
Revised: Graphs of the type G ( n, µ ) (resp.

Original: G ( n, d ) ) are asymptotically well-connected if theirexpected degree d = nµ (resp.
Revised: G ( n, d ) ) are asymptotically well-connected if their expected degree d = nµ (resp.

Original: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [24, 25].
Revised: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [24, 25].

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π , then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: Note here that this again generalizes the uniform case as, when d π is uniform, the conditionbecomes equivalent to 1 ∈ o ( d ) as in the example.
Revised: This again generalizes the uniform case as, when d π is uniform, the condition becomes equivalent to 1 ∈ o ( d ) as in the example.

Original: In the non-uniform case, this condition entailsthat either d π spreads efficiently across states, or the degree d rises fast enough that those states arestill visited sufficiently often.
Revised: In the non-uniform case, this condition entails that either d π spreads efficiently across states, or the degree d rises fast enough that those states are still visited sufficiently often.

Revised: Note that the asymptotically well-connected condition is also satisfied if we use longer back-ups of length k such that k grows with n to + ∞ , i.e 1 ∈

Revised: o ( k ) for the TD update.

Revised: See

Revised: Lemma B.2 for details.

Original: In that case, we have
Revised: In those cases, we have

Original: [Equation]
Revised: [Equation]

Original: This approximation is particularly interesting as it asymptotically conserves  left  right  1 n  the original P π associated to the eigenvalue 1 .
Revised: This approximation is particularly interesting as it asymptotically conserves the left and right eigenvectors, d π and 1 n , of the original P π associated to the eigenvalue 1 .

Original: Furthermore, if allstate are reachable, the sequence of matrix power iterates of P π ,  P  converges to 
Revised: Furthermore, if all stateactions are reachable, the sequence of matrix power iterates of P π , (cid:0) P π (cid:1) k converges to 1 n d ⊤ π .

Original: While we are not the first to use tools from graph theory in the context of RL, related work weare aware of [38, 39, 40, 41] takes a different approach and leverages eigenvectors of the Markovtransition matrix (or Laplacian) to construct better representations for the value function or usethem as options.
Revised: While we are not the first to use tools from graph theory in the context of RL, related works we are aware of [38, 39, 40, 41] take a different approach and leverage eigenvectors of the Markov transition matrix (or Laplacian) to construct better representations for the value function or use them as options.

Original: In contrast, we use spectral properties of P π to study the convergence of Temporal
Revised: In contrast, we use spectral properties of P π to study the convergence of Temporal

Original: Difference learning.
Revised: Difference learning.

Original: 3.3 Spiked Wishart model for TD with uniform sampling
Revised: 3.3 Spiked Wishart model for TD with uniform sampling

Original: Now that we have a simple random graph model for P π , we focus on estimating the spectrumof Φ ⊤ Ξ( I n − γ ˆ P π )Φ .
Revised: Now that we have a simple random graph model for P π , we focus on estimating the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ .

Original: For simplicity, we assume the off-policy sampling is done uniformly, i.eΞ = 1 n I n .
Revised: For simplicity, we assume the off-policy sampling is done uniformly, i.e Ξ = 1 n I n .

Original: While this assumption is not realistic in most RL settings, it is still of interest in thecontext of dynamic programming where we have access to all the states.
Revised: While this assumption is not realistic in most RL settings, it is still of interest in the context of dynamic programming where we have access to all the states.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to 
Revised: When ργ 2 ≥ 1 , there is a phase transition [42] for the minimum non-zero eigenvalue λ +min which separates from bulk and converges to [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =
Revised: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =

Original: ( (cid:112) 1 / γ 2 − 1) 2 .
Revised: ( (cid:112) 1 / γ 2 − 1) 2 .

Original: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .
Revised: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .

Original: In thatcase, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , theeffective horizon of the problem, as lim
Revised: In that case, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , the effective horizon of the problem, as lim

Original: ρ →∞ λ max / λ +min
Revised: ρ →∞ λ max / λ +min

Original: =
Revised: =

Original: 11 − γ .
Revised: 11 − γ .

Original: In the tabular regime, Φ = I n .
Revised: In the tabular regime, Φ = I n .

Original: As
Revised: As

Original: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .
Revised: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .

Original: This would give aworst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterizedcase above.
Revised: This would give a worst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterized case above.

Original: As these eigenvalues govern the optimization speed of TD, the vastly overparameterizedcase and the tabular can be thought of as comparable.
Revised: As these eigenvalues govern the optimization speed of TD, the vastly overparameterized case and the tabular can be thought of as comparable.

Original: 3.4 Non-uniform off-policy sampling
Revised: 3.4 Non-uniform off-policy sampling

Original: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assumingΞ = 1 n I n .
Revised: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assuming Ξ = 1 n I n .

Original: In this subsection we analyze the more general case where Ξ is not uniform.
Revised: In this subsection we analyze the more general case where Ξ is not uniform.

Original: However bydoing so, we lose guarantees and must resort to approximations.
Revised: However by doing so, we lose guarantees and must resort to approximations.

Original: First, we analyze the eigenvaluesof Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .
Revised: First, we analyze the eigenvalues of Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy asecular equation [43]
Revised: γ 1 n d ⊤ π ) satisfy a secular equation [43]

Original: [Equation]
Revised: [Equation]

Original: This expression is another form of the characteristic polynomial of a matrix whose roots are itseigenvalues.
Revised: This expression is another form of the characteristic polynomial of a matrix whose roots are its eigenvalues.

Original: However, this form enables us to derive approximations more easily.
Revised: However, this form enables us to derive approximations more easily.

Original: In particular,for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸
Revised: In particular, for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸

Original: = m , we have for λ ≈ ξ mthat 1 ξ i
Revised: = m , we have for λ ≈ ξ m that

Original: [Equation]
Revised: [Equation]

Original: ≈ 1 ξ i if i ̸ = m . From this assumption, we derive an approximation for the minimumeigenvalue ofΞ( I n − γ ˆ P π )
Revised: From this assumption, we derive an approximation for the minimum eigenvalue of

Original: [Equation]
Revised: [Equation]

Original: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution andthe discount factor.
Revised: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution and the discount factor.

Original: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which isthe product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .
Revised: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which is the product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .

Original: Note that a similar expression can bederived for the maximum eigenvalue under similar assumptions.
Revised: A similar expression can be derived for the maximum eigenvalue under similar assumptions.

Original: Now, again reasoning approximately, if that eigenvalue was very small compared to the other oneswho would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similarspiking behavior as before.
Revised: Now, again reasoning approximately, if that eigenvalue was very small compared to the other ones who would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similar spiking behavior as before.

Original: However, in that case, the spiking eigenvalue would be
Revised: However, in that case, the spiking eigenvalue would be

Original: [Equation]
Revised: [Equation]

Original: While less simple than the estimators developed in the last subsection, we will see in Section 5 thatit stillpredicts accurately  the smallest non-zero eigenvalue.
Revised: While less simple than the estimators developed in the last subsection, we will see in Section 5 that it still accurately predicts the smallest non-zero eigenvalue.

Original:  Optimizing with Temporal Difference learning 
Revised:  Optimizing with Temporal Difference learning 4.1 Decomposition of the error

Original: 4.1 Decomposition of the error

Original: Using results about the spectrum of the TD operator shown in the last section, we analyze thebehavior of the Mean Square Bellman Error L MSBE ( q t ) =
Revised: Using results about the spectrum of the TD operator shown in the last section, we analyze the behavior of the Mean Square Bellman Error L MSBE ( q t ) =

Original: 12 ∥
Revised: 12 ∥

Original: r + γ P π q t − q t ∥
Revised: r + γ P π q t − q t ∥

Original: 2Ξ during theoptimization.
Revised: 2Ξ during the optimization.

Original: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n
Revised: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n

Original: [Equation]
Revised: [Equation]

Original: the q -value asymptotically reached by TD(0), we can decompose this error into two components.
Revised: the q -value asymptotically reached by TD(0), we can decompose this error into two components.

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ),we have the following decomposition
Revised: (2) ), we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .
Revised: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Following [44], we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥
Revised: Following [44], we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥

Original: r + γ P π q TD − q TD 
Revised: r + γ P π q TD − q TD ∥

Original: ∥ Ξ as theestimation error.
Revised: Ξ as the estimation error.

Original: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π )in metric Ξ .
Revised: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π ) in metric Ξ .

Original: Note that, beyond this factor, we are not exactly in the same setting as [44] as theasymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, asknown in the RL community [45].
Revised: Beyond this factor, we are not exactly in the same setting as [44] as the asymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, as known in the RL community [45].

Original: Indeed, TD(0) finds the minimum of the Mean Square Projected Bellman Error (MSPBE) but not of the MSBE [46].
Revised: Indeed, TD(0) finds the minimum of the Mean Square Projected Bellman Error (MSPBE) but not of the MSBE [46].

Original: However in practice it is common to minimizethe MSBE as it does not require an expensive projection step.
Revised: However in practice it is common to minimize the MSBE as it does not require an expensive projection step.

Original: 4.2 The ordinary behavior of the approximation error
Revised: 4.2 The ordinary behavior of the approximation error

Original: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .
Revised: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and that its spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η =2 λ +min + λ max
Revised: Denoting by λ +min and λ max its smallest non-zero and largest eigenvalues, for η =

Original: [Equation]
Revised: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).
Revised: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).

Original: This is a slightly more general version of the linear convergence of gradient descent, however, as the
Revised: This is a slightly more general version of the linear convergence of gradient descent, however, as the

Original: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .
Revised: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .

Original: 4.3 The singular behavior of the estimation error
Revised: 4.3 The singular behavior of the estimation error

Original: Now we turn our attention to the second term; the estimation error which is the residual error after
Revised: Now we turn our attention to the second term; the estimation error which is the residual error after

Original: T → + ∞ steps.
Revised: T → + ∞ steps.

Original: In supervised learning, we expect this error to be the usual bias of our algorithmand thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possibleto interpolate the data.
Revised: In supervised learning, we expect this error to be the usual bias of our algorithm and thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possible to interpolate the data.

Original: However, in reinforcement learning, the behavior of this term can be quitedifferent, in particular it can be non-monotonous.
Revised: However, in reinforcement learning, the behavior of this term can be quite different, in particular it can be non-monotonous.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptoticallyrank one
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically rank one

Original: [Equation]
Revised: [Equation]

Revised: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π

Original:  When γ = 0 , i.e we just perform reward regression, this is equal to ∥
Revised: [Equation] Φ When γ = 0 , i.e we just perform reward regression, this is equal to ∥

Original: Π ⊥ r ∥ 2Ξ which is exactly theresidual error in r that our representation cannot capture.
Revised: Π ⊥ r ∥ 2Ξ which is exactly the residual error in r that our representation cannot capture.

Original: When ρ > 1 , Π ⊥ = 

Revised: When ρ > 1 , Π ⊥ =

Original: so the limit is zeroas we expect from a model able to interpolate the data.
Revised: so the limit is zero as we expect from a model able to interpolate the data.

Original: However, there is a behavior here not presentin supervised learning.
Revised: However, there is a behavior here not present in supervised learning.

Original: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p closeto but less than n , a peaking behavior of the estimation error can happen.
Revised: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p close to but less than n , a peaking behavior of the estimation error can happen.

Original: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased 
Revised: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased [Equation]

Original: [Equation]
Revised: [Equation]

Original: This phenomenon might appear counter-intuitive as we would expected our error to decrease withp .
Revised: This phenomenon might appear counter-intuitive as we would expected our error to decrease with p .

Original: We provide a few insights to make sense of this.
Revised: We provide a few insights to make sense of this.

Original: First, when γ increases, so does the scale of the
Revised: First, when γ increases, so does the scale of the

Original: Q-function, so we can expect our errors to get bigger.
Revised: Q-function, so we can expect our errors to get bigger.

Original: Second, using random features Φ informallyhas a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from11 − γ to 1 1 − γρ .
Revised: Second, using random features Φ informally has a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from 11 − γ to 1 1 − γρ .

Original: This can be observed in the proof in appendix.
Revised: This can be observed in the proof in appendix.

Original: Finally, peaking behaviors of TD suchas this one have been observed in the community [45] but not theoretically explained in a generalsetting as far as we know.
Revised: Finally, peaking behaviors of TD such as this one have been observed in the community [45] but not theoretically explained in a general setting as far as we know.

Original: 4.4 Beyond stationary sampling
Revised: 4.4 Beyond stationary sampling

Original: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξwas non-singular, i.e that our off-policy sampling visited all the states .
Revised: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξ was non-singular, i.e that our off-policy sampling visited all the states .

Original: Here, we take a closer lookat what happens when this is not the case.
Revised: Here, we take a closer look at what happens when this is not the case.

Original: In particular, we study a simple toy model where weassume Ξ only visits n vis ( t ) states at time t with uniform probability.
Revised: In particular, we study a simple toy model where we assume Ξ only visits n vis ( t ) states at time t with uniform probability.

Original: This simply changes all then by n vis ( t ) in the previous results.
Revised: This simply changes all the n by n vis ( t ) in the previous results.

Original: As often observed during training, at t = 0 , π might start as amore exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution( n vis ( t ) low).
Revised: As often observed during training, at t = 0 , π might start as a more exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution ( n vis ( t ) low).

Original: This can be problematic when performing TD learning with a fixed stepsize η .
Revised: This can be problematic when performing TD learning with a fixed stepsize η .

Original: In thatcase, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 →2 / λ max or equivalently
Revised: In that case, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 → 2 / λ max or equivalently

Original: [Equation]
Revised: [Equation]

Original: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .
Revised: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .

Original: We conjecture this phenomenon may be related to the popularity of adaptative step size methods in
Revised: We conjecture this phenomenon may be related to the popularity of adaptive step size methods in

Original: RL over fixed step size ones.
Revised: RL over fixed step size ones.

Original:  Experiments
Revised:  Experiments

Original: In all the experiments we use Φ i.i.d with normal entries.
Revised: In all the experiments we use Φ i.i.d with normal entries.

Original: Per the universality property, our theoreticalresults hold for all distributions satisfying the hypotheses in Section 2.2.
Revised: Per the universality property, our theoretical results hold for all distributions satisfying the hypotheses in Section 2.2.

Original: Study of the spectral distribution:
Revised: Study of the spectral distribution:

Original: On Figure 1, we validate the theory and approximations of
Revised: On Figure 1, we validate the theory and approximations of

Original: Section 3.
Revised: Section 3.

Original: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has aneigenvalue at 1 and the rest of order 1 / √ n [48].
Revised: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has an eigenvalue at 1 and the rest of order 1 / √ n [48].

Original: As predicted, the spectrum of the TD operatormatches closely a spiked MP distribution (b).
Revised: As predicted, the spectrum of the TD operator matches closely a spiked MP distribution (b).

Original: When the graph does not satisfy the well-connectedproperty, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distributionis more complex (d).
Revised: When the graph does not satisfy the well-connected property, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distribution is more complex (d).

Original: The relative error on λ +min is respectively 0 .
Revised: The relative error on λ +min is respectively 0 .

Original: 3% and 2% .
Revised: 3% and 2% .

Original: Despite this, weshow that the empirical and theoretical (eq.
Revised: Despite this, we show that the empirical and theoretical (eq.

Original: values for λ +min and λ max on the 4 rooms domain( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so thatmax i ξ
Revised: values for λ +min and λ max on the 4 rooms domain ( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so that max

Original: i  min i ξ i = 2000 .
Revised: i ξ i min i ξ i = 2000 .

Original: While (b) and (d) show how violating the asymptotically well-connected propertybreaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, whichare the ones that matter for our optimization results, are still well predicted, even in more complexsettings.
Revised: While (b) and (d) show how violating the asymptotically well-connected property breaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, which are the ones that matter for our optimization results, are still well predicted, even in more complex settings.

Original: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.
Revised: : On Figure 2 we discuss the results of Section 4.

Original: We use again a4 rooms domain with a fixed stochastic policy and study the Bellman error during training.
Revised: We use again a 4 rooms domain with a fixed stochastic policy and study the Bellman error during training.

Original: On all 
Revised: On all subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curves for low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than for

Original: [Figure]
Revised: [Figure]

Original: I m ag i na r y pa r t
Revised: I m ag i na r y pa r t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: subfigures of Figure 2, we observe that for ρ≫ 1 or ρ ≪ 1 , the optimization is faster as the curvesfor low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than forρ ≈ 1 .
Revised: ρ ≈ 1 .

Original: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalueswhich governs the optimization speed is λ max / λ +min → 1 for ρ ≪ 1 and to 1 1 − γ for ρ ≫ 1 while itdiverges for ρ → 1 .
Revised: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalues which governs the optimization speed is λ max / λ +min → 1 for ρ ≪ 1 and to 11 − γ for ρ ≫ 1 while it diverges for ρ → 1 .

Original: This leads to a classical double descent phenomenon on (a) and (b) purely dueto optimization as predicted by [7].
Revised: This leads to a classical double descent phenomenon on (a) and (b) purely due to optimization as predicted by [7].

Original: On (a) and (b) because either γ or d ⊤ π r is small, the terms oforder 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning. 
Revised: On (a) and (b) because either γ or d ⊤ π r is small, the terms of order 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning. On (c), the term [Equation]

Original: On (c), the term 1 1 − γ ˜ ρ283 dominates and we observe the peaking behavior of the estimation error (Section 4.3).
Revised: − [Equation] ˜ ρ dominates and we observe the peaking behavior of the estimation error (Section 4.3).

Original: Furthermore 
Revised: Furthermore on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match the observed peaking behavior.

Original: [Figure]
Revised: [Figure]

Original: on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match theobserved peaking behavior.

Original: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD( p = 200 ).
Revised: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD ( p = 200 ).

Original: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .
Revised: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .

Original: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze whichcombination of step size η and ϵ would lead to convergence.
Revised: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze which combination of step size η and ϵ would lead to convergence.

Original: On (a), we show the discounted statedistribution for various ϵ where larger ϵ lead to more exploration.
Revised: On (a), we show the discounted state distribution for various ϵ where larger ϵ lead to more exploration.

Original: While our theoretical argumentin Section 4.4 was about the number of visited states, we take here a more general and realisticapproach where the state visitation is controlled by the greediness of our policy.
Revised: While our theoretical argument in Section 4.4 was about the number of visited states, we take here a more general and realistic approach where the state visitation is controlled by the greediness of our policy.

Original: Typically, duringthe optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.
Revised: Typically, during the optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.

Original: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured bycomputing the spectral radius of our iteration matrix.
Revised: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured by computing the spectral radius of our iteration matrix.

Original: As predicted, for a fixed η , when ϵ decreases,we might go through a phase transition where the TD iteration, stable at the beginning, becomesunstable.
Revised: As predicted, for a fixed η , when ϵ decreases, we might go through a phase transition where the TD iteration, stable at the beginning, becomes unstable.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have analyzed the spectral distribution of the Temporal Difference operator whenusing random features and modeling the Markov transition matrix as the one of a random graph.
Revised: In this work, we have analyzed the spectral distribution of the Temporal Difference operator when using random features and modeling the Markov transition matrix as the one of a random graph.

Original: Thischaracterization allowed us to make predictive theories for the optimization error and stability of TD.
Revised: This characterization allowed us to make predictive theories for the optimization error and stability of TD.

Original: Notably we are able to predict how fast we converge and whether the solution found is accurate asa function of the ratio between the number of parameters and the number of state-action pairs.
Revised: Notably we are able to predict how fast we converge and whether the solution found is accurate as a function of the ratio between the number of parameters and the number of state-action pairs.

Original: Inparticular, we highlighted and theoretically explained several phenomena specific to reinforcementlearning such as the peaking behavior of the estimation error of TD or how changing the policymight cause divergence when using a fixed step size, even in the on-policy setting.
Revised: In particular, we highlighted and theoretically explained several phenomena specific to reinforcement learning such as the peaking behavior of the estimation error of TD or how changing the policy might cause divergence when using a fixed step size, even in the on-policy setting.

Original: We believe our work paves to way toward an average-case theory of reinforcement learning algorithms in the same way this was done for supervised learning and used to understand deep learning.

Original: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations ofthe extreme eigenvalues of the MP law [42] (iii) extend our results to neural networks in the Neural
Revised: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations of the extreme eigenvalues of the MP law [42] (iii) extend our results to neural networks in the Neural

Original: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms. 
Revised: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms. Acknowledgments

Revised: We thank Alexandre Pich´e for useful comments on an earlier version of this paper.

Original: B Appendix: Proofs
Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A]

Revised: This work is a theoretical model for TD(0) in the linear setting.

Revised: As such we do not think there are any applications with a societal impact to be foreseen.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] Some finer details can be found in the proofs in appendices.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] Statements such as Lemmas or propositions have a full proof in appendix.

Revised: Informal statements and approximations have a derivation in appendix where we make clear what the approximation/asssumption is.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] Colab notebooks are supplied to reproduce the figures.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [N/A]

Revised: Used public Colab notebook compute.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A] (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: A Appendix: Experiments

Revised: [Figure]

Revised: On Figure 4 we test the models developed in Section 4.3.

Revised: Note that the models were developed for a ˆ

Revised: P π approximately rank 1.

Revised: We can see on (a) that the models, both the exact and the simplified overestimate the true estimation error in a 14 × 14 4 rooms domain ( n = 784) for various p = 1 . . .

Revised: n .

Revised: However, if we were to do something akin to a k-step return, where we consider one action to be a sequence of k, this would be equivalent to changing ˆ P π to (cid:0) ˆ P π (cid:1) k .

Revised: In (b) and (c) we can see that taking a higher k improve our estimation.

Revised: Taking powers of the transition matrix has for effect to push down all the eigenvalues < 1 and as such make the graph ”better” connected.

Revised: All in all, while our formula is not precise in a realistic scenario, when considering an iterated setting, it recovers the estimation precisely.

Revised: Note that for Figure 4 we chose not to average over random Φ as it highlights the exactness of the rank one model developed in Section 4.3.

Revised: B Appendix: Proofs

Original: In the paper and in the proofs, we extend naturally the notation o ( · ) to matrices with the followingdefinition
Revised: In the paper and in the proofs, we extend naturally the notation o ( · ) to matrices with the following definition

Original: [Equation]
Revised: [Equation]

Original: for ∥ · ∥ a natural operator norm.
Revised: for ∥ · ∥ a natural operator norm.

Original: In this paper we only consider the 2-operator norm which corresponds to the largest singular value.
Revised: In this paper we only consider the 2-operator norm which corresponds to the largest singular value.

Original: B.1 Deformed random graph
Revised: B.1 Deformed random graph

Original: In Section 2.3 we derive informally an expression for the Markov transition matrix of a randomgraph taking into account d π .
Revised: In Section 2.3 we derive informally an expression for the Markov transition matrix of a random graph taking into account d π .

Original: Let us go into more details here, and explain why we called it aninformal derivation.
Revised: Let us go into more details here, and explain why we called it an informal derivation.

Original: Let us recall that A π = n AD π for A the original adjacency matrix and
Revised: Let us recall that A π = n AD π for A the original adjacency matrix and

Original: D π =diag ( d π ) the stationary distribution of π .
Revised: D π = diag ( d π ) the stationary distribution of π .

Original: To get ˆ P π we need to normalize the rows i = 1 , . . .
Revised: To get ˆ P π we need to normalize the rows i = 1 , . . .

Original: n of
Revised: n of

Original: A π by their sum, i.e the degree d i of each state.
Revised: A π by their sum, i.e the degree d i of each state.

Original: In the uniform case, A π = A , [48] provide first a sketch of proof (eq 1.6) based on the fact thatd i = D ii = (cid:80)
Revised: In the uniform case, A π = A , [48] provide first a sketch of proof (eq 1.6) based on the fact that d i = D ii = (cid:80)

Original: j A ij satisfies | d i − nµ | ∈
Revised: j A ij satisfies | d i − nµ | ∈

Original: o ( n ) by the law of large numbers.
Revised: o ( n ) by the law of large numbers.

Original: This allows to ”replace”
Revised: This allows to ”replace”

Original: D by nµ I n in the proofs which ties it back to the general Wigner/i.i.d case.
Revised: D by nµ I n in the proofs which ties it back to the general Wigner/i.i.d case.

Original: In the deformed case, we cannot invoke directly the law of large numbers.
Revised: In the deformed case, we cannot invoke directly the law of large numbers.

Original: Indeed, now d i =511 (cid:80)
Revised: Indeed, now d i = (cid:80)

Original: j A πij = n (cid:80)
Revised: j A πij = n (cid:80)

Original: j A ij d π ( j ) so we have a weighted sum instead of a simple one.
Revised: j A ij d π ( j ) so we have a weighted sum instead of a simple one.

Original: We need a concentration inequality for the weighted sum of random variables.
Revised: We need a concentration inequality for the weighted sum of random variables.

Original: One can be found in[49]
Revised: One can be found in [49]

Original: Lemma B.1 (From [49]) .
Revised: Lemma B.1 (From [49]) .

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: , X n be iid copies of a real random variable X that obeys, for some p ≥
Revised: , X n be iid copies of a real random variable X that obeys, for some p ≥

Original: P( | X | > u ) ≤ exp
Revised: P

Original: [Equation]
Revised: [Equation]

Original: − u p ) for all u > 0 .
Revised: − u p ) for all u > 0 .

Original: Then there exists L > 0 such that, for any s ∈ R n , and for q the conjugate of p ( 1 p + 1 q = 1 )
Revised: Then there exists L > 0 such that, for any s ∈ R n , and for q the conjugate of p ( 1 p + 1 q = 1 )

Original: [Equation]
Revised: [Equation]

Original: In our case, for X j = c · ( A ij − µ ) (where c is chosen so the the tail assumption is satisfied),
Revised: In our case, for X j = c · ( A ij − µ ) (where c is chosen so the the tail assumption is satisfied),

Original: [Equation]
Revised: [Equation]

Original: Thus, if ∥ d π ∥ qp and ∥
Revised: Thus, if ∥ d π ∥ qp and ∥

Original: d π ∥ 22 are ∈
Revised: d π ∥ 22 are ∈

Original: o (1) , and then reasoning on − X
Revised: o (1) , and then reasoning on − X

Original: j we have that | d i n − µ | is, almostsurely when n is large, smaller than any constant t > 0 , therefore o (1) .
Revised: j we have that | d i n − µ | is, almost surely when n is large, smaller than any constant t > 0 , therefore o (1) .

Original: Thus we get the sameargument D = nµ I n + o ( n ) .
Revised: Thus we get the same argument D = nµ I n + o ( n ) .

Original: Note here that in high dimension the stronger condition for d π will befor p = ∞ , ∥ d π ∥ ∞ ∈ o 
Revised: Note here that in high dimension the stronger condition for d π will be for p = ∞ , ∥ d π ∥ ∞ ∈ o (1) .

Original: As we already make a stronger assumption later, we have chosen not toinclude it in the main text.
Revised: As we already make a stronger assumption later, we have chosen not to include it in the main text.

Original: All of this allows to calculate ˆ P π by normalizing A π
Revised: All of this allows to calculate ˆ P π by normalizing A π

Original: [Equation]
Revised: [Equation]

Original: When the entries of A are iid Bernoulli of mean µ , we have σ 2 = µ (1 − µ ) and W has iid whitened
Revised: When the entries of A are iid Bernoulli of mean µ , we have σ 2 = µ (1 − µ ) and W has iid whitened

Original: Bernoulli entries.
Revised: Bernoulli entries.

Original: We consider this proof informal as there are a certain number of technicalities involved when controlling the deviation of all these random quantities (as we have sketched above for the expecteddegree).
Revised: We consider this proof informal as there are a certain number of technicalities involved when controlling the deviation of all these random quantities (as we have sketched above for the expected degree).

Original: B.2 Asymptotically well-connected graph
Revised: B.2 Asymptotically well-connected graph

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π , then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: Proof.
Revised: Proof.

Original: First, recall that
Revised: First, recall that

Original: [Equation]
Revised: [Equation]

Original: For ∥ · ∥ the 2-operator corresponding to the maximum singular value, we have:
Revised: For ∥ · ∥ the 2-operator corresponding to the maximum singular value, we have:

Original: [Equation]
Revised: [Equation]

Original: From [50] we have for both Wigner (Corollary 2.3.6), when the graph is undirected and i.i.d matrices (Corollary 2.3.5) when the edges are iid and directed, that There exists A, C, c >
Revised: From [50] we have for both Wigner (Corollary 2.3.6), when the graph is undirected and i.i.d matrices (Corollary 2.3.5) when the edges are iid and directed, that There exists A, C, c >

Original: [Equation]
Revised: [Equation]

Original: Which can be understood as 1 √ n W has bounded operator norm with overwhelming probability.
Revised: Which can be understood as 1 √ n W has bounded operator norm with overwhelming probability.

Original: Inparticular, taking the limit n → ∞ gives us that 1 √ n W has bounded norm almost surely whenn → ∞ .
Revised: In particular, taking the limit n → ∞ gives us that 1 √ n W has bounded norm almost surely when n → ∞ .

Original: Then ∥ D π ∥ =
Revised: Then ∥ D π ∥ =

Original: ∥ d π ∥ ∞ where ∥ · ∥ ∞ is the vector max-norm.
Revised: ∥ d π ∥ ∞ where ∥ · ∥ ∞ is the vector max-norm.

Original: This is due to the fact that D π is adiagonal matrix with positive entries, therefore its largest singular value is its largest eigenvalue.
Revised: This is due to the fact that D π is a diagonal matrix with positive entries, therefore its largest singular value is its largest eigenvalue.

Original: For ∥ X π ∥ to be o (1) we need
Revised: For ∥ X π ∥ to be o (1) we need

Original: [Equation]
Revised: [Equation]

Original: As 1 − µ ≤ 1 and for nµ = d , it is sufficient to have (almost surely) 
Revised: As 1 − µ ≤ 1 and for nµ = d , it is sufficient to have (almost surely) [Equation]

Revised: Lemma B.2.

Revised: From [51], for a irreducible and aperiodic Markov transition matrix P π , we have that, for P πkij =

Revised: d πj + O ( (cid:113) d

Original: [Equation]
Revised: [Equation]

Revised: πi λ k 2 ) , thus

Revised: [Equation]

Revised: where κ ( · ) is the condition number.

Revised: This lemma means that when using longer back-ups, as is for instance done when using n-step returns, the second eigenvalue of P k will shrink to 0 as long as 1 ∈ o ( k ) .

Original: B.3 Spiked Marchenko-Pastur distribution
Revised: B.3 Spiked Marchenko-Pastur distribution

Original: For, we prove small result showing that the o (1) appearing in ˆ P π remains a o (1) in 1 n Φ ⊤ ( I n −γ ˆ P π (cid:1) Φ .
Revised: For, we prove small result showing that the o (1) appearing in ˆ P π remains a o (1) in 1 n Φ ⊤ ( I n − γ ˆ P π (cid:1) Φ .

Original: Lemma B.2.
Revised: Lemma B.3.

Original: For ˆ P π satisfying the asymptotically well-connected condition, we have
Revised: For ˆ P π satisfying the asymptotically well-connected condition, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For ∥ · ∥ = ∥ · ∥ 2 the 2-operator norm which corresponds to the largest singular value, wehave
Revised: For ∥ · ∥ = ∥ · ∥ 2 the 2-operator norm which corresponds to the largest singular value, we have

Original: [Equation]
Revised: [Equation]

Original: As the maximum eigenvalue of 1 n Φ ⊤ Φ converges a.s to (1+ √ ρ ) 2 per the Marchenko-Pastur theorem
Revised: As the maximum eigenvalue of 1 n Φ ⊤ Φ converges a.s to (1+ √ ρ ) 2 per the Marchenko-Pastur theorem

Original: [Equation]
Revised: [Equation]

Original: This lemma is a useful result to understand that a o (1) perturbation stays a o (1) perturbation whenadding features.
Revised: This lemma is a useful result to understand that a o (1) perturbation stays a o (1) perturbation when adding features.

Original: Proposition B.1 (Asymmetric rank 1 perturbation) .
Revised: Proposition B.1 (Asymmetric rank 1 perturbation) .

Original: We consider the spectrum of 1 n Φ ⊤ ( I n −γ uv ⊤ )Φ for u , v ∈ R n so that v ⊤ u = 1 and γ ∈ [0 , 1[ which is a rank 1 perturbation of the
Revised: We consider the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ for u , v ∈ R n so that v ⊤ u = 1 and γ ∈ [0 , 1[ which is a rank 1 perturbation of the

Original: Wishart covariance model.
Revised: Wishart covariance model.

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ converges to the

Original: Marchenko-Pastur law.
Revised: Marchenko-Pastur law.

Original: When ργ 2 ≥ 1 , the empirical spectral distribution also converges to the
Revised: When ργ 2 ≥ 1 , the empirical spectral distribution also converges to the

Original: Marchenko-Pastur law, but additionally, we observe a phase transition [42], and an eigenvalueλ spiked separates from the bulk 
Revised: Marchenko-Pastur law, but additionally, we observe a phase transition [42], and an eigenvalue λ spiked separates from the bulk [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is based on a simple adaptation of the proof in [51] (page 117).
Revised: The proof is based on a simple adaptation of the proof in [52] (page 117).

Original: As we consider anasymmetric spike, we lose commutative properties.
Revised: As we consider an asymmetric spike, we lose commutative properties.

Original: Thus, we can only prove our result in the casewhere there is a unique spike instead of a finite number as is usual.
Revised: Thus, we can only prove our result in the case where there is a unique spike instead of a finite number as is usual.

Original: We are looking to characterizethe eigenvalues of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ , or equivalently (up to some zeros), the eigenvalues of1 n ΦΦ ⊤ ( I n − γ uv ⊤ ) .
Revised: We are looking to characterize the eigenvalues of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ , or equivalently (up to some zeros), the eigenvalues ofn ΦΦ ⊤ ( I n − γ uv ⊤ ) .

Original: However, in this case, 1 n ΦΦ ⊤ is not a sample from the Wishart distribution,but 1 p ΦΦ ⊤ is.
Revised: However, in this case, 1 n ΦΦ ⊤ is not a sample from the Wishart distribution, but 1 p ΦΦ ⊤ is.

Original: We therefore analyze 1 p ΦΦ ⊤ and will simply have to scale our spectrum by ρ at theend.0 = det
Revised: We therefore analyze 1 p ΦΦ ⊤ and will simply have to scale our spectrum by ρ at the end.

Original: [Equation]
Revised: [Equation]

Original: Using Sherman-Morrison formula, we have ( I n − γ uv ⊤ ) − 1 = I n + γ 1 − γ uv ⊤ as v ⊤ u = 1 .
Revised: Using Sherman-Morrison formula, we have ( I n − γ uv ⊤ ) − 1 = I n + γ 1 − γ uv ⊤ as v ⊤ u = 1 .

Original: Since det (cid:0) I n − γ uv ⊤ ) (cid:1) ̸ = 0 , the second determinant can be discarded.
Revised: Since det (cid:0) I n − γ uv ⊤ ) (cid:1) ̸ = 0 , the second determinant can be discarded.

Original: Then, we can factor the resolvent of the Wishart model.
Revised: Then, we can factor the resolvent of the Wishart model.

Original: That is, letting Q (ˆ λ ) =
Revised: That is, letting Q (ˆ λ ) =

Original: (cid:16) 1 p ΦΦ ⊤ −
Revised: (cid:16) 1 p ΦΦ ⊤ −

Original: ˆ λ I n (cid:17) − 1 , we write
Revised: ˆ λ I n (cid:17) − 1 , we write

Original: [Equation]
Revised: [Equation]

Original: Inverting 1 p ΦΦ ⊤ − ˆ λ
Revised: Inverting 1 p ΦΦ ⊤ − ˆ λ

Original: I n is almost surely possible when ˆ λ < (1 − (cid:112) 1 / ρ ) 2 per the concentration ofthe empirical spectral distribution of Wishart covariance models to the Marchenko-Pastur law andthe term Q − 1 (ˆ λ ) is almost surely non-zero.
Revised: I n is almost surely possible when ˆ λ < (1 − (cid:112) 1 / ρ ) 2 per the concentration of the empirical spectral distribution of Wishart covariance models to the Marchenko-Pastur law and the term Q − 1 (ˆ λ ) is almost surely non-zero.

Original: So the left term can be discarded.
Revised: So the left term can be discarded.

Original: For the second weuse Sylvester’s determinant identity to end up with a scalar equation
Revised: For the second we use Sylvester’s determinant identity to end up with a scalar equation

Original: [Equation]
Revised: [Equation]

Original: As Q (ˆ λ ) is a deterministic equivalent of m (ˆ λ ) I n ([18], Theorem 2.4 in [51]), where m ( · ) is the
Revised: As Q (ˆ λ ) is a deterministic equivalent of m (ˆ λ ) I n ([18], Theorem 2.4 in [52]), where m ( · ) is the

Original: Stieljes transform of the MP distribution, thus
Revised: Stieljes transform of the MP distribution, thus

Original: [Equation]
Revised: [Equation]

Original: The above determinant is zero when
Revised: The above determinant is zero when

Original: [Equation]
Revised: [Equation]

Original: From the definition of m ( · ) as the Stieljes transform of the MP distribution, we know [51] it satisfies
Revised: From the definition of m ( · ) as the Stieljes transform of the MP distribution, we know [52] it satisfies

Original: [Equation]
Revised: [Equation]

Original: for c = lim n, p → ∞ n / p = 1 / ρ
Revised: for c = lim n, p → ∞ n / p = 1 / ρ

Original: Thus, for z = ˆ λ
Revised: Thus, for z = ˆ λ

Original: [Equation]
Revised: [Equation]

Original: Plugging the equation above, asymptotically
Revised: Plugging the equation above, asymptotically

Original: [Equation]
Revised: [Equation]

Original: Now, we have to remember to multiply our spectrum by ρ to obtain the original one, and so weobtain
Revised: Now, we have to remember to multiply our spectrum by ρ to obtain the original one, and so we obtain

Original: [Equation]
Revised: [Equation]

Original: When (1 − γ )( ρ − 1 γ ) ≤ (1 − √ ρ ) 2 which is equivalent to ργ 2 ≥ 1 , this eigenvalue is isolated onthe left of the bulk.
Revised: When (1 − γ )( ρ − 1 γ ) ≤ (1 − √ ρ ) 2 which is equivalent to ργ 2 ≥ 1 , this eigenvalue is isolated on the left of the bulk.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to
Revised: When ργ 2 ≥ 1 , there is a phase transition [42] for the minimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We essentially do the same proof as above, but with an additional o (1) moving around andwe show that it does not impact the Stieljes transform, thus the spectrum, almost surely.
Revised: We essentially do the same proof as above, but with an additional o (1) moving around and we show that it does not impact the Stieljes transform, thus the spectrum, almost surely.

Original: [Equation]
Revised: [Equation]

Original: Where we used the definition of ˆ P π and Lemma B.2.
Revised: Where we used the definition of ˆ P π and Lemma B.3.

Original: Then0 = det
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: As the eigenvalues of I n − γ 1 n d ⊤ π are lower bounded by 1 − γ for n large enough det( I n − γ  + o (1))must be non-zero.
Revised: As the eigenvalues of I n − γ 1 n d ⊤ π are lower bounded by 1 − γ for n large enough det( I n − γ 1 n d ⊤ π + o (1)) must be non-zero.

Original: As ( I n + o (1)) − 1 =
Revised: As ( I n + o (1)) − 1 =

Original: (cid:80) i
Revised: (cid:80) i

Original: ( o (1))i
Revised: ( o (1))

Original:  =  o (1) , using Sherman-Morrison formula, wehave ( I n + o (1) − 
Revised: i = I n + o (1) , using Sherman-Morrison formula, we have ( I n + o (1) − γ 1 n d ⊤ π ) − 1 = I n + o (1)+ o (1) γ (1+ o (1)

Original: [Equation]
Revised: [Equation]

Original:  d ⊤ 
Revised: 1 − γ + o (1) 1 n d ⊤ π

Original: [Equation]
Revised: = I n + γ 1 − γ 1 n d ⊤ π + o (1) as d ⊤ π 1 n =

Original: With the same derivation and arguments as above
Revised: With the same derivation and arguments as above

Original: [Equation]
Revised: [Equation]

Original: And finally we end up with the same equality as before where the o (1) from the perturbation andthe one from the deterministic equivalent fuse together. 
Revised: And finally we end up with the same equality as before where the o (1) from the perturbation and the one from the deterministic equivalent fuse together. As the determinant is continuous

Original: [Equation]
Revised: [Equation]

Original: Therefore the Stieljes transforms when using ˆ P π and 1 n d ⊤ π are asymptotically the same, thus, almost surely, they have the same empirical spectral distribution.
Revised: Therefore the Stieljes transforms when using ˆ P π and 1 n d ⊤ π are asymptotically the same, thus, almost surely, they have the same empirical spectral distribution.

Original: B.4 Approximation of the extreme eigenvalues
Revised: B.4 Approximation of the extreme eigenvalues

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy asecular equation [43]
Revised: γ 1 n d ⊤ π ) satisfy a secular equation [43]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us consider v ̸ = 0 and λ , an eigenvector of Ξ( I n − γ 1 n d ⊤ π ) and its associated eigenvalue.
Revised: Let us consider v ̸ = 0 and λ , an eigenvector of Ξ( I n − γ 1 n d ⊤ π ) and its associated eigenvalue.

Original: [Equation]
Revised: [Equation]

Original: This give on component i
Revised: This give on component i

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Assuming λ ̸ = ξ i , ∀
Revised: Assuming λ ̸ = ξ i , ∀

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  Multiplying by d π ( i ) and summing
Revised: v i Multiplying by d π ( i ) and summing

Original: [Equation]
Revised: [Equation]

Original: Now the same term (cid:80) j d π ( j ) v j = d ⊤ π v appears on both sides.
Revised: Now the same term (cid:80) j d π ( j ) v j = d ⊤ π v appears on both sides.

Original: Let us verify if it can be null.
Revised: Let us verify if it can be null.

Original: If thatwere the case, we would have Ξ( I n − γ 1 n d ⊤ π ) v =
Revised: If that were the case, we would have Ξ( I n − γ 1 n d ⊤ π ) v =

Original: Ξ v = λ v .
Revised: Ξ v = λ v .

Original: Thus, as Ξ is diagonal, the possiblevectors v possible are the canonical basis vectors and the possible eigenvalues are ξ i .
Revised: Thus, as Ξ is diagonal, the possible vectors v possible are the canonical basis vectors and the possible eigenvalues are ξ i .

Original: However, asd π is a positive vector whose entries sum to 1, we cannot have d ⊤ π v = 0 .
Revised: However, as d π is a positive vector whose entries sum to 1, we cannot have d ⊤ π v = 0 .

Original: Thus d ⊤ π v ̸ = 0 and bydividing on each side:
Revised: Thus d ⊤ π v ̸ = 0 and by dividing on each side:

Original: [Equation]
Revised: [Equation]

Original: B.5 Optimization and approximation error
Revised: B.5 Optimization and approximation error

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ),we have the following decomposition
Revised: (2) ), we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: 10) where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ [Equation]
Revised: [Equation] where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Where we used ∥ A ∥ 2 ≤ √ n ∥
Revised: Where we used ∥ A ∥ 2 ≤ √ n ∥

Original: A ∥ ∞ and the fact that P π is a stochastic matrix, thus has entriesbetween 0 and 1.
Revised: A ∥ ∞ and the fact that P π is a stochastic matrix, thus has entries between 0 and 1.

Original: As such, the entries of I n − γ P π are between − γ and 1 , in particular they arebounded by 1 .
Revised: As such, the entries of I n − γ P π are between − γ and 1 , in particular they are bounded by 1 .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and that its spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η =
Revised: Denoting by λ +min and λ max its smallest non-zero and largest eigenvalues, for η =

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: Proof.
Revised: Proof.

Original: Let us begin with the case p > n From the TD iteration Equation (2) multiplied on the leftby Φ
Revised: Let us begin with the case p > n From the TD iteration Equation (2) multiplied on the left by Φ

Original: [Equation]
Revised: [Equation]

Original: For a i.i.d matrix Φ , we have that ΦΦ ⊤ is almost surely invertible.
Revised: For a i.i.d matrix Φ , we have that ΦΦ ⊤ is almost surely invertible.

Original: Thus the fixed point of thisiteration is
Revised: Thus the fixed point of this iteration is

Original: [Equation]

Original: =

Original: q

Original: [Equation]
Revised: [Equation]

Original: Thus, for H TD =
Revised: Thus, for H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ and x t = q t − q π 
Revised: Φ ⊤ Ξ( I n − γ P π )Φ and x t = q t − q π ∥

Original: [Equation]
Revised: [Equation]

Original: As I n − η Λ is diagonal, and as such its largest singular value (2-norm) is its largest eigenvalue (inmodulus).
Revised: As I n − η Λ is diagonal, and as such its largest singular value (2-norm) is its largest eigenvalue (in modulus).

Original: Now, we supposed the eigenvalues were real, max i,λ i =
Revised: Now, we supposed the eigenvalues were real, max i,λ i =

Original: Λ ii {| 1 − ηλ i |} will be minimized when thelargest and smallest terms are equal.
Revised: Λ ii {| 1 − ηλ i |} will be minimized when the largest and smallest terms are equal.

Original: [Equation]
Revised: [Equation]

Original: If λ +min = λ max this can be achieved for any η , however the only solution when they are different isto have the two expressions above of a different sign
Revised: If λ +min = λ max this can be achieved for any η , however the only solution when they are different is to have the two expressions above of a different sign

Original: [Equation]
Revised: [Equation]

Original: Which leads to η = 
Revised: Which leads to η = 2 λ +min +

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: max and

Original: [Equation]
Revised: [Equation]

Original: Now for p < n we do a similar reasoning with θ .
Revised: Now for p < n we do a similar reasoning with θ .

Original: By the exact same reasoning, but using H TD =
Revised: By the exact same reasoning, but using H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ instead and the fixed point θ TD = (cid:0) Φ ⊤ Ξ( I n − γ P π )Φ (cid:1) 1 Φ ⊤ Ξ
Revised: Φ ⊤ Ξ( I n − γ P π )Φ instead and the fixed point θ TD = (cid:0) Φ ⊤ Ξ( I n − γ P π [Equation] Φ [Equation]

Original:  r we get
Revised: 1 Φ ⊤ Ξ r we get

Original: [Equation]
Revised: [Equation]

Original: Now, we would like to obtain a result for the Q-function associated to these parameters.
Revised: Now, we would like to obtain a result for the Q-function associated to these parameters.

Original: We have σ min (Φ)
Revised: We have σ min (Φ)

Original: ∥ x ∥ 2 ≤ ∥ Φ x ∥ 2 ≤ σ max (Φ) ∥ x ∥ 2 for σ max and σ min the max and min singular values.
Revised: ∥ x ∥ 2 ≤ ∥ Φ x ∥ 2 ≤ σ max (Φ) ∥ x ∥ 2 for σ max and σ min the max and min singular values.

Original: Thus, multiplying each side by σ min (Φ) σ max (Φ) , we get
Revised: Thus, multiplying each side by σ min (Φ) σ max (Φ) , we get

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Squaring these results and putting it back in the contraction equation, we get for q TD =
Revised: Squaring these results and putting it back in the contraction equation, we get for q TD =

Original: Φ θ TD
Revised: Φ θ TD

Original: [Equation]
Revised: [Equation]

Original: B.6 Peaking behavior of the estimation error
Revised: B.6 Peaking behavior of the estimation error

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically 
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically rank one

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =
Revised: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =

Original: I n − Π is its complement, and ˜ ρ =
Revised: I n − Π is its complement, and ˜ ρ =

Original: d ⊤ π Π 1 n .
Revised: d ⊤ π Π 1 n .

Original: Proof.
Revised: Proof.

Original: Let us look at the case p < n .
Revised: Let us look at the case p < n .

Original: We have q TD =
Revised: We have q TD =

Original: Φ (cid:0) Φ ⊤ Ξ( I − γ P π )Φ (cid:1) − 1 Φ ⊤ Ξ r We will therefore study more closely the matrix Φ (cid:0) Φ ⊤ Ξ( I − γ P )Φ (cid:1) − 1 Φ ⊤ Ξ , in the case where P π = uv ⊤ is a rank one matrix.
Revised: Φ (cid:0) Φ ⊤ Ξ( I − γ P π )Φ (cid:1) − 1 Φ ⊤ Ξ r We will therefore study more closely the matrix Φ (cid:0) Φ ⊤ Ξ( I − γ P )Φ (cid:1) − 1 Φ ⊤ Ξ , in the case where P π = uv ⊤ is a rank one matrix.

Original: First, Sherman-Morrison formula gives us
Revised: First, Sherman-Morrison formula gives us

Original: [Equation]
Revised: [Equation]

Original: which is the usual formula but for x ≜ − γ x .
Revised: which is the usual formula but for x ≜ − γ x .

Original: Note that this is licit as long as 1 −
Revised: Note that this is licit as long as 1 −

Original: γ y ⊤ A − 1 x ̸ = 0 .
Revised: γ y ⊤ A − 1 x ̸ = 0 .

Original: Using Sherman-Morrison’s formula
Revised: Using Sherman-Morrison’s formula

Original: [Equation]
Revised: [Equation]

Original: For Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ the usual projector unto the span of Φ in metric Ξ .
Revised: For Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ the usual projector unto the span of Φ in metric Ξ .

Original: Now, for the TD error we obtain
Revised: Now, for the TD error we obtain

Original: [Equation]
Revised: [Equation]

Original: Let’s call I −
Revised: Let’s call I −

Original: Π
Revised: Π

Original: =
Revised: =

Original: Π ⊥
Revised: Π ⊥

Original: the complement projector projector and ˜ ρ = v ⊤ Π u .
Revised: the complement projector projector and ˜ ρ = v ⊤ Π u .

Original: Now, for the MSBEerror, we get
Revised: Now, for the MSBE error, we get

Original: [Equation]
Revised: [Equation]

Original:  where we use the fact that Π ⊤ ΞΠ = ΞΠ = Π ⊤ Ξ and same for Π ⊥ .
Revised: Π u [Equation] where we use the fact that Π ⊤ ΞΠ = ΞΠ = Π ⊤ Ξ and same for Π ⊥ .

Original: Using v = d π and u = 1 n gives us the approximation.
Revised: Using v = d π and u = 1 n gives us the approximation.

Original: C Colab links
Revised: C Colab links

Original: For plotting the spiked MP spectra https://colab.research.google.com/drive
Revised: For plotting the spiked MP spectra https://colab.research.google.com/drive/ 1E3DgmZQF1-1FNYeVo4msyGUHwRiK32WE?usp =

Original:  For the extreme eigenvalues approximation https://colab.research.google.com/drive/1H_GkjzlGyUxB0i55jM5VAqLXoEGGxN3h?usp=sharing
Revised: sharing For the extreme eigenvalues approximation https://colab.research.google.com/ drive/1H_GkjzlGyUxB0i55jM5VAqLXoEGGxN3h?usp=sharing

Original: For the peaking behavior https://colab.research.google.com/drive/1EVudyGfieXOUqDLifdhFrs8Pu1OzKOAW?usp=sharing
Revised: For the peaking behavior https://colab.research.google.com/drive/ 1EVudyGfieXOUqDLifdhFrs8Pu1OzKOAW?usp=sharing

Original: For the peaking behavior model https://colab.research.google.com/drive/1pLIqaV5BO1ChIj9y9UmP5jmlWx26FNI7?usp=sharing
Revised: For the peaking behavior model https://colab.research.google.com/drive/ 1pLIqaV5BO1ChIj9y9UmP5jmlWx26FNI7?usp=sharing

Original: For the unstability wrt η and ϵ https://colab.research.google.com/drive/1CuPZhv3eZoINXHCKLtwjVlHrLy9ylBuI?usp=sharing
Revised: For the unstability wrt η and ϵ https://colab.research.google.com/drive/ 1CuPZhv3eZoINXHCKLtwjVlHrLy9ylBuI?usp =sharing