Original: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation
Revised: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation

Original: Abstract
Revised: Abstract

Original: Much of the recent successes of deep learning can be attributed to scaling up thesize of the networks to the point where they often are vastly overparameterized.
Revised: Much of the recent successes of deep learning can be attributed to scaling up thesize of the networks to the point where they often are vastly overparameterized.

Original: Thus, understanding the role of overparameterization is of increasing importance.
Revised: Thus, understanding the role of overparameterization is of increasing importance.

Original: While predictive theories have been developed for supervised learning, little isknown about the Reinforcement Learning case.
Revised: While predictive theories have been developed for supervised learning, little isknown about the Reinforcement Learning case.

Original: In this work, we take a theoreticalapproach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.
Revised: In this work, we take a theoreticalapproach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.

Original: We leverage tools from random matrixtheory and random graph theory to obtain a characterization of the spectrum of the
Revised: We leverage tools from random matrixtheory and random graph theory to obtain a characterization of the spectrum of the

Original: TD operator.
Revised: TD operator.

Original: We use this result to study the stability and optimization dynamicsof TD learning as a function of the number of parameters
Revised: We use this result to study the stability and optimization dynamicsof TD learning as a function of the number of parameters

Original:  Introduction
Revised:  Introduction

Original: Occam’s razor principle states that among many plausible explanations, the simplest one is themost likely to be true.
Revised: Occam’s razor principle states that among many plausible explanations, the simplest one is themost likely to be true.

Original: In statistics and machine learning, this is often interpreted as models witha restricted number of parameters should be privileged as they generalize better [1].
Revised: In statistics and machine learning, this is often interpreted as models witha restricted number of parameters should be privileged as they generalize better [1].

Original: This view issupported by the classical bias-variance trade-off which implies that models that are too flexibleare bound to overfit [2].
Revised: This view issupported by the classical bias-variance trade-off which implies that models that are too flexibleare bound to overfit [2].

Original: Yet, deep neural networks with more parameters than training data pointsachieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.
Revised: Yet, deep neural networks with more parameters than training data pointsachieved remarkable performance in many domains [3, 4, 5], challenging this conventional wisdom.

Original: A significant understanding of this apparent tension has been reached in recent years.
Revised: A significant understanding of this apparent tension has been reached in recent years.

Original: A largebody of work using tools from random matrix theory has shown that, in models with randomfeatures, overparameterization does not lead to overfitting but instead better generalization [6].
Revised: A largebody of work using tools from random matrix theory has shown that, in models with randomfeatures, overparameterization does not lead to overfitting but instead better generalization [6].

Original: Thisphenomenon is known as double descent and unifies the traditional and modern understanding ofbias and variance.
Revised: Thisphenomenon is known as double descent and unifies the traditional and modern understanding ofbias and variance.

Original: Importantly, [7] has shown that the double descent phenomenon is in large partexplained by purely optimization rather than label noise.
Revised: Importantly, [7] has shown that the double descent phenomenon is in large partexplained by purely optimization rather than label noise.

Original: Following their insight, we mainly take anoptimization approach in the rest of our paper.
Revised: Following their insight, we mainly take anoptimization approach in the rest of our paper.

Original: While we now have a theory of overparameterization in supervised learning, the theoretical analysisof overparameterization in the reinforcement learning (RL) setting is largely unexplored [8].
Revised: While we now have a theory of overparameterization in supervised learning, the theoretical analysisof overparameterization in the reinforcement learning (RL) setting is largely unexplored [8].

Original: Indeed,
Revised: Indeed,

Original: RL poses several new additional challenges.
Revised: RL poses several new additional challenges.

Original: First, the agent has to interact with an environment whose structure may be complex.
Revised: First, the agent has to interact with an environment whose structure may be complex.

Original: Second, each state may not sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.
Revised: Second, each state may not sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.

Original: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.
Revised: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.

Original: More precisely, we study the behavior of Temporal Difference learning [9] which has become ubiquitous for policy evaluation in modern Deep RL [10, 11]
Revised: More precisely, we study the behavior of Temporal Difference learning [9] which has become ubiquitous for policy evaluation in modern Deep RL [10, 11]

Original: Our contributions are, in the limit of large number of states and parameters: 1) We introduce aclass of random graphs for which we can compute analytically the spectrum of the TD operatorwhen using uniform off-policy sampling, 2) we derive approximations for the largest and smallesteigenvalue in the non-uniform case.
Revised: Our contributions are, in the limit of large number of states and parameters: 1) We introduce aclass of random graphs for which we can compute analytically the spectrum of the TD operatorwhen using uniform off-policy sampling, 2) we derive approximations for the largest and smallesteigenvalue in the non-uniform case.

Original: Furthermore, we analyze the behavior of the traditional Mean
Revised: Furthermore, we analyze the behavior of the traditional Mean

Original: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a doubledescent phenomenon similar to supervised learning but with a multiplicative factor depending onthe discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displaysa peaking behavior near the interpolation threshold (when the number of parameters approachesthe number of states) that is specific to RL.
Revised: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a doubledescent phenomenon similar to supervised learning but with a multiplicative factor depending onthe discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displaysa peaking behavior near the interpolation threshold (when the number of parameters approachesthe number of states) that is specific to RL.

Original: Finally, 5) we showcase how the non-stationarity of thepolicy can make the optimization of the value function unstable.
Revised: Finally, 5) we showcase how the non-stationarity of thepolicy can make the optimization of the value function unstable.

Original: All of these points are motivatedtheoretically and empirically validated on small gridworlds MDPs.
Revised: All of these points are motivatedtheoretically and empirically validated on small gridworlds MDPs.

Original:  Background2.1 Value estimationDefinition:
Revised:  Background2.1 Value estimationDefinition:

Original: We place ourselves in the Markov Decision Process (MDP) setting [12] definedby {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈ 47 R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.
Revised: We place ourselves in the Markov Decision Process (MDP) setting [12] definedby {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces, P sa → s ∈ 47 R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.

Original: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.
Revised: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.

Original: Fora given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be ableto estimate the expected discounted return q π ( s, a ) =
Revised: Fora given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be ableto estimate the expected discounted return q π ( s, a ) =

Original: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s 

Revised: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s

Original: =
Revised: =

Original: s, a 0 = a ] .
Revised: s, a 0 = a ] .

Original: This quantity is also known as the state-action value or the Q-function.
Revised: This quantity is also known as the state-action value or the Q-function.

Original: Furthermore, it can be written as the solution of the Bellman equation [13] 1 : q π ( s, a ) =
Revised: Furthermore, it can be written as the solution of the Bellman equation [13] 1 : q π ( s, a ) =

Original: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .
Revised: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .

Original: Bydefining P π ∈ R |S|·|A|×|S|·|A|
Revised: Bydefining P π ∈ R |S|·|A|×|S|·|A|

Original: the state-action to state-action transition matrix, we can write it inmatrix form
Revised: the state-action to state-action transition matrix, we can write it inmatrix form

Original: [Equation]
Revised: [Equation]

Original: Linear function approximation: While Equation (1) can be solved efficiently when r , P π areknown and when n = |S|·|A| is not too large, in many cases this cannot be computed.
Revised: Linear function approximation: While Equation (1) can be solved efficiently when r , P π areknown and when n = |S|·|A| is not too large, in many cases this cannot be computed.

Original: One solutionis to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .
Revised: One solutionis to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .

Original: This is referred to as the Linear Function Approximation (LFA) setting.
Revised: This is referred to as the Linear Function Approximation (LFA) setting.

Original: Temporal Difference with LFA:
Revised: Temporal Difference with LFA:

Original: Estimating the Q-function in the Linear Function Approximationsetting can be done using the well-known algorithm TD(0)
Revised: Estimating the Q-function in the Linear Function Approximationsetting can be done using the well-known algorithm TD(0)

Original: [9] whose expected update is
Revised: [9] whose expected update is

Original: [Equation]
Revised: [Equation]

Original: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summingto 1 , weighting each state-action by their probability of being sampled.
Revised: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summingto 1 , weighting each state-action by their probability of being sampled.

Original: Contrary to gradient descentwith an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φmight have a negative eigenvalue.
Revised: Contrary to gradient descentwith an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0) I n − γ P π )Φmight have a negative eigenvalue.

Original: This phenomenon is known as the deadly triad as it can onlyhappen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on thenext value as TD(0) does [14, 15, 16].
Revised: This phenomenon is known as the deadly triad as it can onlyhappen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on thenext value as TD(0) does [14, 15, 16].

Original: 2.2 Random Matrix Theory
Revised: 2.2 Random Matrix Theory

Original: Wishart matrices:
Revised: Wishart matrices:

Original: Extremely relevant in the study of least squares with random features arematrices from the Wishart distribution [17].
Revised: Extremely relevant in the study of least squares with random features arematrices from the Wishart distribution [17].

Original: These matrices can be written as 1 n Φ ⊤ Φ for Φ an × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.
Revised: These matrices can be written as 1 n Φ ⊤ Φ for Φ an × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.

Original: In that case, when p, n → ∞ and the ratio converges to a finitelimit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [18] (MP) distribution whose density can be written as
Revised: In that case, when p, n → ∞ and the ratio converges to a finitelimit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the MarchenkoPastur [18] (MP) distribution whose density can be written as

Original: [Equation]
Revised: [Equation]

Original: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x )are the 0 eigenvalues, which happens when p ≥ n , λ max
Revised: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x )are the 0 eigenvalues, which happens when p ≥ n , λ max

Original: =
Revised: =

Original: (1 + √ ρ ) 2 , the largest, and λ +min =(1 − √ ρ ) 2 , the smallest non-zero eigenvalue.
Revised: (1 + √ ρ ) 2 , the largest, and λ +min =(1 − √ ρ ) 2 , the smallest non-zero eigenvalue.

Original: Wigner-type matrices: [19] historically introduced and studied symmetric n × n matrices ofthe form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance fori ≥ j 2 .
Revised: Wigner-type matrices: [19] historically introduced and studied symmetric n × n matrices ofthe form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance fori ≥ j 2 .

Original: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) =12 π √ 4 − x 2 1 [ − 2 , 2] dx .
Revised: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) =12 π √ 4 − x 2 1 [ − 2 , 2] dx .

Original: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [21].
Revised: When non-symmetric, the spectrum converges to the complex unit disk { z ∈ C , | z | ≤ 1 } 3 [21].

Original: 2.3 Random graphs
Revised: 2.3 Random graphs

Original: Random graphs are useful models to infer structural or spectral properties of typical graphs.
Revised: Random graphs are useful models to infer structural or spectral properties of typical graphs.

Original: Theearliest theoretical analyses of random graphs that we are aware of were done independently by [22]and [23].
Revised: Theearliest theoretical analyses of random graphs that we are aware of were done independently by [22]and [23].

Original: The model introduced in [22] is a random graph where each of the n nodes is connectedat random to d others.
Revised: The model introduced in [22] is a random graph where each of the n nodes is connectedat random to d others.

Original: The second model [23] is slightly different as each node is independentlyconnected to any other with a probability µ ∈ ]0 , 1[ .
Revised: The second model [23] is slightly different as each node is independentlyconnected to any other with a probability µ ∈ ]0 , 1[ .

Original: We refer to the first one as the d -regular Erd˝osR ´ enyi model G ( n, d ) and the second one as the Erd ˝ os-R ´ enyi-Gilbert G ( n, µ ) model.
Revised: We refer to the first one as the d -regular Erd˝osR ´ enyi model G ( n, d ) and the second one as the Erd ˝ os-R ´ enyi-Gilbert G ( n, µ ) model.

Original: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct amatrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rowssumming to 1 ).
Revised: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct amatrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rowssumming to 1 ).

Original: P corresponds to the Markov transition matrix of a random walk on the graph and isconnected to the (left)-normalized Laplacian L by L =
Revised: P corresponds to the Markov transition matrix of a random walk on the graph and isconnected to the (left)-normalized Laplacian L by L =

Original: I n − P .
Revised: I n − P .

Original: For many classes of random graphs,the spectrum of these quantities is known [24, 25].
Revised: For many classes of random graphs,the spectrum of these quantities is known [24, 25].

Original:  Spectrum of TD: from random graphs to spiked models
Revised:  Spectrum of TD: from random graphs to spiked models

Original: 3.1 From Ordinary Least Squares to TD(0)
Revised: 3.1 From Ordinary Least Squares to TD(0)

Original: Random matrix models have been used in supervised learning, especially in the simple least squaresmodel [26, 27, 28], to explain phenomena arising in deep learning.
Revised: Random matrix models have been used in supervised learning, especially in the simple least squaresmodel [26, 27, 28], to explain phenomena arising in deep learning.

Original: Indeed, while much simpler thanlarge neural networks, these models can shed light on the optimization and generalization behaviorof vastly overparameterized models [29, 30], something that most traditional complexity measurescould not [31, 32].
Revised: Indeed, while much simpler thanlarge neural networks, these models can shed light on the optimization and generalization behaviorof vastly overparameterized models [29, 30], something that most traditional complexity measurescould not [31, 32].

Original: Using gradient descent on a least square problem, our parameter update can be written as
Revised: Using gradient descent on a least square problem, our parameter update can be written as

Original: [Equation]
Revised: [Equation]

Original: (4) for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.
Revised: (4) for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.

Original: When choosing thelearning rate η optimally, the convergence rate is expressed as λ
Revised: When choosing thelearning rate η optimally, the convergence rate is expressed as λ

Original: [Equation]
Revised: [Equation]

Original: for λ +min , respectivelyλ max , the smallest, resp.
Revised: for λ +min , respectivelyλ max , the smallest, resp.

Original: largest, non-zero eigenvalue of Φ ⊤ Φ [33].
Revised: largest, non-zero eigenvalue of Φ ⊤ Φ [33].

Original: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we willconsider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.
Revised: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we willconsider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.

Original: In particular, we haveanalytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).
Revised: In particular, we haveanalytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).

Original: Whenlim p,n →∞ p / n = ρ , we have λ max / λ +min
Revised: Whenlim p,n →∞ p / n = ρ , we have λ max / λ +min

Original: →
Revised: →

Original: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .
Revised: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .

Original: This function has asymptotes at 1 , forp
Revised: This function has asymptotes at 1 , forp

Original: = o ( n ) or n = o ( p ) and diverges when n = p .
Revised: = o ( n ) or n = o ( p ) and diverges when n = p .

Original: Note that this peak-shaped function can also beobserved empirically in more complex models such as kernel methods [34] and neural networks inthe limit of infinite width as they are in the Neural Tangent Kernel regime [35, 36].
Revised: Note that this peak-shaped function can also beobserved empirically in more complex models such as kernel methods [34] and neural networks inthe limit of infinite width as they are in the Neural Tangent Kernel regime [35, 36].

Original: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would bereplaced by Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would bereplaced by Φ ⊤ Ξ( I n − γ P π )Φ .

Original: While [8] derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.
Revised: While [8] derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.

Original: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.
Revised: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.

Original: While we can model Φ as ani.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allowus to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: While we can model Φ as ani.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allowus to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ .

Original: We believe this is one ofthe key reasons why overparameterization in Temporal Difference learning has not been studiedextensively.
Revised: We believe this is one ofthe key reasons why overparameterization in Temporal Difference learning has not been studiedextensively.

Original: Thus, finding good typical models for P π is of utmost importance, and the next sectionwill introduce a class of models for which the spectrum of TD can be computed.
Revised: Thus, finding good typical models for P π is of utmost importance, and the next sectionwill introduce a class of models for which the spectrum of TD can be computed.

Original: 3.2 P π as the Markov transition matrix of a random graph
Revised: 3.2 P π as the Markov transition matrix of a random graph

Original: In RL, P π is the Markov transition matrix of a random walk when sampling from π .
Revised: In RL, P π is the Markov transition matrix of a random walk when sampling from π .

Original: As randomgraphs are used to model properties of typical graphs, a candidate for a model of P π would be to takethe Markov transition matrix of a random graph.
Revised: As randomgraphs are used to model properties of typical graphs, a candidate for a model of P π would be to takethe Markov transition matrix of a random graph.

Original: However, random graphs, at least the Erd˝os-R´enyi(-
Revised: However, random graphs, at least the Erd˝os-R´enyi(-

Original: Gilbert) type ones, are not able natively to take into account the impact of π , most importantly thefact that some state-actions are visited more often than others under d π , the stationary distributionof π .
Revised: Gilbert) type ones, are not able natively to take into account the impact of π , most importantly thefact that some state-actions are visited more often than others under d π , the stationary distributionof π .

Original: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markovtransition matrix of a G ( n, µ ) graph deformed by d π .
Revised: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markovtransition matrix of a G ( n, µ ) graph deformed by d π .

Original: Let us start with an undirected graph of the type G ( n, µ ) .
Revised: Let us start with an undirected graph of the type G ( n, µ ) .

Original: To take into account the state-actionvisitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .
Revised: To take into account the state-actionvisitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .

Original: Now, we define thedeformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.
Revised: Now, we define thedeformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.

Original: Notehere that when d π is uniform, we revert back to the original case A π = A .
Revised: Notehere that when d π is uniform, we revert back to the original case A π = A .

Original: This construction keepsthe same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.
Revised: This construction keepsthe same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.

Original: We can informally write the Markov transition matrix associated to this deformed graph as 4ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n Wis a Wigner-type matrix with zero mean and unit variance.
Revised: We can informally write the Markov transition matrix associated to this deformed graph as 4ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n Wis a Wigner-type matrix with zero mean and unit variance.

Original: It can be tempting to understand underwhich conditions the stochastic part becomes negligible and what it entails.
Revised: It can be tempting to understand underwhich conditions the stochastic part becomes negligible and what it entails.

Original: As we will see shortly,this setting will reveal itself to be of importance when studying the spectrum of TD.
Revised: As we will see shortly,this setting will reveal itself to be of importance when studying the spectrum of TD.

Original: Definition 1 (Asymptotically well-connected graphs) .
Revised: Definition 1 (Asymptotically well-connected graphs) .

Original: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its
Revised: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its

Original: Markov transition matrix.
Revised: Markov transition matrix.

Original: The choice for the name of Definition 1 comes from the fact that | λ 2 | is related to the presence ofbottlenecks in the graph through Cheeger’s inequality [37].
Revised: The choice for the name of Definition 1 comes from the fact that | λ 2 | is related to the presence ofbottlenecks in the graph through Cheeger’s inequality [37].

Original: Example.
Revised: Example.

Original: Graphs of the type G ( n, µ ) (resp.
Revised: Graphs of the type G ( n, µ ) (resp.

Original: G ( n, d ) ) are asymptotically well-connected if theirexpected degree d
Revised: G ( n, d ) ) are asymptotically well-connected if theirexpected degree d

Original: = nµ (resp.
Revised: = nµ (resp.

Original: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [24, 25].
Revised: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) [24, 25].

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: Note here that this again generalizes the uniform case as, when d π is uniform, the conditionbecomes equivalent to 1 ∈ o ( d ) as in the example.
Revised: Note here that this again generalizes the uniform case as, when d π is uniform, the conditionbecomes equivalent to 1 ∈ o ( d ) as in the example.

Original: In the non-uniform case, this condition entailsthat either d π spreads efficiently across states, or the degree d rises fast enough that those states arestill visited sufficiently often.
Revised: In the non-uniform case, this condition entailsthat either d π spreads efficiently across states, or the degree d rises fast enough that those states arestill visited sufficiently often.

Original: In that case, we have
Revised: In that case, we have

Original: [Equation]
Revised: [Equation]

Original: This approximation is particularly interesting as it asymptotically conserves left right
Revised: This approximation is particularly interesting as it asymptotically conserves left right

Original: 1 n the original P π associated to the eigenvalue 1 .
Revised: 1 n the original P π associated to the eigenvalue 1 .

Original: Furthermore, if allstate are reachable, the sequence of matrix power iterates of P π ,
Revised: Furthermore, if allstate are reachable, the sequence of matrix power iterates of P π ,

Original: P converges to
Revised: P converges to

Original: While we are not the first to use tools from graph theory in the context of RL, related work weare aware of [38, 39, 40, 41] takes a different approach and leverages eigenvectors of the Markovtransition matrix (or Laplacian) to construct better representations for the value function or usethem as options.
Revised: While we are not the first to use tools from graph theory in the context of RL, related work weare aware of [38, 39, 40, 41] takes a different approach and leverages eigenvectors of the Markovtransition matrix (or Laplacian) to construct better representations for the value function or usethem as options.

Original: In contrast, we use spectral properties of P π to study the convergence of Temporal
Revised: In contrast, we use spectral properties of P π to study the convergence of Temporal

Original: Difference learning.
Revised: Difference learning.

Original: 3.3 Spiked Wishart model for TD with uniform sampling
Revised: 3.3 Spiked Wishart model for TD with uniform sampling

Original: Now that we have a simple random graph model for P π , we focus on estimating the spectrumof Φ ⊤ Ξ( I n − γ ˆ P π )Φ .
Revised: Now that we have a simple random graph model for P π , we focus on estimating the spectrumof Φ ⊤ Ξ( I n − γ ˆ P π )Φ .

Original: For simplicity, we assume the off-policy sampling is done uniformly, i.eΞ = 1 n I n .
Revised: For simplicity, we assume the off-policy sampling is done uniformly, i.eΞ = 1 n I n .

Original: While this assumption is not realistic in most RL settings, it is still of interest in thecontext of dynamic programming where we have access to all the states.
Revised: While this assumption is not realistic in most RL settings, it is still of interest in thecontext of dynamic programming where we have access to all the states.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n −  ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to[Equation] min
Revised: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]
Revised: [Equation]

Original: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =
Revised: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =

Original: ( (cid:112) 1 / γ 2 − 1) 2 .
Revised: ( (cid:112) 1 / γ 2 − 1) 2 .

Original: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .
Revised: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .

Original: In thatcase, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , theeffective horizon of the problem, as lim
Revised: In thatcase, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , theeffective horizon of the problem, as lim

Original: ρ →∞ λ max / λ +min
Revised: ρ →∞ λ max / λ +min

Original: =
Revised: =

Original: 11 − γ .
Revised: 11 − γ .

Original: In the tabular regime, Φ = I n .
Revised: In the tabular regime, Φ = I n .

Original: As
Revised: As

Original: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .
Revised: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .

Original: This would give aworst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterizedcase above.
Revised: This would give aworst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterizedcase above.

Original: As these eigenvalues govern the optimization speed of TD, the vastly overparameterizedcase and the tabular can be thought of as comparable.
Revised: As these eigenvalues govern the optimization speed of TD, the vastly overparameterizedcase and the tabular can be thought of as comparable.

Original: 3.4 Non-uniform off-policy sampling
Revised: 3.4 Non-uniform off-policy sampling

Original: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assumingΞ = 1 n I n .
Revised: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assumingΞ = 1 n I n .

Original: In this subsection we analyze the more general case where Ξ is not uniform.
Revised: In this subsection we analyze the more general case where Ξ is not uniform.

Original: However bydoing so, we lose guarantees and must resort to approximations.
Revised: However bydoing so, we lose guarantees and must resort to approximations.

Original: First, we analyze the eigenvaluesof Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .
Revised: First, we analyze the eigenvaluesof Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy asecular equation [43]
Revised: γ 1 n d ⊤ π ) satisfy asecular equation [43]

Original: [Equation]
Revised: [Equation]

Original: This expression is another form of the characteristic polynomial of a matrix whose roots are itseigenvalues.
Revised: This expression is another form of the characteristic polynomial of a matrix whose roots are itseigenvalues.

Original: However, this form enables us to derive approximations more easily.
Revised: However, this form enables us to derive approximations more easily.

Original: In particular,for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸
Revised: In particular,for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸

Original: = m , we have for λ ≈ ξ mthat 1 ξ i−
Revised: = m , we have for λ ≈ ξ mthat 1 ξ i

Original: [Equation]
Revised: [Equation]

Original: ≈ [Equation] ξ i if i ̸
Revised: ≈ 1 ξ i if i ̸

Original: = m .
Revised: = m .

Original: From this assumption, we derive an approximation for the minimumeigenvalue of Ξ( I n − γ ˆ P π )
Revised: From this assumption, we derive an approximation for the minimumeigenvalue of Ξ( I n − γ ˆ P π )

Original: [Equation]
Revised: [Equation]

Original: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution andthe discount factor.
Revised: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution andthe discount factor.

Original: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which isthe product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .
Revised: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which isthe product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .

Original: Note that a similar expression can bederived for the maximum eigenvalue under similar assumptions.
Revised: Note that a similar expression can bederived for the maximum eigenvalue under similar assumptions.

Original: Now, again reasoning approximately, if that eigenvalue was very small compared to the other oneswho would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similarspiking behavior as before.
Revised: Now, again reasoning approximately, if that eigenvalue was very small compared to the other oneswho would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similarspiking behavior as before.

Original: However, in that case, the spiking eigenvalue would be
Revised: However, in that case, the spiking eigenvalue would be

Original: [Equation]
Revised: [Equation]

Original: While less simple than the estimators developed in the last subsection, we will see in Section 5 thatit still predicts accurately the smallest non-zero eigenvalue.
Revised: While less simple than the estimators developed in the last subsection, we will see in Section 5 thatit still predicts accurately the smallest non-zero eigenvalue.

Original:  Optimizing with Temporal Difference learning
Revised:  Optimizing with Temporal Difference learning

Original: 4.1 Decomposition of the error
Revised: 4.1 Decomposition of the error

Original: Using results about the spectrum of the TD operator shown in the last section, we analyze thebehavior of the Mean Square Bellman Error L MSBE ( q t ) =
Revised: Using results about the spectrum of the TD operator shown in the last section, we analyze thebehavior of the Mean Square Bellman Error L MSBE ( q t ) =

Original: 12 ∥
Revised: 12 ∥

Original: r + γ P π q t − q t ∥
Revised: r + γ P π q t − q t ∥

Original: 2Ξ during theoptimization.
Revised: 2Ξ during theoptimization.

Original: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n
Revised: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n

Original: [Equation]
Revised: [Equation]

Original: the q -value asymptotically reached by TD(0), we can decompose this error into two components.
Revised: the q -value asymptotically reached by TD(0), we can decompose this error into two components.

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ),we have the following decomposition
Revised: (2) ),we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .
Revised: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Following [44], we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥
Revised: Following [44], we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥

Original: r + γ P π q TD − q TD
Revised: r + γ P π q TD − q TD

Original: ∥ Ξ as theestimation error.
Revised: ∥ Ξ as theestimation error.

Original: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π )in metric Ξ .
Revised: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π )in metric Ξ .

Original: Note that, beyond this factor, we are not exactly in the same setting as [44] as theasymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, asknown in the RL community [45].
Revised: Note that, beyond this factor, we are not exactly in the same setting as [44] as theasymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, asknown in the RL community [45].

Original: Indeed, TD(0) finds the minimum of the Mean Square Projected
Revised: Indeed, TD(0) finds the minimum of the Mean Square Projected

Original: Bellman Error (MSPBE) but not of the MSBE [46].
Revised: Bellman Error (MSPBE) but not of the MSBE [46].

Original: However in practice it is common to minimizethe MSBE as it does not require an expensive projection step.
Revised: However in practice it is common to minimizethe MSBE as it does not require an expensive projection step.

Original: 4.2 The ordinary behavior of the approximation error
Revised: 4.2 The ordinary behavior of the approximation error

Original: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .
Revised: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η = 2 λ +min + λ max
Revised: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η = 2 λ +min + λ max

Original: [Equation]
Revised: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).
Revised: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).

Original: This is a slightly more general version of the linear convergence of gradient descent, however, as the
Revised: This is a slightly more general version of the linear convergence of gradient descent, however, as the

Original: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .
Revised: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .

Original: 4.3 The singular behavior of the estimation error
Revised: 4.3 The singular behavior of the estimation error

Original: Now we turn our attention to the second term; the estimation error which is the residual error after
Revised: Now we turn our attention to the second term; the estimation error which is the residual error after

Original: T → + ∞ steps.
Revised: T → + ∞ steps.

Original: In supervised learning, we expect this error to be the usual bias of our algorithmand thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possibleto interpolate the data.
Revised: In supervised learning, we expect this error to be the usual bias of our algorithmand thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possibleto interpolate the data.

Original: However, in reinforcement learning, the behavior of this term can be quitedifferent, in particular it can be non-monotonous.
Revised: However, in reinforcement learning, the behavior of this term can be quitedifferent, in particular it can be non-monotonous.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptoticallyrank one
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptoticallyrank one

Original: [Equation]
Revised: [Equation]

Original: When γ = 0 , i.e we just perform reward regression, this is equal to ∥
Revised: When γ = 0 , i.e we just perform reward regression, this is equal to ∥

Original: Π ⊥ r ∥ 2Ξ which is exactly theresidual error in r that our representation cannot capture.
Revised: Π ⊥ r ∥ 2Ξ which is exactly theresidual error in r that our representation cannot capture.

Original: When ρ > 1 , Π ⊥ = 

Revised: When ρ > 1 , Π ⊥ =

Original: so the limit is zeroas we expect from a model able to interpolate the data.
Revised: so the limit is zeroas we expect from a model able to interpolate the data.

Original: However, there is a behavior here not presentin supervised learning.
Revised: However, there is a behavior here not presentin supervised learning.

Original: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p closeto but less than n , a peaking behavior of the estimation error can happen.
Revised: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p closeto but less than n , a peaking behavior of the estimation error can happen.

Original: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased
Revised: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased

Original: [Equation]
Revised: [Equation]

Original: This phenomenon might appear counter-intuitive as we would expected our error to decrease withp .
Revised: This phenomenon might appear counter-intuitive as we would expected our error to decrease withp .

Original: We provide a few insights to make sense of this.
Revised: We provide a few insights to make sense of this.

Original: First, when γ increases, so does the scale of the
Revised: First, when γ increases, so does the scale of the

Original: Q-function, so we can expect our errors to get bigger.
Revised: Q-function, so we can expect our errors to get bigger.

Original: Second, using random features Φ informallyhas a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from11 − γ to 1 1 − γρ .
Revised: Second, using random features Φ informallyhas a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from11 − γ to 1 1 − γρ .

Original: This can be observed in the proof in appendix.
Revised: This can be observed in the proof in appendix.

Original: Finally, peaking behaviors of TD suchas this one have been observed in the community [45] but not theoretically explained in a generalsetting as far as we know.
Revised: Finally, peaking behaviors of TD suchas this one have been observed in the community [45] but not theoretically explained in a generalsetting as far as we know.

Original: 4.4 Beyond stationary sampling
Revised: 4.4 Beyond stationary sampling

Original: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξwas non-singular, i.e that our off-policy sampling visited all the states .
Revised: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξwas non-singular, i.e that our off-policy sampling visited all the states .

Original: Here, we take a closer lookat what happens when this is not the case.
Revised: Here, we take a closer lookat what happens when this is not the case.

Original: In particular, we study a simple toy model where weassume Ξ only visits n vis ( t ) states at time t with uniform probability.
Revised: In particular, we study a simple toy model where weassume Ξ only visits n vis ( t ) states at time t with uniform probability.

Original: This simply changes all then by n vis ( t ) in the previous results.
Revised: This simply changes all then by n vis ( t ) in the previous results.

Original: As often observed during training, at t = 0 , π might start as amore exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution( n vis ( t ) low).
Revised: As often observed during training, at t = 0 , π might start as amore exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution( n vis ( t ) low).

Original: This can be problematic when performing TD learning with a fixed stepsize η .
Revised: This can be problematic when performing TD learning with a fixed stepsize η .

Original: In thatcase, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 →2 / λ max or equivalently
Revised: In thatcase, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 →2 / λ max or equivalently

Original: [Equation]
Revised: [Equation]

Original: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .
Revised: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .

Original: We conjecture this phenomenon may be related to the popularity of adaptative step size methods in
Revised: We conjecture this phenomenon may be related to the popularity of adaptative step size methods in

Original: RL over fixed step size ones.
Revised: RL over fixed step size ones.

Original:  Experiments
Revised:  Experiments

Original: In all the experiments we use Φ i.i.d with normal entries.
Revised: In all the experiments we use Φ i.i.d with normal entries.

Original: Per the universality property, our theoreticalresults hold for all distributions satisfying the hypotheses in Section 2.2.
Revised: Per the universality property, our theoreticalresults hold for all distributions satisfying the hypotheses in Section 2.2.

Original: Study of the spectral distribution:
Revised: Study of the spectral distribution:

Original: On Figure 1, we validate the theory and approximations of
Revised: On Figure 1, we validate the theory and approximations of

Original: Section 3.
Revised: Section 3.

Original: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has aneigenvalue at 1 and the rest of order 1 / √ n [48].
Revised: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has aneigenvalue at 1 and the rest of order 1 / √ n [48].

Original: As predicted, the spectrum of the TD operatormatches closely a spiked MP distribution (b).
Revised: As predicted, the spectrum of the TD operatormatches closely a spiked MP distribution (b).

Original: When the graph does not satisfy the well-connectedproperty, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distributionis more complex (d).
Revised: When the graph does not satisfy the well-connectedproperty, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distributionis more complex (d).

Original: The relative error on λ +min is respectively 0 .
Revised: The relative error on λ +min is respectively 0 .

Original: 3% and 2% .
Revised: 3% and 2% .

Original: Despite this, weshow that the empirical and theoretical (eq.
Revised: Despite this, weshow that the empirical and theoretical (eq.

Original: values for λ +min and λ max on the 4 rooms domain( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so thatmax i ξ
Revised: values for λ +min and λ max on the 4 rooms domain( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so thatmax i ξ

Original: i min i ξ i = 2000 .
Revised: i min i ξ i = 2000 .

Original: While (b) and (d) show how violating the asymptotically well-connected propertybreaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, whichare the ones that matter for our optimization results, are still well predicted, even in more complexsettings.
Revised: While (b) and (d) show how violating the asymptotically well-connected propertybreaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, whichare the ones that matter for our optimization results, are still well predicted, even in more complexsettings.

Original: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.
Revised: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.

Original: We use again a4 rooms domain with a fixed stochastic policy and study the Bellman error during training.
Revised: We use again a4 rooms domain with a fixed stochastic policy and study the Bellman error during training.

Original: On all
Revised: On all

Original: [Figure]
Revised: [Figure]

Original: I m ag i na r y pa r t
Revised: I m ag i na r y pa r t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curvesfor low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than forρ ≈ 1 .
Revised: subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curvesfor low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than forρ ≈ 1 .

Original: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalueswhich governs the optimization speed is λ max / λ +min
Revised: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalueswhich governs the optimization speed is λ max / λ +min

Original: → 1 for ρ ≪ 1 and to 1 1 − γ for ρ ≫ 1 while itdiverges for ρ → 1 .
Revised: → 1 for ρ ≪ 1 and to 1 1 − γ for ρ ≫ 1 while itdiverges for ρ → 1 .

Original: This leads to a classical double descent phenomenon on (a) and (b) purely dueto optimization as predicted by [7].
Revised: This leads to a classical double descent phenomenon on (a) and (b) purely dueto optimization as predicted by [7].

Original: On (a) and (b) because either γ or d ⊤ π r is small, the terms oforder 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.
Revised: On (a) and (b) because either γ or d ⊤ π r is small, the terms oforder 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.

Original: On (c), the term 1 1 − γ ˜ ρ 283 dominates and we observe the peaking behavior of the estimation error (Section 4.3).
Revised: On (c), the term 1 1 − γ ˜ ρ 283 dominates and we observe the peaking behavior of the estimation error (Section 4.3).

Original: Furthermore
Revised: Furthermore

Original: [Figure]
Revised: [Figure]

Original: on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match theobserved peaking behavior.
Revised: on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match theobserved peaking behavior.

Original: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD( p = 200 ).
Revised: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD( p = 200 ).

Original: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .
Revised: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .

Original: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze whichcombination of step size η and ϵ would lead to convergence.
Revised: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze whichcombination of step size η and ϵ would lead to convergence.

Original: On (a), we show the discounted statedistribution for various ϵ where larger ϵ lead to more exploration.
Revised: On (a), we show the discounted statedistribution for various ϵ where larger ϵ lead to more exploration.

Original: While our theoretical argumentin Section 4.4 was about the number of visited states, we take here a more general and realisticapproach where the state visitation is controlled by the greediness of our policy.
Revised: While our theoretical argumentin Section 4.4 was about the number of visited states, we take here a more general and realisticapproach where the state visitation is controlled by the greediness of our policy.

Original: Typically, duringthe optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.
Revised: Typically, duringthe optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.

Original: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured bycomputing the spectral radius of our iteration matrix.
Revised: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured bycomputing the spectral radius of our iteration matrix.

Original: As predicted, for a fixed η , when ϵ decreases,we might go through a phase transition where the TD iteration, stable at the beginning, becomesunstable.
Revised: As predicted, for a fixed η , when ϵ decreases,we might go through a phase transition where the TD iteration, stable at the beginning, becomesunstable.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have analyzed the spectral distribution of the Temporal Difference operator whenusing random features and modeling the Markov transition matrix as the one of a random graph.
Revised: In this work, we have analyzed the spectral distribution of the Temporal Difference operator whenusing random features and modeling the Markov transition matrix as the one of a random graph.

Original: Thischaracterization allowed us to make predictive theories for the optimization error and stability of TD.
Revised: Thischaracterization allowed us to make predictive theories for the optimization error and stability of TD.

Original: Notably we are able to predict how fast we converge and whether the solution found is accurate asa function of the ratio between the number of parameters and the number of state-action pairs.
Revised: Notably we are able to predict how fast we converge and whether the solution found is accurate asa function of the ratio between the number of parameters and the number of state-action pairs.

Original: Inparticular, we highlighted and theoretically explained several phenomena specific to reinforcementlearning such as the peaking behavior of the estimation error of TD or how changing the policymight cause divergence when using a fixed step size, even in the on-policy setting.
Revised: Inparticular, we highlighted and theoretically explained several phenomena specific to reinforcementlearning such as the peaking behavior of the estimation error of TD or how changing the policymight cause divergence when using a fixed step size, even in the on-policy setting.

Original: We believe our work paves to way toward an average-case theory of reinforcement learning algorithms in the same way this was done for supervised learning and used to understand deep learning.
Revised: We believe our work paves to way toward an average-case theory of reinforcement learning algorithms in the same way this was done for supervised learning and used to understand deep learning.

Original: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations ofthe extreme eigenvalues of the MP law [42] (iii) extend our results to neural networks in the Neural
Revised: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations ofthe extreme eigenvalues of the MP law [42] (iii) extend our results to neural networks in the Neural

Original: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms.
Revised: Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms.

Original: B Appendix: Proofs
Revised: B Appendix: Proofs

Original: In the paper and in the proofs, we extend naturally the notation o ( · ) to matrices with the followingdefinition
Revised: In the paper and in the proofs, we extend naturally the notation o ( · ) to matrices with the followingdefinition

Original: [Equation]
Revised: [Equation]

Original: for ∥ · ∥ a natural operator norm.
Revised: for ∥ · ∥ a natural operator norm.

Original: In this paper we only consider the 2-operator norm which corresponds to the largest singular value.
Revised: In this paper we only consider the 2-operator norm which corresponds to the largest singular value.

Original: B.1 Deformed random graph
Revised: B.1 Deformed random graph

Original: In Section 2.3 we derive informally an expression for the Markov transition matrix of a randomgraph taking into account d π .
Revised: In Section 2.3 we derive informally an expression for the Markov transition matrix of a randomgraph taking into account d π .

Original: Let us go into more details here, and explain why we called it aninformal derivation.
Revised: Let us go into more details here, and explain why we called it aninformal derivation.

Original: Let us recall that A π = n AD π for A the original adjacency matrix and
Revised: Let us recall that A π = n AD π for A the original adjacency matrix and

Original: D π =diag ( d π ) the stationary distribution of π .
Revised: D π =diag ( d π ) the stationary distribution of π .

Original: To get ˆ P π we need to normalize the rows i = 1 , . . .
Revised: To get ˆ P π we need to normalize the rows i = 1 , . . .

Original: n of
Revised: n of

Original: A π by their sum, i.e the degree d i of each state.
Revised: A π by their sum, i.e the degree d i of each state.

Original: In the uniform case, A π = A , [48] provide first a sketch of proof (eq 1.6) based on the fact thatd i = D ii = (cid:80)
Revised: In the uniform case, A π = A , [48] provide first a sketch of proof (eq 1.6) based on the fact thatd i = D ii = (cid:80)

Original: j A ij satisfies | d i − nµ | ∈
Revised: j A ij satisfies | d i − nµ | ∈

Original: o ( n ) by the law of large numbers.
Revised: o ( n ) by the law of large numbers.

Original: This allows to ”replace”
Revised: This allows to ”replace”

Original: D by nµ I n in the proofs which ties it back to the general Wigner/i.i.d case.
Revised: D by nµ I n in the proofs which ties it back to the general Wigner/i.i.d case.

Original: In the deformed case, we cannot invoke directly the law of large numbers.
Revised: In the deformed case, we cannot invoke directly the law of large numbers.

Original: Indeed, now d i = 511 (cid:80)
Revised: Indeed, now d i = 511 (cid:80)

Original: j A πij = n (cid:80)
Revised: j A πij = n (cid:80)

Original: j A ij d π ( j ) so we have a weighted sum instead of a simple one.
Revised: j A ij d π ( j ) so we have a weighted sum instead of a simple one.

Original: We need a concentration inequality for the weighted sum of random variables.
Revised: We need a concentration inequality for the weighted sum of random variables.

Original: One can be found in[49]
Revised: One can be found in[49]

Original: Lemma B.1 (From [49]) .
Revised: Lemma B.1 (From [49]) .

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: , X n be iid copies of a real random variable X that obeys, for some p ≥
Revised: , X n be iid copies of a real random variable X that obeys, for some p ≥

Original: P ( | X | > u ) ≤ exp
Revised: P ( | X | > u ) ≤ exp

Original: [Equation]
Revised: [Equation]

Original: − u p ) for all u > 0 .
Revised: − u p ) for all u > 0 .

Original: Then there exists L > 0 such that, for any s ∈ R n , and for q the conjugate of p ( 1 p + 1 q = 1 )
Revised: Then there exists L > 0 such that, for any s ∈ R n , and for q the conjugate of p ( 1 p + 1 q = 1 )

Original: [Equation]
Revised: [Equation]

Original: In our case, for X j = c · ( A ij − µ ) (where c is chosen so the the tail assumption is satisfied),
Revised: In our case, for X j = c · ( A ij − µ ) (where c is chosen so the the tail assumption is satisfied),

Original: [Equation]
Revised: [Equation]

Original: Thus, if ∥ d π ∥ qp and ∥
Revised: Thus, if ∥ d π ∥ qp and ∥

Original: d π ∥ 22 are ∈
Revised: d π ∥ 22 are ∈

Original: o (1) , and then reasoning on − X
Revised: o (1) , and then reasoning on − X

Original: j we have that | d i n − µ | is, almostsurely when n is large, smaller than any constant t > 0 , therefore o (1) .
Revised: j we have that | d i n − µ | is, almostsurely when n is large, smaller than any constant t > 0 , therefore o (1) .

Original: Thus we get the sameargument D = nµ I n + o ( n ) .
Revised: Thus we get the sameargument D = nµ I n + o ( n ) .

Original: Note here that in high dimension the stronger condition for d π will befor p = ∞ , ∥ d π ∥ ∞ ∈
Revised: Note here that in high dimension the stronger condition for d π will befor p = ∞ , ∥ d π ∥ ∞ ∈

Original: o
Revised: o

Original: As we already make a stronger assumption later, we have chosen not toinclude it in the main text.
Revised: As we already make a stronger assumption later, we have chosen not toinclude it in the main text.

Original: All of this allows to calculate ˆ P π by normalizing A π
Revised: All of this allows to calculate ˆ P π by normalizing A π

Original: [Equation]
Revised: [Equation]

Original: When the entries of A are iid Bernoulli of mean µ , we have σ 2 = µ (1 − µ ) and W has iid whitened
Revised: When the entries of A are iid Bernoulli of mean µ , we have σ 2 = µ (1 − µ ) and W has iid whitened

Original: Bernoulli entries.
Revised: Bernoulli entries.

Original: We consider this proof informal as there are a certain number of technicalities involved when controlling the deviation of all these random quantities (as we have sketched above for the expecteddegree).
Revised: We consider this proof informal as there are a certain number of technicalities involved when controlling the deviation of all these random quantities (as we have sketched above for the expecteddegree).

Original: B.2 Asymptotically well-connected graph
Revised: B.2 Asymptotically well-connected graph

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π ,then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: Proof.
Revised: Proof.

Original: First, recall that
Revised: First, recall that

Original: [Equation]
Revised: [Equation]

Original: For ∥ · ∥ the 2-operator corresponding to the maximum singular value, we have:
Revised: For ∥ · ∥ the 2-operator corresponding to the maximum singular value, we have:

Original: [Equation]
Revised: [Equation]

Original: From [50] we have for both Wigner (Corollary 2.3.6), when the graph is undirected and i.i.d matrices (Corollary 2.3.5) when the edges are iid and directed, that There exists A, C, c >
Revised: From [50] we have for both Wigner (Corollary 2.3.6), when the graph is undirected and i.i.d matrices (Corollary 2.3.5) when the edges are iid and directed, that There exists A, C, c >

Original: [Equation]
Revised: [Equation]

Original: Which can be understood as 1 √ n W has bounded operator norm with overwhelming probability.
Revised: Which can be understood as 1 √ n W has bounded operator norm with overwhelming probability.

Original: Inparticular, taking the limit n → ∞ gives us that 1 √ n W has bounded norm almost surely whenn → ∞ .
Revised: Inparticular, taking the limit n → ∞ gives us that 1 √ n W has bounded norm almost surely whenn → ∞ .

Original: Then ∥ D π ∥ =
Revised: Then ∥ D π ∥ =

Original: ∥ d π ∥ ∞ where ∥ · ∥ ∞ is the vector max-norm.
Revised: ∥ d π ∥ ∞ where ∥ · ∥ ∞ is the vector max-norm.

Original: This is due to the fact that D π is adiagonal matrix with positive entries, therefore its largest singular value is its largest eigenvalue.
Revised: This is due to the fact that D π is adiagonal matrix with positive entries, therefore its largest singular value is its largest eigenvalue.

Original: For ∥ X π ∥ to be o (1) we need
Revised: For ∥ X π ∥ to be o (1) we need

Original: [Equation]
Revised: [Equation]

Original: As 1 − µ ≤ 1 and for nµ = d , it is sufficient to have (almost surely)
Revised: As 1 − µ ≤ 1 and for nµ = d , it is sufficient to have (almost surely)

Original: [Equation]
Revised: [Equation]

Original: B.3 Spiked Marchenko-Pastur distribution
Revised: B.3 Spiked Marchenko-Pastur distribution

Original: For, we prove small result showing that the o (1) appearing in ˆ P π remains a o (1) in 1 n Φ ⊤ ( I n −γ ˆ P π (cid:1) Φ .
Revised: For, we prove small result showing that the o (1) appearing in ˆ P π remains a o (1) in 1 n Φ ⊤ ( I n −γ ˆ P π (cid:1) Φ .

Original: Lemma B.2.
Revised: Lemma B.2.

Original: For ˆ P π satisfying the asymptotically well-connected condition, we have
Revised: For ˆ P π satisfying the asymptotically well-connected condition, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For ∥ · ∥ = ∥ · ∥ 2 the 2-operator norm which corresponds to the largest singular value, wehave
Revised: For ∥ · ∥ = ∥ · ∥ 2 the 2-operator norm which corresponds to the largest singular value, wehave

Original: [Equation]
Revised: [Equation]

Original: As the maximum eigenvalue of 1 n Φ ⊤ Φ converges a.s to (1+ √ ρ ) 2 per the Marchenko-Pastur theorem
Revised: As the maximum eigenvalue of 1 n Φ ⊤ Φ converges a.s to (1+ √ ρ ) 2 per the Marchenko-Pastur theorem

Original: [Equation]
Revised: [Equation]

Original: This lemma is a useful result to understand that a o (1) perturbation stays a o (1) perturbation whenadding features.
Revised: This lemma is a useful result to understand that a o (1) perturbation stays a o (1) perturbation whenadding features.

Original: Proposition B.1 (Asymmetric rank 1 perturbation) .
Revised: Proposition B.1 (Asymmetric rank 1 perturbation) .

Original: We consider the spectrum of 1 n Φ ⊤ ( I n −γ uv ⊤ )Φ for u , v ∈ R n so that v ⊤ u = 1 and γ ∈ [0 , 1[ which is a rank 1 perturbation of the
Revised: We consider the spectrum of 1 n Φ ⊤ ( I n −γ uv ⊤ )Φ for u , v ∈ R n so that v ⊤ u = 1 and γ ∈ [0 , 1[ which is a rank 1 perturbation of the

Original: Wishart covariance model.
Revised: Wishart covariance model.

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ converges to the

Original: Marchenko-Pastur law.
Revised: Marchenko-Pastur law.

Original: When ργ 2 ≥ 1 , the empirical spectral distribution also converges to the
Revised: When ργ 2 ≥ 1 , the empirical spectral distribution also converges to the

Original: Marchenko-Pastur law, but additionally, we observe a phase transition [42], and an eigenvalueλ spiked separates from the bulk
Revised: Marchenko-Pastur law, but additionally, we observe a phase transition [42], and an eigenvalueλ spiked separates from the bulk

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is based on a simple adaptation of the proof in [51] (page 117).
Revised: The proof is based on a simple adaptation of the proof in [51] (page 117).

Original: As we consider anasymmetric spike, we lose commutative properties.
Revised: As we consider anasymmetric spike, we lose commutative properties.

Original: Thus, we can only prove our result in the casewhere there is a unique spike instead of a finite number as is usual.
Revised: Thus, we can only prove our result in the casewhere there is a unique spike instead of a finite number as is usual.

Original: We are looking to characterizethe eigenvalues of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ , or equivalently (up to some zeros), the eigenvalues of1 n ΦΦ ⊤ ( I n − γ uv ⊤ ) .
Revised: We are looking to characterizethe eigenvalues of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ , or equivalently (up to some zeros), the eigenvalues of1 n ΦΦ ⊤ ( I n − γ uv ⊤ ) .

Original: However, in this case, 1 n ΦΦ ⊤ is not a sample from the Wishart distribution,but 1 p ΦΦ ⊤ is.
Revised: However, in this case, 1 n ΦΦ ⊤ is not a sample from the Wishart distribution,but 1 p ΦΦ ⊤ is.

Original: We therefore analyze 1 p ΦΦ ⊤ and will simply have to scale our spectrum by ρ at theend.0 = det
Revised: We therefore analyze 1 p ΦΦ ⊤ and will simply have to scale our spectrum by ρ at theend.0 = det

Original: [Equation]
Revised: [Equation]

Original: Using Sherman-Morrison formula, we have ( I n − γ uv ⊤ ) − 1 = I n + γ 1 − γ uv ⊤ as v ⊤ u = 1 .
Revised: Using Sherman-Morrison formula, we have ( I n − γ uv ⊤ ) − 1 = I n + γ 1 − γ uv ⊤ as v ⊤ u = 1 .

Original: Since det (cid:0) I n − γ uv ⊤ ) (cid:1) ̸ = 0 , the second determinant can be discarded.
Revised: Since det (cid:0) I n − γ uv ⊤ ) (cid:1) ̸ = 0 , the second determinant can be discarded.

Original: Then, we can factor the resolvent of the Wishart model.
Revised: Then, we can factor the resolvent of the Wishart model.

Original: That is, letting Q (ˆ λ ) =
Revised: That is, letting Q (ˆ λ ) =

Original: (cid:16) 1 p ΦΦ ⊤ −
Revised: (cid:16) 1 p ΦΦ ⊤ −

Original: ˆ λ I n (cid:17) − 1 , we write
Revised: ˆ λ I n (cid:17) − 1 , we write

Original: [Equation]
Revised: [Equation]

Original: Inverting 1 p ΦΦ ⊤ − ˆ λ
Revised: Inverting 1 p ΦΦ ⊤ − ˆ λ

Original: I n is almost surely possible when ˆ λ < (1 − (cid:112) 1 / ρ ) 2 per the concentration ofthe empirical spectral distribution of Wishart covariance models to the Marchenko-Pastur law andthe term Q − 1 (ˆ λ ) is almost surely non-zero.
Revised: I n is almost surely possible when ˆ λ < (1 − (cid:112) 1 / ρ ) 2 per the concentration ofthe empirical spectral distribution of Wishart covariance models to the Marchenko-Pastur law andthe term Q − 1 (ˆ λ ) is almost surely non-zero.

Original: So the left term can be discarded.
Revised: So the left term can be discarded.

Original: For the second weuse Sylvester’s determinant identity to end up with a scalar equation
Revised: For the second weuse Sylvester’s determinant identity to end up with a scalar equation

Original: [Equation]
Revised: [Equation]

Original: As Q (ˆ λ ) is a deterministic equivalent of m (ˆ λ ) I n ([18], Theorem 2.4 in [51]), where m ( · ) is the
Revised: As Q (ˆ λ ) is a deterministic equivalent of m (ˆ λ ) I n ([18], Theorem 2.4 in [51]), where m ( · ) is the

Original: Stieljes transform of the MP distribution, thus
Revised: Stieljes transform of the MP distribution, thus

Original: [Equation]
Revised: [Equation]

Original: The above determinant is zero when
Revised: The above determinant is zero when

Original: [Equation]
Revised: [Equation]

Original: From the definition of m ( · ) as the Stieljes transform of the MP distribution, we know [51] it satisfies
Revised: From the definition of m ( · ) as the Stieljes transform of the MP distribution, we know [51] it satisfies

Original: [Equation]
Revised: [Equation]

Original: for c = lim n, p → ∞ n / p = 1 / ρ
Revised: for c = lim n, p → ∞ n / p = 1 / ρ

Original: Thus, for z = ˆ λ
Revised: Thus, for z = ˆ λ

Original: [Equation]
Revised: [Equation]

Original: Plugging the equation above, asymptotically
Revised: Plugging the equation above, asymptotically

Original: [Equation]
Revised: [Equation]

Original: Now, we have to remember to multiply our spectrum by ρ to obtain the original one, and so weobtain
Revised: Now, we have to remember to multiply our spectrum by ρ to obtain the original one, and so weobtain

Original: [Equation]
Revised: [Equation]

Original: When (1 − γ )( ρ − 1 γ ) ≤ (1 − √ ρ ) 2 which is equivalent to ργ 2 ≥ 1 , this eigenvalue is isolated onthe left of the bulk.
Revised: When (1 − γ )( ρ − 1 γ ) ≤ (1 − √ ρ ) 2 which is equivalent to ργ 2 ≥ 1 , this eigenvalue is isolated onthe left of the bulk.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n −  ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to
Revised: When ργ 2 ≥ 1 , there is a phase transition [42] for theminimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We essentially do the same proof as above, but with an additional o (1) moving around andwe show that it does not impact the Stieljes transform, thus the spectrum, almost surely.
Revised: We essentially do the same proof as above, but with an additional o (1) moving around andwe show that it does not impact the Stieljes transform, thus the spectrum, almost surely.

Original: [Equation]
Revised: [Equation]

Original: Where we used the definition of ˆ P π and Lemma B.2.
Revised: Where we used the definition of ˆ P π and Lemma B.2.

Original: Then0 =
Revised: Then0 =

Original: det
Revised: det

Original: [Equation]
Revised: [Equation]

Original: As the eigenvalues of I n − γ 1 n d ⊤ π are lower bounded by 1 − γ for n large enough det( I n − γ + o (1))must be non-zero.
Revised: As the eigenvalues of I n − γ 1 n d ⊤ π are lower bounded by 1 − γ for n large enough det( I n − γ + o (1))must be non-zero.

Original: As ( I n + o (1)) − 1 =
Revised: As ( I n + o (1)) − 1 =

Original: (cid:80) i
Revised: (cid:80) i

Original: ( o (1)) i
Revised: ( o (1)) i

Original: = o (1) , using Sherman-Morrison formula, wehave ( I n + o (1) −γ
Revised: = o (1) , using Sherman-Morrison formula, wehave ( I n + o (1) −

Original: [Equation]
Revised: [Equation]

Original: n d ⊤π
Revised: d ⊤

Original: [Equation]
Revised: [Equation]

Original: d

Original: [Equation]

Original: With the same derivation and arguments as above
Revised: With the same derivation and arguments as above

Original: [Equation]
Revised: [Equation]

Original: And finally we end up with the same equality as before where the o (1) from the perturbation andthe one from the deterministic equivalent fuse together.
Revised: And finally we end up with the same equality as before where the o (1) from the perturbation andthe one from the deterministic equivalent fuse together.

Original: [Equation]
Revised: [Equation]

Original: Therefore the Stieljes transforms when using ˆ P π and 1 n d ⊤ π are asymptotically the same, thus, almost surely, they have the same empirical spectral distribution.
Revised: Therefore the Stieljes transforms when using ˆ P π and 1 n d ⊤ π are asymptotically the same, thus, almost surely, they have the same empirical spectral distribution.

Original: B.4 Approximation of the extreme eigenvalues
Revised: B.4 Approximation of the extreme eigenvalues

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy asecular equation [43]
Revised: γ 1 n d ⊤ π ) satisfy asecular equation [43]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us consider v ̸ = 0 and λ , an eigenvector of Ξ( I n − γ 1 n d ⊤ π ) and its associated eigenvalue.
Revised: Let us consider v ̸ = 0 and λ , an eigenvector of Ξ( I n − γ 1 n d ⊤ π ) and its associated eigenvalue.

Original: [Equation]
Revised: [Equation]

Original: This give on component i
Revised: This give on component i

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Assuming λ ̸ = ξ i , ∀
Revised: Assuming λ ̸ = ξ i , ∀

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Multiplying by d π ( i ) and summing
Revised: Multiplying by d π ( i ) and summing

Original: [Equation]
Revised: [Equation]

Original: Now the same term (cid:80) j d π ( j ) v j = d ⊤ π v appears on both sides.
Revised: Now the same term (cid:80) j d π ( j ) v j = d ⊤ π v appears on both sides.

Original: Let us verify if it can be null.
Revised: Let us verify if it can be null.

Original: If thatwere the case, we would have Ξ( I n − γ 1 n d ⊤ π ) v =
Revised: If thatwere the case, we would have Ξ( I n − γ 1 n d ⊤ π ) v =

Original: Ξ v = λ v .
Revised: Ξ v = λ v .

Original: Thus, as Ξ is diagonal, the possiblevectors v possible are the canonical basis vectors and the possible eigenvalues are ξ i .
Revised: Thus, as Ξ is diagonal, the possiblevectors v possible are the canonical basis vectors and the possible eigenvalues are ξ i .

Original: However, asd π is a positive vector whose entries sum to 1, we cannot have d ⊤ π v = 0 .
Revised: However, asd π is a positive vector whose entries sum to 1, we cannot have d ⊤ π v = 0 .

Original: Thus d ⊤ π v ̸ = 0 and bydividing on each side:
Revised: Thus d ⊤ π v ̸ = 0 and bydividing on each side:

Original: [Equation]
Revised: [Equation]

Original: B.5 Optimization and approximation error
Revised: B.5 Optimization and approximation error

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ),we have the following decomposition
Revised: (2) ),we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: 10) where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ
Revised: 10) where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Where we used ∥ A ∥ 2 ≤ √ n ∥
Revised: Where we used ∥ A ∥ 2 ≤ √ n ∥

Original: A ∥ ∞ and the fact that P π is a stochastic matrix, thus has entriesbetween 0 and 1.
Revised: A ∥ ∞ and the fact that P π is a stochastic matrix, thus has entriesbetween 0 and 1.

Original: As such, the entries of I n − γ P π are between − γ and 1 , in particular they arebounded by 1 .
Revised: As such, the entries of I n − γ P π are between − γ and 1 , in particular they arebounded by 1 .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and thatits spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η =
Revised: Denoting by λ +min and λ max its smallest non-zero and largesteigenvalues, for η =

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: Proof.
Revised: Proof.

Original: Let us begin with the case p > n From the TD iteration Equation (2) multiplied on the leftby Φ
Revised: Let us begin with the case p > n From the TD iteration Equation (2) multiplied on the leftby Φ

Original: [Equation]
Revised: [Equation]

Original: For a i.i.d matrix Φ , we have that ΦΦ ⊤ is almost surely invertible.
Revised: For a i.i.d matrix Φ , we have that ΦΦ ⊤ is almost surely invertible.

Original: Thus the fixed point of thisiteration is
Revised: Thus the fixed point of thisiteration is

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: Thus, for H TD =
Revised: Thus, for H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ and x t = q t − q π
Revised: Φ ⊤ Ξ( I n − γ P π )Φ and x t = q t − q π

Original: [Equation]
Revised: [Equation]

Original: As I n − η Λ is diagonal, and as such its largest singular value (2-norm) is its largest eigenvalue (inmodulus).
Revised: As I n − η Λ is diagonal, and as such its largest singular value (2-norm) is its largest eigenvalue (inmodulus).

Original: Now, we supposed the eigenvalues were real, max i,λ i =
Revised: Now, we supposed the eigenvalues were real, max i,λ i =

Original: Λ ii {| 1 − ηλ i |} will be minimized when thelargest and smallest terms are equal.
Revised: Λ ii {| 1 − ηλ i |} will be minimized when thelargest and smallest terms are equal.

Original: [Equation]
Revised: [Equation]

Original: If λ +min = λ max this can be achieved for any η , however the only solution when they are different isto have the two expressions above of a different sign
Revised: If λ +min = λ max this can be achieved for any η , however the only solution when they are different isto have the two expressions above of a different sign

Original: [Equation]
Revised: [Equation]

Original: Which leads to η =
Revised: Which leads to η =

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Now for p < n we do a similar reasoning with θ .
Revised: Now for p < n we do a similar reasoning with θ .

Original: By the exact same reasoning, but using H TD =
Revised: By the exact same reasoning, but using H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ instead and the fixed point θ TD = (cid:0) Φ ⊤ Ξ( I n − γ P π )Φ (cid:1) 1 Φ ⊤ Ξ
Revised: Φ ⊤ Ξ( I n − γ P π )Φ instead and the fixed point θ TD = (cid:0) Φ ⊤ Ξ( I n − γ P π )Φ (cid:1) 1 Φ ⊤ Ξ

Original: r
Revised: r

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Now, we would like to obtain a result for the Q-function associated to these parameters.
Revised: Now, we would like to obtain a result for the Q-function associated to these parameters.

Original: We have σ min (Φ)
Revised: We have σ min (Φ)

Original: ∥ x ∥ 2 ≤ ∥ Φ x ∥ 2 ≤ σ max (Φ) ∥ x ∥ 2 for σ max and σ min the max and min singular values.
Revised: ∥ x ∥ 2 ≤ ∥ Φ x ∥ 2 ≤ σ max (Φ) ∥ x ∥ 2 for σ max and σ min the max and min singular values.

Original: Thus, multiplying each side by σ min (Φ) σ max (Φ) , we get
Revised: Thus, multiplying each side by σ min (Φ) σ max (Φ) , we get

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Squaring these results and putting it back in the contraction equation, we get for q TD =
Revised: Squaring these results and putting it back in the contraction equation, we get for q TD =

Original: Φ θ TD
Revised: Φ θ TD

Original: [Equation]
Revised: [Equation]

Original: B.6 Peaking behavior of the estimation error
Revised: B.6 Peaking behavior of the estimation error

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =
Revised: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =

Original: I n − Π is its complement, and ˜ ρ =
Revised: I n − Π is its complement, and ˜ ρ =

Original: d ⊤ π Π 1 n .
Revised: d ⊤ π Π 1 n .

Original: Proof.
Revised: Proof.

Original: Let us look at the case p < n .
Revised: Let us look at the case p < n .

Original: We have q TD =
Revised: We have q TD =

Original: Φ (cid:0) Φ ⊤ Ξ( I − γ P π )Φ (cid:1) − 1 Φ ⊤ Ξ r We will therefore study more closely the matrix Φ (cid:0) Φ ⊤ Ξ( I − γ P )Φ (cid:1) − 1 Φ ⊤ Ξ , in the case where P π = uv ⊤ is a rank one matrix.
Revised: Φ (cid:0) Φ ⊤ Ξ( I − γ P π )Φ (cid:1) − 1 Φ ⊤ Ξ r We will therefore study more closely the matrix Φ (cid:0) Φ ⊤ Ξ( I − γ P )Φ (cid:1) − 1 Φ ⊤ Ξ , in the case where P π = uv ⊤ is a rank one matrix.

Original: First, Sherman-Morrison formula gives us
Revised: First, Sherman-Morrison formula gives us

Original: [Equation]
Revised: [Equation]

Original: which is the usual formula but for x ≜ − γ x .
Revised: which is the usual formula but for x ≜ − γ x .

Original: Note that this is licit as long as 1 −
Revised: Note that this is licit as long as 1 −

Original: γ y ⊤ A − 1 x ̸ = 0 .
Revised: γ y ⊤ A − 1 x ̸ = 0 .

Original: Using Sherman-Morrison’s formula
Revised: Using Sherman-Morrison’s formula

Original: [Equation]
Revised: [Equation]

Original: For Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ the usual projector unto the span of Φ in metric Ξ .
Revised: For Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ the usual projector unto the span of Φ in metric Ξ .

Original: Now, for the TD error we obtain
Revised: Now, for the TD error we obtain

Original: [Equation]
Revised: [Equation]

Original: Let’s call I −
Revised: Let’s call I −

Original: Π
Revised: Π

Original: =
Revised: =

Original: Π ⊥
Revised: Π ⊥

Original: the complement projector projector and ˜ ρ = v ⊤ Π u .
Revised: the complement projector projector and ˜ ρ = v ⊤ Π u .

Original: Now, for the MSBEerror, we get
Revised: Now, for the MSBEerror, we get

Original: [Equation]
Revised: [Equation]

Original: where we use the fact that Π ⊤ ΞΠ = ΞΠ = Π ⊤ Ξ and same for Π ⊥ .
Revised: where we use the fact that Π ⊤ ΞΠ = ΞΠ = Π ⊤ Ξ and same for Π ⊥ .

Original: Using v = d π and u = 1 n gives us the approximation.
Revised: Using v = d π and u = 1 n gives us the approximation.

Original: C Colab links
Revised: C Colab links

Original: For plotting the spiked MP spectra https://colab.research.google.com/drive
Revised: For plotting the spiked MP spectra https://colab.research.google.com/drive

Original: For the extreme eigenvalues approximation https://colab.research.google.com/drive/1H_GkjzlGyUxB0i55jM5VAqLXoEGGxN3h?usp=sharing
Revised: For the extreme eigenvalues approximation https://colab.research.google.com/drive/1H_GkjzlGyUxB0i55jM5VAqLXoEGGxN3h?usp=sharing

Original: For the peaking behavior https://colab.research.google.com/drive/1EVudyGfieXOUqDLifdhFrs8Pu1OzKOAW?usp=sharing
Revised: For the peaking behavior https://colab.research.google.com/drive/1EVudyGfieXOUqDLifdhFrs8Pu1OzKOAW?usp=sharing

Original: For the peaking behavior model https://colab.research.google.com/drive/1pLIqaV5BO1ChIj9y9UmP5jmlWx26FNI7?usp=sharing
Revised: For the peaking behavior model https://colab.research.google.com/drive/1pLIqaV5BO1ChIj9y9UmP5jmlWx26FNI7?usp=sharing

Original: For the unstability wrt η and ϵ https://colab.research.google.com/drive/1CuPZhv3eZoINXHCKLtwjVlHrLy9ylBuI?usp=sharing
Revised: For the unstability wrt η and ϵ https://colab.research.google.com/drive/1CuPZhv3eZoINXHCKLtwjVlHrLy9ylBuI?usp=sharing