Original: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation
Revised: On the role of overparameterization in off-policy Temporal Difference learning with linear function approximation

Original: Abstract
Revised: Abstract

Original: Much of the recent successes of deep learning can be attributed to scaling up the size of the networks to the point where they often are vastly overparameterized.
Revised: Much of the recent successes of deep learning can be attributed to scaling up the size of the networks to the point where they often are vastly overparameterized.

Original: Thus, understanding the role of overparameterization is of increasing importance.
Revised: Thus, understanding the role of overparameterization is of increasing importance.

Original: While predictive theories have been developed for supervised learning, little is known about the Reinforcement Learning case.
Revised: While predictive theories have been developed for supervised learning, little is known about the Reinforcement Learning case.

Original: In this work, we take a theoretical approach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.
Revised: In this work, we take a theoretical approach and study the role of overparameterization for off-policy Temporal Difference (TD) learning in the linear setting.

Original: We leverage tools from random matrix theory and random graph theory to obtain a characterization of the spectrum of the
Revised: We leverage tools from random matrix theory and random graph theory to obtain a characterization of the spectrum of the

Original: TD operator.
Revised: TD operator.

Original: We use this result to study the stability and optimization dynamics of TD learning as a function of the number of parameters.
Revised: We use this result to study the stability and optimization dynamics of TD learning as a function of the number of parameters.

Original:  Introduction
Revised:  Introduction

Original: Occam’s razor principle states that among many plausible explanations, the simplest one is the most likely to be true.
Revised: Occam’s razor principle states that among many plausible explanations, the simplest one is the most likely to be true.

Original: In statistics and machine learning, this is often interpreted as models with a restricted number of parameters should be privileged as they generalize better (Akaike, 1974).
Revised: In statistics and machine learning, this is often interpreted as models with a restricted number of parameters should be privileged as they generalize better (Akaike, 1974).

Original: This view is supported by the classical bias-variance trade-off which implies that models that are too flexible are bound to overfit (Geman et al., 1992).
Revised: This view is supported by the classical bias-variance trade-off which implies that models that are too flexible are bound to overfit (Geman et al., 1992).

Original: Yet, deep neural networks with more parameters than training data points achieved remarkable performance in many domains (LeCun et al., 1989; Krizhevsky et al., 2012;
Revised: Yet, deep neural networks with more parameters than training data points achieved remarkable performance in many domains (LeCun et al., 1989; Krizhevsky et al., 2012;

Original: Devlin et al., 2018), challenging this conventional wisdom.
Revised: Devlin et al., 2018), challenging this conventional wisdom.

Original: A significant understanding of this apparent tension has been reached in recent years.
Revised: A significant understanding of this apparent tension has been reached in recent years.

Original: A large body of work using tools from random matrix theory has shown that, in models with random features, overparameterization does not lead to overfitting but instead better generalization (Belkin et al., 2020).
Revised: A large body of work using tools from random matrix theory has shown that, in models with random features, overparameterization does not lead to overfitting but instead better generalization (Belkin et al., 2020).

Original: This phenomenon is known as double descent and unifies the traditional and modern understanding of bias and variance.
Revised: This phenomenon is known as double descent and unifies the traditional and modern understanding of bias and variance.

Original: Importantly, Kuzborskij et al.
Revised: Importantly, Kuzborskij et al.

Original: (2021) has shown that the double descent phenomenon is in large part explained by purely optimization rather than label noise.
Revised: (2021) has shown that the double descent phenomenon is in large part explained by purely optimization rather than label noise.

Original: Following their insight, we mainly take an optimization approach in the rest of our paper.
Revised: Following their insight, we mainly take an optimization approach in the rest of our paper.

Original: While we now have a theory of overparameterization in supervised learning, the theoretical analysis of overparameterization in the reinforcement learning (RL) setting is largely unexplored (Xiao et al., 2021).
Revised: While we now have a theory of overparameterization in supervised learning, the theoretical analysis of overparameterization in the reinforcement learning (RL) setting is largely unexplored (Xiao et al., 2021).

Original: Indeed, RL poses several new additional challenges.
Revised: Indeed, RL poses several new additional challenges.

Original: First, the agent has to interact with an environment whose structure may be complex.
Revised: First, the agent has to interact with an environment whose structure may be complex.

Original: Second, each state may not be sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.
Revised: Second, each state may not be sampled with equal probability while each datapoint is usually sampled uniformly in supervised learning.

Original: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.
Revised: In this work, we focus mainly on policy evaluation , i.e evaluating the expected return of a fixed policy.

Original: More precisely, we study the behavior of Temporal Difference learning (Sutton, 1988) which has become ubiquitous for policy evaluation in modern Deep RL (Mnih et al., 2013; Haarnoja et al., 2018).
Revised: More precisely, we study the behavior of Temporal Difference learning (Sutton, 1988) which has become ubiquitous for policy evaluation in modern Deep RL (Mnih et al., 2013; Haarnoja et al., 2018).

Original: Our contributions are, in the limit of large number of states and parameters: 1) We introduce a class of random graphs for which we can compute analytically the spectrum of the TD operator 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: Our contributions are, in the limit of large number of states and parameters: 1) We introduce a class of random graphs for which we can compute analytically the spectrum of the TD operator 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: when using uniform off-policy sampling, 2) we derive approximations for the largest and smallest eigenvalue in the non-uniform case.
Revised: when using uniform off-policy sampling, 2) we derive approximations for the largest and smallest eigenvalue in the non-uniform case.

Original: Furthermore, we analyze the behavior of the traditional Mean
Revised: Furthermore, we analyze the behavior of the traditional Mean

Original: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a double descent phenomenon similar to supervised learning but with a multiplicative factor depending on the discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displays a peaking behavior near the interpolation threshold (when the number of parameters approaches the number of states) that is specific to RL.
Revised: Square Bellman Error (MSBE) when learning with TD(0) and show that 3) it exhibits a double descent phenomenon similar to supervised learning but with a multiplicative factor depending on the discount factor in the overparameterized regime and 4) the asymptotic residual MSBE displays a peaking behavior near the interpolation threshold (when the number of parameters approaches the number of states) that is specific to RL.

Original: Finally, 5) we showcase how the non-stationarity of the policy can make the optimization of the value function unstable.
Revised: Finally, 5) we showcase how the non-stationarity of the policy can make the optimization of the value function unstable.

Original: All of these points are motivated theoretically and empirically validated on small gridworlds MDPs.
Revised: All of these points are motivated theoretically and empirically validated on small gridworlds MDPs.

Original:  Background 2.1 Value estimation
Revised:  Background 2.1 Value estimation

Original: Definition: We place ourselves in the Markov Decision Process (MDP) setting (Puterman, 2014) defined by {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces,
Revised: Definition: We place ourselves in the Markov Decision Process (MDP) setting (Puterman, 2014) defined by {S , A , P sa → s , r , γ, p 0 } , where S and A denote the finite state and action spaces,

Original: P sa → s ∈ R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.
Revised: P sa → s ∈ R |S|·|A|×|S| is the environment transition probability matrix and r ∈ R |S|·|A| denotes the (deterministic) reward vector.

Original: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.
Revised: γ ∈ [0 , 1[ is the discount factor, and lastly, p 0 is the initial state distribution.

Original: For a given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be able to estimate the expected discounted return q π ( s, a ) =
Revised: For a given policy π , i.e a distribution over actions given a state, the goal of value estimation is to be able to estimate the expected discounted return q π ( s, a ) =

Original: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s 

Revised: E P sa → s ,π (cid:2) (cid:80) t γ t r ( s t , a t ) | s

Original: = s, a 0 = a ] .
Revised: = s, a 0 = a ] .

Original: This quantity is also known as the state-action value or the Q-function.
Revised: This quantity is also known as the state-action value or the Q-function.

Original: Furthermore, it can be written as the solution of the Bellman equation (Bellman, 1954) 1 : q π ( s, a ) =
Revised: Furthermore, it can be written as the solution of the Bellman equation (Bellman, 1954) 1 : q π ( s, a ) =

Original: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .
Revised: r ( s, a ) + γ E P sa → s ,π [ q π ( s ′ , a ′ )] .

Original: By defining P π ∈ R |S|·|A|×|S|·|A|
Revised: By defining P π ∈ R |S|·|A|×|S|·|A|

Original: the state-action to state-action transition matrix, we can write it in matrix form
Revised: the state-action to state-action transition matrix, we can write it in matrix form

Original: [Equation]
Revised: [Equation]

Original: Linear function approximation: While Equation (1) can be solved efficiently when r , P π are known and when n = |S|·|A| is not too large, in many cases this cannot be computed.
Revised: Linear function approximation: While Equation (1) can be solved efficiently when r , P π are known and when n = |S|·|A| is not too large, in many cases this cannot be computed.

Original: One solution is to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .
Revised: One solution is to introduce a parameter vector θ ∈ R p and a feature matrix Φ ∈ R n × p and optimize θ to approximate q π as q π ≈ Φ θ .

Original: This is referred to as the Linear Function Approximation (LFA) setting.
Revised: This is referred to as the Linear Function Approximation (LFA) setting.

Original: Temporal Difference with LFA:
Revised: Temporal Difference with LFA:

Original: Estimating the Q-function in the Linear Function Approximation setting can be done using the well-known algorithm TD(0)
Revised: Estimating the Q-function in the Linear Function Approximation setting can be done using the well-known algorithm TD(0)

Original: (Sutton, 1988) whose expected update is
Revised: (Sutton, 1988) whose expected update is

Original: [Equation]
Revised: [Equation]

Original: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summing to 1 , weighting each state-action by their probability of being sampled.
Revised: where Ξ is the off-policy distribution matrix, a n × n diagonal matrix with positive entries summing to 1 , weighting each state-action by their probability of being sampled.

Original: Contrary to gradient descent with an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0)
Revised: Contrary to gradient descent with an appropriate step-size, TD(0) is not guaranteed to converge as the matrix Φ ⊤ Ξ (cid:0)

Original: I n − γ P π )Φ might have a negative eigenvalue.
Revised: I n − γ P π )Φ might have a negative eigenvalue.

Original: This phenomenon is known as the deadly triad as it can only happen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on the next value as TD(0) does (Baird, 1995; Tsitsiklis & Van Roy, 1997; Van Hasselt et al., 2018).
Revised: This phenomenon is known as the deadly triad as it can only happen when we are (i) off-policy, (ii) using function approximation and (iii) bootstrapping on the next value as TD(0) does (Baird, 1995; Tsitsiklis & Van Roy, 1997; Van Hasselt et al., 2018).

Original: 2.2 Random Matrix Theory
Revised: 2.2 Random Matrix Theory

Original: Wishart matrices:
Revised: Wishart matrices:

Original: Extremely relevant in the study of least squares with random features are matrices from the Wishart distribution (Wishart, 1928).
Revised: Extremely relevant in the study of least squares with random features are matrices from the Wishart distribution (Wishart, 1928).

Original: These matrices can be written as 1 n Φ ⊤ Φ for Φ a n × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.
Revised: These matrices can be written as 1 n Φ ⊤ Φ for Φ a n × p matrix with entries of zero mean, unit variance and bounded 4 th moment sampled independently and identically distributed.

Original: In that case, when p, n → ∞ and the ratio converges to a finite limit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the
Revised: In that case, when p, n → ∞ and the ratio converges to a finite limit lim p,n →∞ p / n = ρ ∈ ]0 , + ∞ [ , the empirical spectral distribution converges to the

Original: Marchenko-Pastur (Marˇcenko & Pastur, 1967) (MP) distribution whose density can be written as
Revised: Marchenko-Pastur (Marˇcenko & Pastur, 1967) (MP) distribution whose density can be written as

Original: [Equation]
Revised: [Equation]

Original: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x ) are the 0 eigenvalues, which exist when p ≥ n , λ max
Revised: where 1 A ( · ) is the indicator function, valued at 1 if x ∈ A , 0 otherwise and max { 1 − 1 / ρ , 0 } δ { 0 } ( x ) are the 0 eigenvalues, which exist when p ≥ n , λ max

Original: =
Revised: =

Original: (1 + √ ρ ) 2 , the largest, and λ +min = (1 − √ ρ ) 2 , the smallest non-zero eigenvalue.
Revised: (1 + √ ρ ) 2 , the largest, and λ +min = (1 − √ ρ ) 2 , the smallest non-zero eigenvalue.

Original: Wigner-type matrices:
Revised: Wigner-type matrices:

Original: Wigner (1955) historically introduced and studied symmetric n × n matrices of the form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance for i ≥ j 2 .
Revised: Wigner (1955) historically introduced and studied symmetric n × n matrices of the form W where W ij , i ≥ j are sampled i.i.d from a law of mean 0 and unit variance for i ≥ j 2 .

Original: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) = 12 π √ 4 − x 2 1 [ − 2 , 2] dx .
Revised: 1 √ n W has a real spectrum which converges to the semi-circle distribution dµ ( x ) = 12 π √ 4 − x 2 1 [ − 2 , 2] dx .

Original: When non-symmetric, the spectrum converges to the complex unit disk { z ∈
Revised: When non-symmetric, the spectrum converges to the complex unit disk { z ∈

Original: C , | z | ≤ 1 } 3 (Tao et al., 2010).
Revised: C , | z | ≤ 1 } 3 (Tao et al., 2010).

Original: 2.3 Random graphs
Revised: 2.3 Random graphs

Original: Random graphs are useful models to infer structural or spectral properties of typical graphs.
Revised: Random graphs are useful models to infer structural or spectral properties of typical graphs.

Original: The earliest theoretical analyses of random graphs that we are aware of were done independently by Erd˝os et al.
Revised: The earliest theoretical analyses of random graphs that we are aware of were done independently by Erd˝os et al.

Original: (1960) and Gilbert (1959).
Revised: (1960) and Gilbert (1959).

Original: The model introduced in Erd˝os et al.
Revised: The model introduced in Erd˝os et al.

Original: (1960) is a random graph where each of the n nodes is connected at random to d others.
Revised: (1960) is a random graph where each of the n nodes is connected at random to d others.

Original: The second model (Gilbert, 1959) is slightly different as each node is independently connected to any other with a probability µ ∈ ]0 , 1[ .
Revised: The second model (Gilbert, 1959) is slightly different as each node is independently connected to any other with a probability µ ∈ ]0 , 1[ .

Original: We refer to the first one as the d -regular Erd˝os-R´enyi model G ( n, d ) and the second one as the
Revised: We refer to the first one as the d -regular Erd˝os-R´enyi model G ( n, d ) and the second one as the

Original: Erd˝os-R´enyi-Gilbert G ( n, µ ) model.
Revised: Erd˝os-R´enyi-Gilbert G ( n, µ ) model.

Original: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct a matrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rows summing to 1 ).
Revised: Given the adjacency matrix A , with A ij = 1 if i is connected to j , it is possible to construct a matrix P by renormalizing the rows of A so that P is a stochastic matrix (positive entries and rows summing to 1 ).

Original: P corresponds to the Markov transition matrix of a random walk on the graph and is connected to the (left)-normalized Laplacian L by L
Revised: P corresponds to the Markov transition matrix of a random walk on the graph and is connected to the (left)-normalized Laplacian L by L

Original: =
Revised: =

Original: I n − P .
Revised: I n − P .

Original: For many classes of random graphs, the spectrum of these quantities is known (Zhao, 2012; Tran et al., 2010).
Revised: For many classes of random graphs, the spectrum of these quantities is known (Zhao, 2012; Tran et al., 2010).

Original:  Spectrum of TD: from random graphs to spiked models 3.1 From Ordinary Least Squares to TD(0)
Revised:  Spectrum of TD: from random graphs to spiked models 3.1 From Ordinary Least Squares to TD(0)

Original: Random matrix models have been used in supervised learning, especially in the simple least squares model (Cun et al., 1991;
Revised: Random matrix models have been used in supervised learning, especially in the simple least squares model (Cun et al., 1991;

Original: Hastie et al., 2019; Derezinski et al., 2020), to explain phenomena arising in deep learning.
Revised: Hastie et al., 2019; Derezinski et al., 2020), to explain phenomena arising in deep learning.

Original: Indeed, while much simpler than large neural networks, these models can shed light on the optimization and generalization behavior of vastly overparameterized models (Pennington & Bahri, 2017; Wei et al., 2022), something that most traditional complexity measures could not (Zhang et al., 2021; Dziugaite et al., 2020).
Revised: Indeed, while much simpler than large neural networks, these models can shed light on the optimization and generalization behavior of vastly overparameterized models (Pennington & Bahri, 2017; Wei et al., 2022), something that most traditional complexity measures could not (Zhang et al., 2021; Dziugaite et al., 2020).

Original: Using gradient descent on a least square problem, our parameter update can be written as
Revised: Using gradient descent on a least square problem, our parameter update can be written as

Original: [Equation]
Revised: [Equation]

Original: for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.
Revised: for Φ : R n × p the feature matrix and θ ∗ the solution found by gradient descent.

Original: When choosing the learning rate η optimally, the convergence rate is expressed as λ max / λ +min − 1 λ max / λ +min +1 for λ +min , respectively λ max , the smallest, resp.
Revised: When choosing the learning rate η optimally, the convergence rate is expressed as λ max / λ +min − 1 λ max / λ +min +1 for λ +min , respectively λ max , the smallest, resp.

Original: largest, non-zero eigenvalue of Φ ⊤ Φ (Nesterov, 2003).
Revised: largest, non-zero eigenvalue of Φ ⊤ Φ (Nesterov, 2003).

Original: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we will consider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.
Revised: When Φ has i.i.d entries with zero mean, unit variance and bounded 4th moment, as we will consider in the rest of this paper, 1 n Φ ⊤ Φ follows the Wishart distribution.

Original: In particular, we have analytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).
Revised: In particular, we have analytical expressions for the asymptotic minimum and maximum eigenvalue (section 2.2).

Original: When lim p,n →∞ p / n = ρ , we have λ max / λ +min
Revised: When lim p,n →∞ p / n = ρ , we have λ max / λ +min

Original: →
Revised: →

Original: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .
Revised: (cid:0) 1+ √ ρ 1 −√ ρ (cid:1) 2 ≥ 1 .

Original: This function has asymptotes at 1 , for p
Revised: This function has asymptotes at 1 , for p

Original: =
Revised: =

Original: o ( n ) or n = o ( p ) and diverges when n = p .
Revised: o ( n ) or n = o ( p ) and diverges when n = p .

Original: This peak-shaped function can also be observed empirically in more complex models such as kernel methods (Poggio et al., 2019) and neural networks in the limit of infinite width as they are in the Neural Tangent Kernel regime (Lee et al., 2017; Jacot et al., 2018).
Revised: This peak-shaped function can also be observed empirically in more complex models such as kernel methods (Poggio et al., 2019) and neural networks in the limit of infinite width as they are in the Neural Tangent Kernel regime (Lee et al., 2017; Jacot et al., 2018).

Original: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would be replaced by Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: However, when using TD(0), presented in section 2.1, the feature covariance matrix Φ ⊤ Φ would be replaced by Φ ⊤ Ξ( I n − γ P π )Φ .

Original: While Xiao et al. (2021) derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.
Revised: While Xiao et al. (2021) derived worst-case bounds for overparameterized TD, their dependency of the overparamerization ratio ρ = p / n is not obvious.

Original: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.
Revised: In this work we chose to use tools for random matrix theory for modeling the behavior of TD.

Original: While we can model Φ as an i.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allow us to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ .
Revised: While we can model Φ as an i.i.d matrix, it is not obvious how to model Ξ and P π in a realistic manner that would still allow us to characterize the spectrum of the TD operator Φ ⊤ Ξ( I n − γ P π )Φ .

Original: For general Ξ and P π , no closed form solution for the spectrum exists.
Revised: For general Ξ and P π , no closed form solution for the spectrum exists.

Original: We believe this is one of the key reasons why overparameterization in Temporal Difference learning has not been studied extensively.
Revised: We believe this is one of the key reasons why overparameterization in Temporal Difference learning has not been studied extensively.

Original: Thus, finding good typical models for P π is of utmost importance, and the next section will introduce a class of models for which the spectrum of TD can be computed.
Revised: Thus, finding good typical models for P π is of utmost importance, and the next section will introduce a class of models for which the spectrum of TD can be computed.

Original: 3.2 P π as the Markov transition matrix of a random graph
Revised: 3.2 P π as the Markov transition matrix of a random graph

Original: In RL, P π is the Markov transition matrix of a random walk when sampling from π .
Revised: In RL, P π is the Markov transition matrix of a random walk when sampling from π .

Original: We will assume in the rest of the paper that the Markov chain is irreducible and aperiodic so that P π has a unique stationary distribution denoted d π .
Revised: We will assume in the rest of the paper that the Markov chain is irreducible and aperiodic so that P π has a unique stationary distribution denoted d π .

Original: As we wish to understand the expected behavior of the TD algorithm, we propose in this subsection a simple yet expressive model of P π that will allow us to do so.
Revised: As we wish to understand the expected behavior of the TD algorithm, we propose in this subsection a simple yet expressive model of P π that will allow us to do so.

Original: As random graphs are used to model properties of typical graphs, a candidate for a model of P π would be to take the Markov transition matrix of a random graph.
Revised: As random graphs are used to model properties of typical graphs, a candidate for a model of P π would be to take the Markov transition matrix of a random graph.

Original: However, random graphs, at least the Erd˝os-R´enyi(-Gilbert) type ones, are not able natively to take into account the impact of π , most importantly the fact that some state-actions are visited more often than others under d π , the stationary distribution of π .
Revised: However, random graphs, at least the Erd˝os-R´enyi(-Gilbert) type ones, are not able natively to take into account the impact of π , most importantly the fact that some state-actions are visited more often than others under d π , the stationary distribution of π .

Original: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markov transition matrix of a G ( n, µ ) graph deformed by d π .
Revised: In this subsection, we propose a simple model for P π , denoted by ˆ P π , which is the Markov transition matrix of a G ( n, µ ) graph deformed by d π .

Original: Let us start with an undirected graph of the type G ( n, µ ) .
Revised: Let us start with an undirected graph of the type G ( n, µ ) .

Original: To take into account the state-action visitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .
Revised: To take into account the state-action visitation d π , we construct the diagonal matrix D π so that D π = diag ( d π ) .

Original: Now, we define the deformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.
Revised: Now, we define the deformed adjacency matrix A π as A π ≜ n AD π where A is the original adjacency matrix.

Original: When d π is uniform, we revert back to the original case A π = A .
Revised: When d π is uniform, we revert back to the original case A π = A .

Original: This construction keeps the same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.
Revised: This construction keeps the same edges as in the original graph but re-weights them by how likely d π would visit that stateaction.

Original: We can informally write the Markov transition matrix associated with this deformed graph as 4 ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1 as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n W is a Wigner-type matrix with zero mean and unit variance.
Revised: We can informally write the Markov transition matrix associated with this deformed graph as 4 ˆ P π = 1 n d ⊤ π + X π where (i) 1 n d ⊤ π is a deterministic rank one matrix associated to the eigenvalue 1 as 1 ⊤ n d π = 1 and (ii) X π = (cid:113) n (1 − µ ) µ 1 √ n WD π is a stochastic matrix of null expectation as 1 √ n W is a Wigner-type matrix with zero mean and unit variance.

Original: It can be tempting to understand under which conditions the stochastic part becomes negligible and what it entails.
Revised: It can be tempting to understand under which conditions the stochastic part becomes negligible and what it entails.

Original: As we will see shortly, this setting will reveal itself to be of importance when studying the spectrum of TD.
Revised: As we will see shortly, this setting will reveal itself to be of importance when studying the spectrum of TD.

Original: Definition 1 (Asymptotically well-connected graphs) .
Revised: Definition 1 (Asymptotically well-connected graphs) .

Original: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its
Revised: We say a graph G is asymptotically wellconnected if | λ 2 | ∈ o (1) for 1 = λ 1 ≥ | λ 2 | ≥ · · · ≥ | λ n | the (modulus) ordered eigenvalues of its

Original: Markov transition matrix.
Revised: Markov transition matrix.

Original: The choice for the name of Definition 1 comes from the fact that the second smallest eigenvalue of the Laplacian is related to the presence of bottlenecks in the graph through Cheeger’s inequality (Cheeger, 1970).
Revised: The choice for the name of Definition 1 comes from the fact that the second smallest eigenvalue of the Laplacian is related to the presence of bottlenecks in the graph through Cheeger’s inequality (Cheeger, 1970).

Original: Example.
Revised: Example.

Original: Graphs of the type G ( n, µ ) (resp.
Revised: Graphs of the type G ( n, µ ) (resp.

Original: G ( n, d ) ) are asymptotically well-connected if their expected degree d = nµ (resp.
Revised: G ( n, d ) ) are asymptotically well-connected if their expected degree d = nµ (resp.

Original: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) (Zhao, 2012; Tran et al., 2010).
Revised: d ) grows to + ∞ with n , i.e 1 ∈ o ( d ) (Zhao, 2012; Tran et al., 2010).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π , then the graph is asymptotically well-connected if n ∥
Revised: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π , then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .
Revised: d π ∥ ∞ ∈ o ( √ d ) .

Original: This again generalizes the uniform case as, when d π is uniform, the condition becomes equivalent to 1 ∈ o
Revised: This again generalizes the uniform case as, when d π is uniform, the condition becomes equivalent to 1 ∈ o

Original: ( d ) as in the example.
Revised: ( d ) as in the example.

Original: In the non-uniform case, this condition entails that either d π spreads efficiently across states, or the degree d rises fast enough that those states are still visited sufficiently often.
Revised: In the non-uniform case, this condition entails that either d π spreads efficiently across states, or the degree d rises fast enough that those states are still visited sufficiently often.

Original: Note that the asymptotically well-connected condition is also satisfied if we use longer back-ups of length k such that k grows with n to + ∞ , i.e 1 ∈
Revised: Note that the asymptotically well-connected condition is also satisfied if we use longer back-ups of length k such that k grows with n to + ∞ , i.e 1 ∈

Original: o ( k ) for the TD update.
Revised: o ( k ) for the TD update.

Original: See
Revised: See

Original: Lemma B.2 for details.
Revised: Lemma B.2 for details.

Original: In those cases, we have
Revised: In those cases, we have

Original: [Equation]
Revised: [Equation]

Original: This approximation is particularly interesting as it asymptotically conserves the left and right eigenvectors, d π and 1 n , of the original P π associated to the eigenvalue 1 .
Revised: This approximation is particularly interesting as it asymptotically conserves the left and right eigenvectors, d π and 1 n , of the original P π associated to the eigenvalue 1 .

Original: Furthermore, if all stateactions are reachable, the sequence of matrix power iterates of P π , (cid:0) P π (cid:1) k converges to 1 n d ⊤ π .
Revised: Furthermore, if all stateactions are reachable, the sequence of matrix power iterates of P π , (cid:0) P π (cid:1) k converges to 1 n d ⊤ π .

Original: While we are not the first to use tools from graph theory in the context of RL, related works we are aware of (Mahadevan & Maggioni, 2007; Machado et al., 2017, 2018; Wu et al., 2018) take a different approach and leverage eigenvectors of the Markov transition matrix (or Laplacian) to construct better representations for the value function or use them as options.
Revised: While we are not the first to use tools from graph theory in the context of RL, related works we are aware of (Mahadevan & Maggioni, 2007; Machado et al., 2017, 2018; Wu et al., 2018) take a different approach and leverage eigenvectors of the Markov transition matrix (or Laplacian) to construct better representations for the value function or use them as options.

Original: In contrast, we use spectral properties of P π to study the convergence of Temporal Difference learning.
Revised: In contrast, we use spectral properties of P π to study the convergence of Temporal Difference learning.

Original: 3.3 Spiked Wishart model for TD with uniform sampling
Revised: 3.3 Spiked Wishart model for TD with uniform sampling

Original: Now that we have a simple random graph model for P π , we focus on estimating the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ .
Revised: Now that we have a simple random graph model for P π , we focus on estimating the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ .

Original: For simplicity, we assume the off-policy sampling is done uniformly, i.e Ξ = 1 n I n .
Revised: For simplicity, we assume the off-policy sampling is done uniformly, i.e Ξ = 1 n I n .

Original: While this assumption is not realistic in most RL settings, it is still of interest in the context of dynamic programming where we have access to all the states.
Revised: While this assumption is not realistic in most RL settings, it is still of interest in the context of dynamic programming where we have access to all the states.

Original: Proposition 3.2 (Spiked MP) .
Revised: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the
Revised: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .
Revised: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition (Baik et al., 2005) for the minimum non-zero eigenvalue λ +min which separates from bulk and converges to
Revised: When ργ 2 ≥ 1 , there is a phase transition (Baik et al., 2005) for the minimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]
Revised: [Equation]

Original: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =
Revised: Note that the phase transition is smooth as for ρ = 1 / γ 2 , λ spiked =

Original: ( (cid:112) 1 / γ 2 − 1) 2 .
Revised: ( (cid:112) 1 / γ 2 − 1) 2 .

Original: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .
Revised: Vast overparameterization and tabular case: An interesting regime is when ρ ≫ 1 γ 2 > 1 .

Original: In that case, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , the effective horizon of the problem, as lim
Revised: In that case, the maximum eigenvalue of the MP law and the spiked one differ by a factor H = 11 − γ , the effective horizon of the problem, as lim

Original: ρ →∞ λ max / λ +min
Revised: ρ →∞ λ max / λ +min

Original: =
Revised: =

Original: 11 − γ .
Revised: 11 − γ .

Original: In the tabular regime, Φ = I n .
Revised: In the tabular regime, Φ = I n .

Original: As
Revised: As

Original: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .
Revised: P π has eigenvalues in [ − 1 , 1] , the eigenvalues of I n − γ P π are in [1 − γ, 1 + γ ] .

Original: This would give a worst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterized case above.
Revised: This would give a worst case ratio of 1+ γ 1 − γ which, up to a factor 1 + γ ≤ 2 , is the same as the vastly overparameterized case above.

Original: As these eigenvalues govern the optimization speed of TD, the vastly overparameterized case and the tabular can be thought of as comparable.
Revised: As these eigenvalues govern the optimization speed of TD, the vastly overparameterized case and the tabular can be thought of as comparable.

Original: 3.4 Non-uniform off-policy sampling
Revised: 3.4 Non-uniform off-policy sampling

Original: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assuming Ξ = 1 n I n .
Revised: In the previous subsection we were able to characterize the spectrum of Φ ⊤ Ξ( I n − γ ˆ P π )Φ assuming Ξ = 1 n I n .

Original: In this subsection we analyze the more general case where Ξ is not uniform.
Revised: In this subsection we analyze the more general case where Ξ is not uniform.

Original: However by doing so, we lose guarantees and must resort to approximations.
Revised: However by doing so, we lose guarantees and must resort to approximations.

Original: First, we analyze the eigenvalues of Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .
Revised: First, we analyze the eigenvalues of Ξ( I n − γ ˆ P π ) , or rather, in the limit of large n , the eigenvalues of Ξ( I n − γ 1 n d ⊤ π ) .

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −
Revised: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy a secular equation (Golub, 1973)
Revised: γ 1 n d ⊤ π ) satisfy a secular equation (Golub, 1973)

Original: [Equation]
Revised: [Equation]

Original: This expression is another form of the characteristic polynomial of a matrix whose roots are its eigenvalues.
Revised: This expression is another form of the characteristic polynomial of a matrix whose roots are its eigenvalues.

Original: However, this form enables us to derive approximations more easily.
Revised: However, this form enables us to derive approximations more easily.

Original: In particular, for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸
Revised: In particular, for the least visited state-action m = arg min i =1 ,...,n ξ i , if ξ m ≪ ξ i , i ̸

Original: = m , we have for λ ≈ ξ m that
Revised: = m , we have for λ ≈ ξ m that

Original: [Equation]
Revised: [Equation]

Original: ̸
Revised: ̸

Original: [Equation]
Revised: [Equation]

Original: m .
Revised: m .

Original: From this assumption, we derive an approximation for the minimum eigenvalue of
Revised: From this assumption, we derive an approximation for the minimum eigenvalue of

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution and the discount factor.
Revised: This eigenvalue is inferior to ξ m and reflects the interaction between the off-policy distribution and the discount factor.

Original: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which is the product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .
Revised: At worst, if d π mostly visits state m , it will be close to ξ m (1 − γ ) which is the product of the minimum eigenvalues of Ξ and I n − γ ˆ P π .

Original: A similar expression can be derived for the maximum eigenvalue under similar assumptions.
Revised: A similar expression can be derived for the maximum eigenvalue under similar assumptions.

Original: Now, again reasoning approximately, if that eigenvalue was very small compared to the other ones who would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similar spiking behavior as before.
Revised: Now, again reasoning approximately, if that eigenvalue was very small compared to the other ones who would form a bulk of similar values, we could expect Φ ⊤ Ξ( I n − γ ˆ P π )Φ to have a similar spiking behavior as before.

Original: However, in that case, the spiking eigenvalue would be
Revised: However, in that case, the spiking eigenvalue would be

Original: [Equation]
Revised: [Equation]

Original: While less simple than the estimators developed in the last subsection, we will see in Section 5 that it still accurately predicts the smallest non-zero eigenvalue.
Revised: While less simple than the estimators developed in the last subsection, we will see in Section 5 that it still accurately predicts the smallest non-zero eigenvalue.

Original:  Optimizing with Temporal Difference learning 4.1 Decomposition of the error
Revised:  Optimizing with Temporal Difference learning 4.1 Decomposition of the error

Original: Using results about the spectrum of the TD operator shown in the last section, we analyze the behavior of the Mean Square Bellman Error L MSBE ( q t ) =
Revised: Using results about the spectrum of the TD operator shown in the last section, we analyze the behavior of the Mean Square Bellman Error L MSBE ( q t ) =

Original: 12 ∥
Revised: 12 ∥

Original: r + γ P π q t − q t ∥
Revised: r + γ P π q t − q t ∥

Original: 2Ξ during the optimization.
Revised: 2Ξ during the optimization.

Original: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n
Revised: In particular, assuming Φ ⊤ Ξ( I n − γ P π )Φ is non-singular for p < n

Original: [Equation]
Revised: [Equation]

Original: the q -value asymptotically reached by TD(0), we can decompose this error into two components.
Revised: the q -value asymptotically reached by TD(0), we can decompose this error into two components.

Original: Lemma 4.1 (Decomposition of the error) .
Revised: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.
Revised: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ), we have the following decomposition
Revised: (2) ), we have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .
Revised: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Following Bottou & Bousquet (2007), we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥
Revised: Following Bottou & Bousquet (2007), we refer to ∥ q t − q TD ∥ 2 as the approximation error and ∥

Original: r + γ P π q TD − q TD ∥
Revised: r + γ P π q TD − q TD ∥

Original: Ξ as the estimation error.
Revised: Ξ as the estimation error.

Original: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π ) in metric Ξ .
Revised: The factor (cid:112) n ∥ Ξ ∥ ∞ ≥ 1 comes from bounding the spectral norm of ( I n − γ P π ) in metric Ξ .

Original: Beyond this factor, we are not exactly in the same setting as (Bottou & Bousquet, 2007) as the asymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, as known in the RL community (Scherrer, 2010).
Revised: Beyond this factor, we are not exactly in the same setting as (Bottou & Bousquet, 2007) as the asymptotic TD(0) value estimate q TD may not be the one reaching the lowest estimation error, as known in the RL community (Scherrer, 2010).

Original: Indeed,
Revised: Indeed,

Original: TD(0) finds the minimum of the Mean Square Projected Bellman Error (MSPBE) but not of the
Revised: TD(0) finds the minimum of the Mean Square Projected Bellman Error (MSPBE) but not of the

Original: MSBE (Sutton & Barto, 2018).
Revised: MSBE (Sutton & Barto, 2018).

Original: However in practice it is common to minimize the MSBE as it does not require an expensive projection step.
Revised: However in practice it is common to minimize the MSBE as it does not require an expensive projection step.

Original: 4.2 The ordinary behavior of the approximation error
Revised: 4.2 The ordinary behavior of the approximation error

Original: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .
Revised: In this subsection, we take a look at the approximation error, i.e how efficiently we converge to q TD .

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and that its spectrum is real and positive.
Revised: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and that its spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largest eigenvalues, for η =
Revised: Denoting by λ +min and λ max its smallest non-zero and largest eigenvalues, for η =

Original: [Equation]
Revised: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.
Revised: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).
Revised: The assumptions above can seem strong however they are verified under our model (cf Section 3.3).

Original: This is a slightly more general version of the linear convergence of gradient descent, however, as the
Revised: This is a slightly more general version of the linear convergence of gradient descent, however, as the

Original: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .
Revised: TD operator may not be diagonalizable in an orthonormal basis, we incur an additional factor K 2 .

Original: 4.3 The singular behavior of the estimation error
Revised: 4.3 The singular behavior of the estimation error

Original: Now we turn our attention to the second term; the estimation error which is the residual error after
Revised: Now we turn our attention to the second term; the estimation error which is the residual error after

Original: T → + ∞ steps.
Revised: T → + ∞ steps.

Original: In supervised learning, we expect this error to be the usual bias of our algorithm and thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possible to interpolate the data.
Revised: In supervised learning, we expect this error to be the usual bias of our algorithm and thus to decrease as p / n increases until p = n for which the bias would reach 0 as it is possible to interpolate the data.

Original: However, in reinforcement learning, the behavior of this term can be quite different, in particular it can be non-monotonous.
Revised: However, in reinforcement learning, the behavior of this term can be quite different, in particular it can be non-monotonous.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically rank one
Revised: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically rank one

Original: [Equation]
Revised: [Equation]

Original: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =
Revised: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =

Original: I n − Π is its complement, and ˜ ρ =
Revised: I n − Π is its complement, and ˜ ρ =

Original: d ⊤ π Π 1 n .
Revised: d ⊤ π Π 1 n .

Original: When γ = 0 , i.e we just perform reward regression, this is equal to ∥
Revised: When γ = 0 , i.e we just perform reward regression, this is equal to ∥

Original: Π ⊥ r ∥ 2Ξ which is exactly the residual error in r that our representation cannot capture.
Revised: Π ⊥ r ∥ 2Ξ which is exactly the residual error in r that our representation cannot capture.

Original: When ρ > 1 , Π ⊥ = 

Revised: When ρ > 1 , Π ⊥ =

Original: so the limit is zero as we expect from a model able to interpolate the data.
Revised: so the limit is zero as we expect from a model able to interpolate the data.

Original: However, there is a behavior here not present in supervised learning.
Revised: However, there is a behavior here not present in supervised learning.

Original: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p close to but less than n , a peaking behavior of the estimation error can happen.
Revised: When ˜ ρ ≈ 1 , which we can expect when Π is closer to identity, i.e p close to but less than n , a peaking behavior of the estimation error can happen.

Original: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased
Revised: Replacing Π by its expectation 5 ρ I n above can lead to an equation simpler to analyze, albeit biased

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This phenomenon might appear counter-intuitive as we would expected our error to decrease with p .
Revised: This phenomenon might appear counter-intuitive as we would expected our error to decrease with p .

Original: We provide a few insights to make sense of this.
Revised: We provide a few insights to make sense of this.

Original: First, when γ increases, so does the scale of the
Revised: First, when γ increases, so does the scale of the

Original: Q-function, so we can expect our errors to get bigger.
Revised: Q-function, so we can expect our errors to get bigger.

Original: Second, using random features Φ informally has a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from 11 − γ to 1 1 − γρ .
Revised: Second, using random features Φ informally has a smoothing effect, stronger when ρ is small, which causes the reduction in magnitude from 11 − γ to 1 1 − γρ .

Original: This can be observed in the proof in appendix.
Revised: This can be observed in the proof in appendix.

Original: Finally, peaking behaviors of TD such as this one have been observed in the community (Scherrer, 2010) but not theoretically explained in a general setting as far as we know.
Revised: Finally, peaking behaviors of TD such as this one have been observed in the community (Scherrer, 2010) but not theoretically explained in a general setting as far as we know.

Original: 4.4 Beyond stationary sampling
Revised: 4.4 Beyond stationary sampling

Original: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξ was non-singular, i.e that our off-policy sampling visited all the state-actions .
Revised: Until now, we always mentioned n as the number of state-action pairs with the assumption that Ξ was non-singular, i.e that our off-policy sampling visited all the state-actions .

Original: Here, we take a closer look at what happens when this is not the case.
Revised: Here, we take a closer look at what happens when this is not the case.

Original: In particular, we study a simple toy model where we assume Ξ only visits n vis ( t ) states at time t with uniform probability.
Revised: In particular, we study a simple toy model where we assume Ξ only visits n vis ( t ) states at time t with uniform probability.

Original: This simply changes all the n by n vis ( t ) in the previous results.
Revised: This simply changes all the n by n vis ( t ) in the previous results.

Original: As often observed during training, at t = 0 , π might start as a more exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution ( n vis ( t ) low).
Revised: As often observed during training, at t = 0 , π might start as a more exploratory policy ( n vis ( t = 0) high) and over time converges on a more deterministic solution ( n vis ( t ) low).

Original: This can be problematic when performing TD learning with a fixed stepsize η .
Revised: This can be problematic when performing TD learning with a fixed stepsize η .

Original: In that case, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 → 2 / λ max or equivalently
Revised: In that case, under the spiked model developed in Section 3.3 we need to ensure η ≤ 2 / (cid:0) 1+ √ p/n vis ( t ) (cid:1) 2 → 2 / λ max or equivalently

Original: [Equation]
Revised: [Equation]

Original: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .
Revised: As n vis ( t ) can typically decrease, this condition can be broken and thus TD learning will diverge .

Original: We conjecture this phenomenon may be related to the popularity of adaptive step size methods in
Revised: We conjecture this phenomenon may be related to the popularity of adaptive step size methods in

Original: RL over fixed step size ones.
Revised: RL over fixed step size ones.

Original:  Experiments
Revised:  Experiments

Original: In all the experiments we use Φ i.i.d with normal entries.
Revised: In all the experiments we use Φ i.i.d with normal entries.

Original: Per the universality property, our theoretical results hold for all distributions satisfying the hypotheses in Section 2.2.
Revised: Per the universality property, our theoretical results hold for all distributions satisfying the hypotheses in Section 2.2.

Original: Study of the spectral distribution:
Revised: Study of the spectral distribution:

Original: On Figure 1, we validate the theory and approximations of
Revised: On Figure 1, we validate the theory and approximations of

Original: Section 3.
Revised: Section 3.

Original: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has an eigenvalue at 1 and the rest of order 1 / √ n (Bordenave et al., 2012).
Revised: For an asymmetric random graph n = 4000 , µ = 1 / 2 , the spectrum of P (a) has an eigenvalue at 1 and the rest of order 1 / √ n (Bordenave et al., 2012).

Original: As predicted, the spectrum of the TD operator matches closely a spiked MP distribution (b).
Revised: As predicted, the spectrum of the TD operator matches closely a spiked MP distribution (b).

Original: When the graph does not satisfy the well-connected property, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distribution is more complex (d).
Revised: When the graph does not satisfy the well-connected property, for instance on a 4 rooms domain ( n = 4096 , π stochastic 6 ) (c) the spectral distribution is more complex (d).

Original: The relative error on λ +min is respectively 0 .
Revised: The relative error on λ +min is respectively 0 .

Original: 3% and 2% .
Revised: 3% and 2% .

Original: Despite this, we show that the empirical and theoretical (eq.
Revised: Despite this, we show that the empirical and theoretical (eq.

Original: values for λ +min and λ max on therooms domain ( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so that max
Revised: values for λ +min and λ max on therooms domain ( n = 256 ) closely match, both for Ξ uniform or highly ill-conditioned with evenly spaced ξ i so that max

Original: i ξ i min i ξ i = 2000 .
Revised: i ξ i min i ξ i = 2000 .

Original: While (b) and (d) show how violating the asymptotically wellconnected property breaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, which are the ones that matter for our optimization results, are still well predicted, even in more complex settings.
Revised: While (b) and (d) show how violating the asymptotically wellconnected property breaks our spiked MP distribution result, we show on (e) and (f) that the extreme eigenvalues, which are the ones that matter for our optimization results, are still well predicted, even in more complex settings.

Original: [Figure]
Revised: [Figure]

Original: I m ag i na r y pa r t
Revised: I m ag i na r y pa r t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.
Revised: Optimization behavior of TD: On Figure 2 we discuss the results of Section 4.

Original: We use again a 4 rooms domain with a fixed stochastic policy and study the Bellman error during training.
Revised: We use again a 4 rooms domain with a fixed stochastic policy and study the Bellman error during training.

Original: On all subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curves for low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than for ρ
Revised: On all subfigures of Figure 2, we observe that for ρ ≫ 1 or ρ ≪ 1 , the optimization is faster as the curves for low T are much closer to the asymptotic solution T = + ∞ (computed analytically) than for ρ

Original: ≈ 1 .
Revised: ≈ 1 .

Original: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalues which governs the optimization speed is λ max / λ +min → 1 for ρ ≪ 1 and to 11 − γ for ρ ≫ 1 while it diverges for ρ → 1 .
Revised: This is in line with the result of section 4.2 for a spiked model: the ratio of eigenvalues which governs the optimization speed is λ max / λ +min → 1 for ρ ≪ 1 and to 11 − γ for ρ ≫ 1 while it diverges for ρ → 1 .

Original: This leads to a classical double descent phenomenon on (a)
Revised: This leads to a classical double descent phenomenon on (a)

Original: [Figure]
Revised: [Figure]

Original: and (b) purely due to optimization as predicted by Kuzborskij et al.
Revised: and (b) purely due to optimization as predicted by Kuzborskij et al.

Original: On (a) and (b) because either γ or d ⊤ π r is small, the terms of order 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.
Revised: On (a) and (b) because either γ or d ⊤ π r is small, the terms of order 1 1 − γ ˜ ρ are negligible and thus it behaves similarly to supervised learning.

Original: On (c), the term
Revised: On (c), the term

Original: [Equation]
Revised: [Equation]

Original: ρ dominates
Revised: ρ dominates

Original: and we observe the peaking behavior of the estimation error (Section 4.3).
Revised: and we observe the peaking behavior of the estimation error (Section 4.3).

Original: Furthermore on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match the observed peaking behavior.
Revised: Furthermore on Figure 4 in appendix we study how our model and its approximation of Section 3.3 match the observed peaking behavior.

Original: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD ( p = 200 ).
Revised: Stability and non-stationarity: On Figure 3 we highlight how Ξ can affect the convergence of TD ( p = 200 ).

Original: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .
Revised: For this, we use a small 4 rooms ( n = 144 ) environment 7 and compute an optimal deterministic policy π ∗ .

Original: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze which combination of step size η and ϵ would lead to convergence.
Revised: Then, by varying the ϵ of a ϵ -greedy policy based on π ∗ , we can analyze which combination of step size η and ϵ would lead to convergence.

Original: On (a), we show the discounted state distribution for various ϵ where larger ϵ lead to more exploration.
Revised: On (a), we show the discounted state distribution for various ϵ where larger ϵ lead to more exploration.

Original: While our theoretical argument in Section 4.4 was about the number of visited states, we take here a more general and realistic approach where the state visitation is controlled by the greediness of our policy.
Revised: While our theoretical argument in Section 4.4 was about the number of visited states, we take here a more general and realistic approach where the state visitation is controlled by the greediness of our policy.

Original: Typically, during the optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.
Revised: Typically, during the optimization of a value based method, ϵ is annealed to 0 and the policy becomes more deterministic.

Original: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured by computing the spectral radius of our iteration matrix.
Revised: On (b) we show the probability of diverging (averaged over 100 samples of Φ ) measured by computing the spectral radius of our iteration matrix.

Original: As predicted, for a fixed η , when ϵ decreases, we might go through a phase transition where the TD iteration, stable at the beginning, becomes unstable.
Revised: As predicted, for a fixed η , when ϵ decreases, we might go through a phase transition where the TD iteration, stable at the beginning, becomes unstable.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we have analyzed the spectral distribution of the Temporal Difference operator when using random features and modeling the Markov transition matrix as the one of a random graph.
Revised: In this work, we have analyzed the spectral distribution of the Temporal Difference operator when using random features and modeling the Markov transition matrix as the one of a random graph.

Original: This characterization allowed us to make predictive theories for the optimization error and stability of TD.
Revised: This characterization allowed us to make predictive theories for the optimization error and stability of TD.

Original: Notably we are able to predict how fast we converge and whether the solution found is accurate as a function of the ratio between the number of parameters and the number of state-action pairs.
Revised: Notably we are able to predict how fast we converge and whether the solution found is accurate as a function of the ratio between the number of parameters and the number of state-action pairs.

Original: In particular, we highlighted and theoretically explained several phenomena specific to reinforcement learning such as the peaking behavior of the estimation error of TD or how changing the policy might cause divergence when using a fixed step size, even in the on-policy setting.
Revised: In particular, we highlighted and theoretically explained several phenomena specific to reinforcement learning such as the peaking behavior of the estimation error of TD or how changing the policy might cause divergence when using a fixed step size, even in the on-policy setting.

Original: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations of the extreme eigenvalues of the MP law (Baik et al., 2005) (iii) extend our results to neural networks in the Neural Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms.
Revised: Important directions for future work include: (i) improving the model for P π through careful analysis of more expressive random graphs, (ii) derive non-asymptotic results using the fluctuations of the extreme eigenvalues of the MP law (Baik et al., 2005) (iii) extend our results to neural networks in the Neural Tangent Kernel regime and finally (iv) study the behavior of policy optimization algorithms.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank Alexandre Pich´e for useful comments on an earlier version of this paper.
Revised: We thank Alexandre Pich´e for useful comments on an earlier version of this paper.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: This work is a theoretical model for TD(0) in the linear setting.
Revised: This work is a theoretical model for TD(0) in the linear setting.

Original: As such we do not think there are any applications with a societal impact to be foreseen.
Revised: As such we do not think there are any applications with a societal impact to be foreseen.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Some finer details can be found in the proofs in appendices.
Revised: [Yes] Some finer details can be found in the proofs in appendices.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Statements such as Lemmas or propositions have a full proof in appendix.
Revised: [Yes] Statements such as Lemmas or propositions have a full proof in appendix.

Original: Informal statements and approximations have a derivation in appendix where we make clear what the approximation/asssumption is.
Revised: Informal statements and approximations have a derivation in appendix where we make clear what the approximation/asssumption is.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] Colab notebooks are supplied to reproduce the figures.
Revised: [Yes] Colab notebooks are supplied to reproduce the figures.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: Used public Colab notebook compute.
Revised: Used public Colab notebook compute.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Appendix: Experiments
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [Figure]

Original: On Figure 4 we test the models developed in Section 4.3.

Original: Note that the models were developed for a ˆ

Original: P π approximately rank 1.

Original: We can see on (a) that the models, both the exact and the simplified overestimate the true estimation error in a 14 × 14 4 rooms domain ( n = 784) for various p = 1 . . .

Original: n .

Original: However, if we were to do something akin to a k-step return, where we consider one action to be a sequence of k, this would be equivalent to changing ˆ P π to (cid:0) ˆ P π (cid:1) k .

Original: In (b) and (c) we can see that taking a higher k improve our estimation.

Original: Taking powers of the transition matrix has for effect to push down all the eigenvalues < 1 and as such make the graph ”better” connected.

Original: All in all, while our formula is not precise in a realistic scenario, when considering an iterated setting, it recovers the estimation precisely.

Original: Note that for Figure 4 we chose not to average over random Φ as it highlights the exactness of the rank one model developed in Section 4.3.

Original: B Appendix: Proofs

Original: In the paper and in the proofs, we extend naturally the notation o ( · ) to matrices with the following definition

Original: [Equation]

Original: for ∥ · ∥ a natural operator norm.

Original: In this paper we only consider the 2-operator norm which corresponds to the largest singular value.

Original: B.1 Deformed random graph

Original: In Section 2.3 we derive informally an expression for the Markov transition matrix of a random graph taking into account d π .

Original: Let us go into more details here, and explain why we called it an informal derivation.

Original: Let us recall that A π = n AD π for A the original adjacency matrix and

Original: D π = diag ( d π ) the stationary distribution of π .

Original: To get ˆ P π we need to normalize the rows i = 1 , . . .

Original: n of

Original: A π by their sum, i.e the degree d i of each state.

Original: In the uniform case, A π = A , (Bordenave et al., 2012) provide first a sketch of proof (eq 1.6) based on the fact that d i = D ii

Original: = (cid:80)

Original: j

Original: A ij satisfies | d i − nµ | ∈

Original: o ( n ) by the law of large numbers.

Original: This allows to ”replace” D by nµ I n in the proofs which ties it back to the general Wigner/i.i.d case.

Original: In the deformed case, we cannot invoke directly the law of large numbers.

Original: Indeed, now d i = (cid:80)

Original: j A πij = n (cid:80)

Original: j A ij d π ( j ) so we have a weighted sum instead of a simple one.

Original: We need a concentration inequality for the weighted sum of random variables.

Original: One can be found in (Boucheron et al., 2013)

Original: Lemma B.1 (From (Boucheron et al., 2013)) .

Original: Let X 1 , X 2 , . . .

Original: , X n be iid copies of a real random variable X that obeys, for some p ≥

Original: [Equation]

Original: for all u > 0 .

Original: Then there exists L > 0 such that, for any s ∈ R n , and for q the conjugate of p ( 1 p + 1 q = 1 )

Original: [Equation]

Original: In our case, for X j = c · ( A ij − µ ) (where c is chosen so the the tail assumption is satisfied),

Original: [Equation]

Original: Thus, if ∥ d π ∥ qp and ∥

Original: d π ∥ 22 are ∈

Original: o (1) , and then reasoning on − X

Original: j we have that | d i n − µ | is, almost surely when n is large, smaller than any constant t > 0 , therefore o (1) .

Original: Thus we get the same argument D

Original: = nµ I n + o ( n ) .

Original: Note here that in high dimension the stronger condition for d π will be for p = ∞ , ∥ d π ∥ ∞ ∈ o (1) .

Original: As we already make a stronger assumption later, we have chosen not to include it in the main text.

Original: All of this allows to calculate ˆ P π by normalizing A π

Original: [Equation]

Original: When the entries of A are iid Bernoulli of mean µ , we have σ 2 = µ (1 − µ ) and W has iid whitened

Original: Bernoulli entries.

Original: We consider this proof informal as there are a certain number of technicalities involved when controlling the deviation of all these random quantities (as we have sketched above for the expected degree).

Original: B.2 Asymptotically well-connected graph

Original: Proposition 3.1.

Original: For the d π -deformed G ( n, µ ) graph studied above, assuming ˆ P π = 1 n d ⊤ π + X π , then the graph is asymptotically well-connected if n ∥

Original: d π ∥ ∞ ∈ o ( √ d ) .

Original: Proof.

Original: First, recall that

Original: [Equation]

Original: For ∥ · ∥ the 2-operator corresponding to the maximum singular value, we have:

Original: [Equation]

Original: From (Tao, 2012) we have for both Wigner (Corollary 2.3.6), when the graph is undirected and i.i.d matrices (Corollary 2.3.5) when the edges are iid and directed, that There exists A, C, c >

Original: [Equation]

Original: Which can be understood as 1 √ n W has bounded operator norm with overwhelming probability.

Original: In particular, taking the limit n → ∞ gives us that 1 √ n W has bounded norm almost surely when n → ∞ .

Original: Then ∥ D π ∥ =

Original: ∥ d π ∥ ∞ where ∥ · ∥ ∞ is the vector max-norm.

Original: This is due to the fact that D π is a diagonal matrix with positive entries, therefore its largest singular value is its largest eigenvalue.

Original: For ∥ X π ∥ to be o (1) we need

Original: [Equation]

Original: As 1 − µ ≤ 1 and for nµ = d , it is sufficient to have (almost surely)

Original: [Equation]

Original: Lemma B.2.

Original: From (Broder & Karlin, 1989), for a irreducible and aperiodic Markov transition matrix P π , we have that, for (cid:0) P πij (cid:1) k

Original: = d πj + O ( (cid:113)

Original: d πj d πi λ k 2 ) , thus

Original: [Equation]

Original: where κ ( · ) is the condition number.

Original: This lemma means that when using longer back-ups, as is for instance done when using n-step returns, the second eigenvalue of P k will shrink to 0 as long as 1 ∈ o ( k ) .

Original: B.3 Spiked Marchenko-Pastur distribution

Original: For, we prove small result showing that the o (1) appearing in ˆ P π remains a o (1) in 1 n Φ ⊤ ( I n − γ ˆ P π (cid:1) Φ .

Original: Lemma B.3.

Original: For ˆ P π satisfying the asymptotically well-connected condition, we have

Original: [Equation]

Original: Proof.

Original: For ∥ · ∥ = ∥ · ∥ 2 the 2-operator norm which corresponds to the largest singular value, we have

Original: [Equation]

Original: As the maximum eigenvalue of 1 n Φ ⊤ Φ converges a.s to (1+ √ ρ ) 2 per the Marchenko-Pastur theorem

Original: [Equation]

Original: This lemma is a useful result to understand that a o (1) perturbation stays a o (1) perturbation when adding features.

Original: Proposition B.1 (Asymmetric rank 1 perturbation) .

Original: We consider the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ for u , v ∈ R n so that v ⊤ u = 1 and γ ∈ [0 , 1[ which is a rank 1 perturbation of the

Original: Wishart covariance model.

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ converges to the

Original: Marchenko-Pastur law.

Original: When ργ 2 ≥ 1 , the empirical spectral distribution also converges to the

Original: Marchenko-Pastur law, but additionally, we observe a phase transition (Baik et al., 2005), and an eigenvalue λ spiked separates from the bulk

Original: [Equation]

Original: Proof.

Original: The proof is based on a simple adaptation of the proof in (Couillet & Liao, 2021)

Original: (page 117).

Original: As we consider an asymmetric spike, we lose commutative properties.

Original: Thus, we can only prove our result in the case where there is a unique spike instead of a finite number as is usual.

Original: We are looking to characterize the eigenvalues of 1 n Φ ⊤ ( I n − γ uv ⊤ )Φ , or equivalently (up to some zeros), the eigenvalues of 1 n ΦΦ ⊤ ( I n − γ uv ⊤ ) .

Original: However, in this case, 1 n ΦΦ ⊤ is not a sample from the Wishart distribution, but 1 p ΦΦ ⊤ is.

Original: We therefore analyze 1 p ΦΦ ⊤ and will simply have to scale our spectrum by ρ at the end.

Original: [Equation]

Original: Using Sherman-Morrison formula, we have ( I n − γ uv ⊤ ) − 1 = I n + γ 1 − γ uv ⊤ as v ⊤ u = 1 .

Original: Since det (cid:0) I n − γ uv ⊤ ) (cid:1) ̸ = 0 , the second determinant can be discarded.

Original: Then, we can factor the resolvent of the Wishart model.

Original: That is, letting Q (ˆ λ ) =

Original: (cid:16) 1 p ΦΦ ⊤ −

Original: ˆ λ I n (cid:17) − 1 , we write

Original: [Equation]

Original: Inverting 1 p ΦΦ ⊤ − ˆ λ

Original: I n is almost surely possible when ˆ λ < (1 − (cid:112) 1 / ρ ) 2 per the concentration of the empirical spectral distribution of Wishart covariance models to the Marchenko-Pastur law and the term Q − 1 (ˆ λ ) is almost surely non-zero.

Original: So the left term can be discarded.

Original: For the second we use Sylvester’s determinant identity to end up with a scalar equation

Original: [Equation]

Original: As Q (ˆ λ ) is a deterministic equivalent of m (ˆ λ )

Original: I n ((Marˇcenko & Pastur, 1967), Theorem 2.4 in (Couillet & Liao, 2021)), where m ( · ) is the Stieljes transform of the MP distribution, thus

Original: [Equation]

Original: The above determinant is zero when

Original: [Equation]

Original: From the definition of m ( · ) as the Stieljes transform of the MP distribution, we know (Couillet &

Original: Liao, 2021) it satisfies

Original: [Equation]

Original: for c = lim n, p → ∞ n / p = 1 / ρ

Original: Thus, for z = ˆ λ

Original: [Equation]

Original: Plugging the equation above, asymptotically

Original: [Equation]
Revised: [N/A]

Original: Now, we have to remember to multiply our spectrum by ρ to obtain the original one, and so we obtain

Original: [Equation]

Original: When (1 − γ )( ρ − 1 γ ) ≤ (1 − √ ρ ) 2 which is equivalent to ργ 2 ≥ 1 , this eigenvalue is isolated on the left of the bulk.

Original: Proposition 3.2 (Spiked MP) .

Original: If ργ 2 < 1 the spectrum of 1 n Φ ⊤ ( I n − γ ˆ P π )Φ converges to the

Original: Marchenko-Pastur law with parameter ρ .

Original: When ργ 2 ≥ 1 , there is a phase transition (Baik et al., 2005) for the minimum non-zero eigenvalue λ +min which separates from bulk and converges to

Original: [Equation]

Original: min

Original: [Equation]

Original: Proof.

Original: We essentially do the same proof as above, but with an additional o (1) moving around and we show that it does not impact the Stieljes transform, thus the spectrum, almost surely.

Original: [Equation]

Original: Where we used the definition of ˆ P π and Lemma B.3.

Original: Then

Original: [Equation]

Original: As the eigenvalues of I n − γ 1 n d ⊤ π are lower bounded by 1 − γ for n large enough det( I n − γ 1 n d ⊤ π + o (1)) must be non-zero.

Original: As ( I n + o (1)) − 1 =

Original: (cid:80) i

Original: ( o (1))

Original: i = I n + o (1) , using Sherman-Morrison formula, we have ( I n + o (1) − γ

Original: [Equation]

Original: ⊤ π ) − 1 = I n + o (1)+ o

Original: [Equation]

Original: 1) γ

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ⊤ π

Original: [Equation]

Original: 1) as d ⊤ π 1 n =

Original: With the same derivation and arguments as above

Original: [Equation]

Original: And finally we end up with the same equality as before where the o (1) from the perturbation and the one from the deterministic equivalent fuse together.

Original: As the determinant is continuous

Original: [Equation]

Original: Therefore the Stieljes transforms when using ˆ P π and 1 n d ⊤ π are asymptotically the same, thus, almost surely, they have the same empirical spectral distribution.

Original: B.4 Approximation of the extreme eigenvalues

Original: Lemma 3.1.

Original: If Ξ = diag ( ξ 1 , . . . , ξ n ) is non-singular, the eigenvalues of Ξ( I n −

Original: γ 1 n d ⊤ π ) satisfy a secular equation (Golub, 1973)

Original: [Equation]

Original: Proof.

Original: Let us consider v ̸ = 0 and λ , an eigenvector of Ξ( I n − γ 1 n d ⊤ π ) and its associated eigenvalue.

Original: [Equation]

Original: This give on component i

Original: [Equation]

Original: Then

Original: [Equation]

Original: Assuming λ ̸ = ξ i , ∀ i γ ξ

Original: i

Original: [Equation]

Original: d

Original: [Equation]

Original: [Equation]

Original: Multiplying by d π ( i ) and summing

Original: [Equation]

Original: Now the same term (cid:80) j d π ( j ) v j = d ⊤ π v appears on both sides.

Original: Let us verify if it can be null.

Original: If that were the case, we would have Ξ( I n − γ 1 n d ⊤ π ) v =

Original: Ξ v = λ v .

Original: Thus, as Ξ is diagonal, the possible vectors v possible are the canonical basis vectors and the possible eigenvalues are ξ i .

Original: However, as d π is a positive vector whose entries sum to 1, we cannot have d ⊤ π v = 0 .

Original: Thus d ⊤ π v ̸ = 0 and by dividing on each side:

Original: [Equation]

Original: B.5 Optimization and approximation error

Original: Lemma 4.1 (Decomposition of the error) .

Original: For q t = Φ θ t , where θ t is updated with TD(0) (eq.

Original: (2) ), we have the following decomposition

Original: [Equation]

Original: where ∥ x ∥ Ξ = √ x ⊤ Ξ x is the 2-norm in metric Ξ .

Original: Proof.

Original: [Equation]

Original: Where we used ∥ A ∥ 2 ≤ √ n ∥

Original: A ∥ ∞ and the fact that P π is a stochastic matrix, thus has entries between 0 and 1.

Original: As such, the entries of I n − γ P π are between − γ and 1 , in particular they are bounded by 1 .

Original: Proposition 4.1.

Original: Let us assume that Φ ⊤ Ξ( I n − γ P π )Φ is diagonalizable as Q Λ Q − 1 , and that its spectrum is real and positive.

Original: Denoting by λ +min and λ max its smallest non-zero and largest eigenvalues, for η =

Original: [Equation]

Original: where K = κ (Φ) κ ( Q ) when p < n and K = κ ( Q ) when p > n where κ is the condition number.

Original: Proof.

Original: Let us begin with the case p > n From the TD iteration Equation (2) multiplied on the left by Φ

Original: [Equation]

Original: For a i.i.d matrix Φ , we have that ΦΦ ⊤ is almost surely invertible.

Original: Thus the fixed point of this iteration is

Original: [Equation]

Original: Thus, for H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ and x t = q t − q π

Original: [Equation]

Original: As I n − η Λ is diagonal, and as such its largest singular value (2-norm) is its largest eigenvalue (in modulus).

Original: Now, we supposed the eigenvalues were real, max i,λ i =

Original: Λ ii {| 1 − ηλ i |} will be minimized when the largest and smallest terms are equal.

Original: [Equation]

Original: If λ +min = λ max this can be achieved for any η , however the only solution when they are different is to have the two expressions above of a different sign

Original: [Equation]

Original: Which leads to η =

Original: [Equation]

Original: Now for p < n we do a similar reasoning with θ .

Original: By the exact same reasoning, but using H TD =

Original: Φ ⊤ Ξ( I n − γ P π )Φ instead and the fixed point θ TD = (cid:0) Φ ⊤ Ξ( I n − γ P π )Φ (cid:1) 1 Φ ⊤ Ξ

Original: r

Original: we get

Original: [Equation]

Original: Now, we would like to obtain a result for the Q-function associated to these parameters.

Original: We have σ min (Φ)

Original: ∥ x ∥ 2 ≤ ∥ Φ x ∥ 2 ≤ σ max (Φ) ∥ x ∥ 2 for σ max and σ min the max and min singular values.

Original: Thus, multiplying each side by σ min (Φ) σ max (Φ) , we get

Original: [Equation]

Original: and

Original: [Equation]

Original: Squaring these results and putting it back in the contraction equation, we get for q TD =

Original: Φ θ TD

Original: [Equation]

Original: B.6 Peaking behavior of the estimation error

Original: Proposition 4.2.

Original: Let us assume ˆ P π satisfies the well-connected property, i.e that it is asymptotically rank one

Original: [Equation]

Original: where Π is the orthogonal projector onto the span of Φ in norm Ξ : Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ , Π ⊥ =

Original: I n − Π is its complement, and ˜ ρ =

Original: d ⊤ π Π 1 n .

Original: Proof.

Original: Let us look at the case p < n .

Original: We have q TD =

Original: Φ (cid:0) Φ ⊤ Ξ( I − γ P π )Φ (cid:1) − 1 Φ ⊤ Ξ r We will therefore study more closely the matrix Φ (cid:0) Φ ⊤ Ξ( I − γ P )Φ (cid:1) − 1 Φ ⊤ Ξ , in the case where P π = uv ⊤ is a rank one matrix.

Original: First, Sherman-Morrison formula gives us

Original: [Equation]

Original: which is the usual formula but for x ≜ − γ x .

Original: Note that this is licit as long as 1 −

Original: γ y ⊤ A − 1 x ̸ = 0 .

Original: Using Sherman-Morrison’s formula

Original: [Equation]

Original: For Π = Φ(Φ ⊤ ΞΦ) − 1 Φ ⊤ Ξ the usual projector unto the span of Φ in metric Ξ .

Original: Now, for the TD error we obtain

Original: [Equation]

Original: Let’s call I −

Original: Π

Original: =

Original: Π ⊥

Original: the complement projector projector and ˜ ρ = v ⊤ Π u .

Original: Now, for the MSBE error, we get

Original: [Equation]

Original: where we use the fact that Π ⊤ ΞΠ = ΞΠ = Π ⊤ Ξ and same for Π ⊥ .

Original: Using v = d π and u = 1 n

Original: C Colab links

Original: For plotting the spiked MP spectra https://colab.research.google.com/drive/ 1E3DgmZQF1-1FNYeVo4msyGUHwRiK32WE?usp

Original: =

Original: sharing

Original: For the extreme eigenvalues approximation https://colab.research.google.com/ drive/1H_GkjzlGyUxB0i55jM5VAqLXoEGGxN3h?usp=sharing

Original: For the peaking behavior https://colab.research.google.com/drive/ 1EVudyGfieXOUqDLifdhFrs8Pu1OzKOAW?usp=sharing

Original: For the peaking behavior model https://colab.research.google.com/drive/ 1pLIqaV5BO1ChIj9y9UmP5jmlWx26FNI7?usp=sharing

Original: For the unstability wrt η and ϵ https://colab.research.google.com/drive/ 1CuPZhv3eZoINXHCKLtwjVlHrLy9ylBuI?usp

Original: =sharing