Original: I MPLICIT U NDER -P ARAMETERIZATION I NHIBITS
Revised: I MPLICIT U NDER -P ARAMETERIZATION I NHIBITS

Original: D
Revised: D

Original: ATA
Revised: ATA

Original: -E FFICIENT D EEP R EINFORCEMENT
Revised: -E FFICIENT D EEP R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We identify a fundamental implicit under-parameterization phenomenon in valuebased deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network.
Revised: We identify a fundamental implicit under-parameterization phenomenon in valuebased deep RL methods that use bootstrapping: when value functions, approximated using deep neural networks, are trained with gradient descent using iterated regression onto target values generated by previous instances of the value network, more gradient updates decrease the expressivity of the current value network.

Original: We characterize this loss of expressivity via a rank collapse of the learned value network features, and show that it corresponds to a drop in performance.
Revised: We characterize this loss of expressivity via a rank collapse of the learned value network features, and show that it corresponds to a drop in performance.

Original: We demonstrate this phenomenon on popular domains including Atari and Gym benchmarks, and in both ofﬂine and online RL settings.
Revised: We demonstrate this phenomenon on popular domains including Atari and Gym benchmarks, and in both ofﬂine and online RL settings.

Original: We formally analyze this phenomenon and show that it results from a pathological interaction between bootstrapping and gradient-based optimization.
Revised: We formally analyze this phenomenon and show that it results from a pathological interaction between bootstrapping and gradient-based optimization.

Original: Finally, we show that mitigating implicit underparameterization by controlling rank collapse improves performance.
Revised: Finally, we show that mitigating implicit underparameterization by controlling rank collapse improves performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many pervasive deep reinforcement learning (RL) algorithms estimate value functions using bootstrapping, that is, by sequentially ﬁtting value functions to target value estimates generated from the value function learned in the previous iteration.
Revised: Many pervasive deep reinforcement learning (RL) algorithms estimate value functions using bootstrapping, that is, by sequentially ﬁtting value functions to target value estimates generated from the value function learned in the previous iteration.

Original: Despite high-proﬁle achievements (Silver et al., 2017), these algorithms are highly unreliable (Henderson et al., 2017) due to poorly understood optimization issues.
Revised: Despite high-proﬁle achievements (Silver et al., 2017), these algorithms are highly unreliable (Henderson et al., 2017) due to poorly understood optimization issues.

Original: Although a number of hypotheses have been proposed to explain these issues (Achiam et al., 2019; Bengio et al., 2020; Fu et al., 2019; Igl et al., 2020; Liu et al., 2018;
Revised: Although a number of hypotheses have been proposed to explain these issues (Achiam et al., 2019; Bengio et al., 2020; Fu et al., 2019; Igl et al., 2020; Liu et al., 2018;

Original: Kumar et al., 2020a), a complete understanding remains elusive.
Revised: Kumar et al., 2020a), a complete understanding remains elusive.

Original: We identify a fundamental “implicit under-parameterization” phenomenon that emerges when value networks are trained using gradient descent combined with bootstrapping.
Revised: We identify a fundamental “implicit under-parameterization” phenomenon that emerges when value networks are trained using gradient descent combined with bootstrapping.

Original: This phenomenon manifests as an excessive aliasing of features learned by the value network across states which is exacerbated with more gradient updates.
Revised: This phenomenon manifests as an excessive aliasing of features learned by the value network across states which is exacerbated with more gradient updates.

Original: While the supervised deep learning literature suggests that some feature aliasing is desirable for generalization ( e g. , Gunasekar et al., 2017; Arora et al., 2019), implicit under-parameterization exhibits more pronounced aliasing than in supervised learning.
Revised: While the supervised deep learning literature suggests that some feature aliasing is desirable for generalization ( e g. , Gunasekar et al., 2017; Arora et al., 2019), implicit under-parameterization exhibits more pronounced aliasing than in supervised learning.

Original: This over-aliasing causes an otherwise expressive value network to implicitly behave as an under-parameterized network, often resulting in poor performance.
Revised: This over-aliasing causes an otherwise expressive value network to implicitly behave as an under-parameterized network, often resulting in poor performance.

Original: Implicit under-parameterization becomes aggravated when the rate of data re-use is increased, restricting the sample efﬁciency of deep RL methods.
Revised: Implicit under-parameterization becomes aggravated when the rate of data re-use is increased, restricting the sample efﬁciency of deep RL methods.

Original: In online RL, increasing the number of gradient steps in between data collection steps for data-efﬁcient RL (Fu et al., 2019; Fedus et al., 2020b) causes the problem to emerge more frequently.
Revised: In online RL, increasing the number of gradient steps in between data collection steps for data-efﬁcient RL (Fu et al., 2019; Fedus et al., 2020b) causes the problem to emerge more frequently.

Original: In the extreme case when no additional data is collected, referred to as ofﬂine RL (Lange et al., 2012; Agarwal et al., 2020; Levine et al., 2020), implicit under-parameterization manifests consistently, limiting the viability of ofﬂine methods.
Revised: In the extreme case when no additional data is collected, referred to as ofﬂine RL (Lange et al., 2012; Agarwal et al., 2020; Levine et al., 2020), implicit under-parameterization manifests consistently, limiting the viability of ofﬂine methods.

Original: We demonstrate the existence of implicit under-parameterization in common value-based deep RL methods, including Q-learning (Mnih et al., 2015; Hessel et al., 2018) and actor-critic (Haarnoja et al., 2018), as well as neural ﬁtted-Q iteration (Riedmiller, 2005; Ernst et al., 2005).
Revised: We demonstrate the existence of implicit under-parameterization in common value-based deep RL methods, including Q-learning (Mnih et al., 2015; Hessel et al., 2018) and actor-critic (Haarnoja et al., 2018), as well as neural ﬁtted-Q iteration (Riedmiller, 2005; Ernst et al., 2005).

Original: To isolate the issue, we study the effective rank of the features in the penultimate layer of the value network (Section 3).
Revised: To isolate the issue, we study the effective rank of the features in the penultimate layer of the value network (Section 3).

Original: We observe that after an initial learning period, the rank of the learned features drops steeply.
Revised: We observe that after an initial learning period, the rank of the learned features drops steeply.

Original: As the rank decreases, the ability of the features to ﬁt subsequent target values and the optimal value function generally deteriorates and results in a sharp decrease in performance (Section 3.1).
Revised: As the rank decreases, the ability of the features to ﬁt subsequent target values and the optimal value function generally deteriorates and results in a sharp decrease in performance (Section 3.1).

Original: To better understand the emergence of implicit under-parameterization, we formally study the dynamics of Q-learning under two distinct models of neural net behavior (Section 4): kernel re- gression (Jacot et al., 2018; Mobahi et al., 2020) and deep linear networks (Arora et al., 2018).
Revised: To better understand the emergence of implicit under-parameterization, we formally study the dynamics of Q-learning under two distinct models of neural net behavior (Section 4): kernel re- gression (Jacot et al., 2018; Mobahi et al., 2020) and deep linear networks (Arora et al., 2018).

Original: We corroborate the existence of this phenomenon in both models, and show that implicit underparameterization stems from a pathological interaction between bootstrapping and the implicit regularization of gradient descent.
Revised: We corroborate the existence of this phenomenon in both models, and show that implicit underparameterization stems from a pathological interaction between bootstrapping and the implicit regularization of gradient descent.

Original: Since value networks are trained to regress towards targets generated by a previous version of the same model, this leads to a sequence of value networks of potentially decreasing expressivity, which can result in degenerate behavior and a drop in performance.
Revised: Since value networks are trained to regress towards targets generated by a previous version of the same model, this leads to a sequence of value networks of potentially decreasing expressivity, which can result in degenerate behavior and a drop in performance.

Original: The main contribution of this work is the identiﬁcation of implicit under-parameterization in deep
Revised: The main contribution of this work is the identiﬁcation of implicit under-parameterization in deep

Original: RL methods that use bootstrapping.
Revised: RL methods that use bootstrapping.

Original: Empirically, we demonstrate a collapse in the rank of the learned features during training, and show it typically corresponds to a drop in performance in the
Revised: Empirically, we demonstrate a collapse in the rank of the learned features during training, and show it typically corresponds to a drop in performance in the

Original: Atari (Bellemare et al., 2013) and continuous control Gym (Brockman et al., 2016) benchmarks in both the ofﬂine and data-efﬁcient online RL settings.
Revised: Atari (Bellemare et al., 2013) and continuous control Gym (Brockman et al., 2016) benchmarks in both the ofﬂine and data-efﬁcient online RL settings.

Original: We verify the emergence of this phenomenon theoretically and characterize settings where implicit under-parameterization can emerge.
Revised: We verify the emergence of this phenomenon theoretically and characterize settings where implicit under-parameterization can emerge.

Original: We then show that mitigating this phenomenon via a simple penalty on the singular values of the learned features improves performance of value-based RL methods in the ofﬂine setting on Atari.
Revised: We then show that mitigating this phenomenon via a simple penalty on the singular values of the learned features improves performance of value-based RL methods in the ofﬂine setting on Atari.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: The goal in RL is to maximize long-term discounted reward in a Markov decision process (MDP), deﬁned as a tuple ( S , A , R, P, γ ) (Puterman, 1994), with state space S , action space A , a reward function R ( s , a ) , transition dynamics P ( s (cid:48) | s , a ) and a discount factor γ ∈ [0 , 1) .
Revised: The goal in RL is to maximize long-term discounted reward in a Markov decision process (MDP), deﬁned as a tuple ( S , A , R, P, γ ) (Puterman, 1994), with state space S , action space A , a reward function R ( s , a ) , transition dynamics P ( s (cid:48) | s , a ) and a discount factor γ ∈ [0 , 1) .

Original: The Q-function Q π ( s , a ) for a policy π ( a | s ) , is the expected long-term discounted reward obtained by executing action a at state s and following π ( a | s ) thereafter,
Revised: The Q-function Q π ( s , a ) for a policy π ( a | s ) , is the expected long-term discounted reward obtained by executing action a at state s and following π ( a | s ) thereafter,

Original: Q π ( s , a ) :
Revised: Q π ( s , a ) :

Original: =
Revised: =

Original: E [ (cid:80) ∞ t =0 γ t R ( s t , a t )] .
Revised: E [ (cid:80) ∞ t =0 γ t R ( s t , a t )] .

Original: Q π ( s , a ) is the ﬁxed point of the Bellman operator T π , ∀ s , a : T π Q ( s , a ) :
Revised: Q π ( s , a ) is the ﬁxed point of the Bellman operator T π , ∀ s , a : T π Q ( s , a ) :

Original: = R ( s , a ) + γ E s (cid:48) ∼ P ( ·| s , a ) , a (cid:48) ∼ π ( ·| s (cid:48) )
Revised: = R ( s , a ) + γ E s (cid:48) ∼ P ( ·| s , a ) , a (cid:48) ∼ π ( ·| s (cid:48) )

Original: [ Q ( s (cid:48) , a (cid:48) )] , which can be written in vector form as: Q π = R + γP π Q π .
Revised: [ Q ( s (cid:48) , a (cid:48) )] , which can be written in vector form as: Q π = R + γP π Q π .

Original: The optimal Q-function, Q ∗ ( s , a ) , is the ﬁxed point of the Bellman optimality operator T : T Q ( s , a ) :
Revised: The optimal Q-function, Q ∗ ( s , a ) , is the ﬁxed point of the Bellman optimality operator T : T Q ( s , a ) :

Original: = R ( s , a ) + γ E s (cid:48) ∼ P ( ·| s , a ) [max a (cid:48) Q ( s (cid:48) , a (cid:48) )] .
Revised: = R ( s , a ) + γ E s (cid:48) ∼ P ( ·| s , a ) [max a (cid:48) Q ( s (cid:48) , a (cid:48) )] .

Original: Practical Q-learning methods ( e.g. , Mnih et al., 2015; Hessel et al., 2018; Haarnoja et al., 2018) convert the Bellman equation into an bootstrapping-based objective for training a Q-network, Q θ , via gradient descent.
Revised: Practical Q-learning methods ( e.g. , Mnih et al., 2015; Hessel et al., 2018; Haarnoja et al., 2018) convert the Bellman equation into an bootstrapping-based objective for training a Q-network, Q θ , via gradient descent.

Original: This objective, known as mean-squared temporal difference (TD) error, is given by:
Revised: This objective, known as mean-squared temporal difference (TD) error, is given by:

Original: L ( θ ) =
Revised: L ( θ ) =

Original: (cid:80) s , a (cid:0) R ( s , a )
Revised: (cid:80) s , a (cid:0) R ( s , a )

Original: + γ ¯ Q θ ( s (cid:48) , a (cid:48) )
Revised: + γ ¯ Q θ ( s (cid:48) , a (cid:48) )

Original: − Q ( s , a ) (cid:1) 2 , where ¯ Q θ is a delayed copy of the
Revised: − Q ( s , a ) (cid:1) 2 , where ¯ Q θ is a delayed copy of the

Original: Q-function, typically referred to as the target network .
Revised: Q-function, typically referred to as the target network .

Original: These methods train Q-networks via gradient descent and slowly update the target network via Polyak averaging on its parameters.
Revised: These methods train Q-networks via gradient descent and slowly update the target network via Polyak averaging on its parameters.

Original: We refer the output of the penultimate layer of the deep Q-network as the learned feature matrix Φ , such that
Revised: We refer the output of the penultimate layer of the deep Q-network as the learned feature matrix Φ , such that

Original: Q ( s , a ) =
Revised: Q ( s , a ) =

Original: w T Φ( s , a ) , where w ∈ R d and Φ ∈ R |S||A|× d . do
Revised: w T Φ( s , a ) , where w ∈ R d and Φ ∈ R |S||A|× d . do

Original: [Table]
Revised: [Table]

Original: θ ( s , a ) and target values y k
Revised: θ ( s , a ) and target values y k

Original: ( s , a ) =
Revised: ( s , a ) =

Original: r + γ max a (cid:48)
Revised: r + γ max a (cid:48)

Original: Q k − 1 ( s (cid:48) , a (cid:48) ) on { ( s , a ) } ∼ µ for training 4:
Revised: Q k − 1 ( s (cid:48) , a (cid:48) ) on { ( s , a ) } ∼ µ for training 4:

Original: Minimize TD error for Q θ via t = 1 , · · · , T gradient descent updates, min θ ( Q θ ( s , a ) − y k ) 2 5: end for
Revised: Minimize TD error for Q θ via t = 1 , · · · , T gradient descent updates, min θ ( Q θ ( s , a ) − y k ) 2 5: end for

Original: For simplicity of analysis, we abstract deep Q-learning methods into a generic ﬁtted Q-iteration ( FQI ) framework (Ernst et al., 2005).
Revised: For simplicity of analysis, we abstract deep Q-learning methods into a generic ﬁtted Q-iteration ( FQI ) framework (Ernst et al., 2005).

Original: We refer to FQI with neural nets as neural FQI (Riedmiller, 2005).
Revised: We refer to FQI with neural nets as neural FQI (Riedmiller, 2005).

Original: In the k -th ﬁtting iteration, FQI trains the Q-function, Q k , to match the target values, y k
Revised: In the k -th ﬁtting iteration, FQI trains the Q-function, Q k , to match the target values, y k

Original: = R + γP π Q k − 1 generated using previous
Revised: = R + γP π Q k − 1 generated using previous

Original: Q-function, Q k − 1 (Algorithm 1).
Revised: Q-function, Q k − 1 (Algorithm 1).

Original: Practical methods can be instantiated as variants of FQI, with different target update styles, different optimizers, etc.
Revised: Practical methods can be instantiated as variants of FQI, with different target update styles, different optimizers, etc.

Original:  I MPLICIT U NDER -P ARAMETERIZATION IN D EEP Q-L EARNING
Revised:  I MPLICIT U NDER -P ARAMETERIZATION IN D EEP Q-L EARNING

Original: In this section, we empirically demonstrate the existence of implicit under-parameterization in deep
Revised: In this section, we empirically demonstrate the existence of implicit under-parameterization in deep

Original: RL methods that use bootstrapping.
Revised: RL methods that use bootstrapping.

Original: We characterize implicit under-parameterization in terms of the effective rank (Yang et al., 2019) of the features learned by a Q-network.
Revised: We characterize implicit under-parameterization in terms of the effective rank (Yang et al., 2019) of the features learned by a Q-network.

Original: The effective rank of the feature matrix Φ , for a threshold δ (we choose δ = 0 . 01 ), denoted as srank δ (Φ) , is given by srank δ (Φ) = min (cid:110)
Revised: The effective rank of the feature matrix Φ , for a threshold δ (we choose δ = 0 . 01 ), denoted as srank δ (Φ) , is given by srank δ (Φ) = min (cid:110)

Original: k : (cid:80) ki
Revised: k : (cid:80) ki

Original: =1 σ i (Φ) (cid:80) d i =

Revised: =1 σ i (Φ) (cid:80) d i =

Original: σ i (Φ) ≥ 1 − δ (cid:111) , where { σ i (Φ) } are the singular values of Φ in decreasing order, i.e., σ 1 ≥ · · · ≥ σ d ≥ 0 .
Revised: σ i (Φ) ≥ 1 − δ (cid:111) , where { σ i (Φ) } are the singular values of Φ in decreasing order, i.e., σ 1 ≥ · · · ≥ σ d ≥ 0 .

Original: Intuitively, this quantity represents the number of “effective” unique components of the feature matrix Φ that form the basis for linearly approximating the
Revised: Intuitively, this quantity represents the number of “effective” unique components of the feature matrix Φ that form the basis for linearly approximating the

Original: Q-values.
Revised: Q-values.

Original: When the network maps different states to orthogonal feature vectors, such as when Φ is an identity matrix, then srank δ (Φ) has high values close to d .
Revised: When the network maps different states to orthogonal feature vectors, such as when Φ is an identity matrix, then srank δ (Φ) has high values close to d .

Original: When the network “aliases” stateaction pairs by mapping them to a smaller subspace, Φ has only a few active singular directions, and srank δ (Φ) takes on a small value.
Revised: When the network “aliases” stateaction pairs by mapping them to a smaller subspace, Φ has only a few active singular directions, and srank δ (Φ) takes on a small value.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Implicit under-parameterization refers to a reduction in the effective rank of the features, srank δ (Φ) , that occurs implicitly as a by-product of learning deep neural network Q-functions.
Revised: Implicit under-parameterization refers to a reduction in the effective rank of the features, srank δ (Φ) , that occurs implicitly as a by-product of learning deep neural network Q-functions.

Original: While rank decrease also occurs in supervised learning, it is usually beneﬁcial for obtaining generalizable solutions (Gunasekar et al., 2017; Arora et al., 2019).
Revised: While rank decrease also occurs in supervised learning, it is usually beneﬁcial for obtaining generalizable solutions (Gunasekar et al., 2017; Arora et al., 2019).

Original: However, we will show that in deep
Revised: However, we will show that in deep

Original: Q-learning, an interaction between bootstrapping and gradient descent can lead to more aggressive rank reduction (or rank collapse), which can hurt performance.
Revised: Q-learning, an interaction between bootstrapping and gradient descent can lead to more aggressive rank reduction (or rank collapse), which can hurt performance.

Original: Experimental setup.
Revised: Experimental setup.

Original: To study implicit under-parameterization empirically, we compute srank δ (Φ) on a minibatch of state-action pairs sampled i.i.d.
Revised: To study implicit under-parameterization empirically, we compute srank δ (Φ) on a minibatch of state-action pairs sampled i.i.d.

Original: from the training data ( i.e. , the dataset in the ofﬂine setting, and the replay buffer in the online setting).
Revised: from the training data ( i.e. , the dataset in the ofﬂine setting, and the replay buffer in the online setting).

Original: We investigate ofﬂine and online RL settings on benchmarks including Atari games (Bellemare et al., 2013) and Gym environments (Brockman et al., 2016).
Revised: We investigate ofﬂine and online RL settings on benchmarks including Atari games (Bellemare et al., 2013) and Gym environments (Brockman et al., 2016).

Original: We also utilize gridworlds described by Fu et al.
Revised: We also utilize gridworlds described by Fu et al.

Original: (2019) to compare the learned Q-function against the oracle solution computed using tabular value iteration.
Revised: (2019) to compare the learned Q-function against the oracle solution computed using tabular value iteration.

Original: We evaluate DQN (Mnih et al., 2015) on gridworld and Atari and SAC (Haarnoja et al., 2018) on Gym domains.
Revised: We evaluate DQN (Mnih et al., 2015) on gridworld and Atari and SAC (Haarnoja et al., 2018) on Gym domains.

Original: Data-efﬁcient ofﬂine RL.
Revised: Data-efﬁcient ofﬂine RL.

Original: In ofﬂine RL, our goal is to learn effective policies by performing Qlearning on a ﬁxed dataset of transitions.
Revised: In ofﬂine RL, our goal is to learn effective policies by performing Qlearning on a ﬁxed dataset of transitions.

Original: We investigate the presence of rank collapse when deep
Revised: We investigate the presence of rank collapse when deep

Original: Q-learning is used with broad state coverage ofﬂine datasets from Agarwal et al.
Revised: Q-learning is used with broad state coverage ofﬂine datasets from Agarwal et al.

Original: In the top row of Figure 1, we show that after an initial learning period, srank δ (Φ) decreases in all domains (Atari, Gym and the gridworld).
Revised: In the top row of Figure 1, we show that after an initial learning period, srank δ (Φ) decreases in all domains (Atari, Gym and the gridworld).

Original: The ﬁnal value of srank δ (Φ) is often quite small – e.g. , in
Revised: The ﬁnal value of srank δ (Φ) is often quite small – e.g. , in

Original: Atari, only 20-100 singular components are active for 512 -dimensional features, implying significant underutilization of network capacity.
Revised: Atari, only 20-100 singular components are active for 512 -dimensional features, implying significant underutilization of network capacity.

Original: Since under-parameterization is implicitly induced by the learning process, even high-capacity value networks behave as low-capacity networks as more training is performed with a bootstrapped objective ( e.g. , mean squared TD error).
Revised: Since under-parameterization is implicitly induced by the learning process, even high-capacity value networks behave as low-capacity networks as more training is performed with a bootstrapped objective ( e.g. , mean squared TD error).

Original: On the gridworld environment, regressing to Q ∗ using supervised regression results in a much higher srank δ (Φ) (black dashed line in Figure 1(left)) than when using neural FQI.
Revised: On the gridworld environment, regressing to Q ∗ using supervised regression results in a much higher srank δ (Φ) (black dashed line in Figure 1(left)) than when using neural FQI.

Original: On Atari, even when a 4x larger ofﬂine dataset with much broader coverage is used (blue line in Figure 1), rank collapse still persists, indicating that implicit under-parameterization is not due to limited ofﬂine dataset size.
Revised: On Atari, even when a 4x larger ofﬂine dataset with much broader coverage is used (blue line in Figure 1), rank collapse still persists, indicating that implicit under-parameterization is not due to limited ofﬂine dataset size.

Original: Figure 1 ( 2 nd row) illustrates that policy performance generally deteriorates as srank (Φ) drops, and eventually collapses simultaneously with the rank collapse.
Revised: Figure 1 ( 2 nd row) illustrates that policy performance generally deteriorates as srank (Φ) drops, and eventually collapses simultaneously with the rank collapse.

Original: While we do not claim that implicit under-parameterization is the only issue in deep Q-learning, the results in Figure 1 show that the emergence of this under-parameterization is strongly associated with poor performance.
Revised: While we do not claim that implicit under-parameterization is the only issue in deep Q-learning, the results in Figure 1 show that the emergence of this under-parameterization is strongly associated with poor performance.

Original: To prevent confounding from the distribution mismatch between the learned policy and the ofﬂine dataset, which often affects the performance of Q-learning methods, we also study CQL (Kumar et al., 2020b), a strong ofﬂine RL algorithm designed to handle distribution mismatch.
Revised: To prevent confounding from the distribution mismatch between the learned policy and the ofﬂine dataset, which often affects the performance of Q-learning methods, we also study CQL (Kumar et al., 2020b), a strong ofﬂine RL algorithm designed to handle distribution mismatch.

Original: We ﬁnd a similar degradation in effective rank and performance for CQL (Figure A.3), implying that underparameterization does not stem from distribution mismatch.
Revised: We ﬁnd a similar degradation in effective rank and performance for CQL (Figure A.3), implying that underparameterization does not stem from distribution mismatch.

Original: We provide evidence for implicit underparameterization in more Atari and Gym environments in Appendix A.1.
Revised: We provide evidence for implicit underparameterization in more Atari and Gym environments in Appendix A.1.

Original: Data-efﬁcient online RL.
Revised: Data-efﬁcient online RL.

Original: Deep Q-learning methods typically use very few gradient updates ( n ) per environment step ( e.g. , DQN takes 1 update every 4 steps on Atari, n = 0 . 25 ).
Revised: Deep Q-learning methods typically use very few gradient updates ( n ) per environment step ( e.g. , DQN takes 1 update every 4 steps on Atari, n = 0 . 25 ).

Original: Improving the
Revised: Improving the

Original: [Figure]
Revised: [Figure]

Original: sample efﬁciency of these methods requires increasing n to utilize the replay data more effectively.
Revised: sample efﬁciency of these methods requires increasing n to utilize the replay data more effectively.

Original: However, we ﬁnd that using larger values of n results in higher levels of rank collapse as well as performance degradation.
Revised: However, we ﬁnd that using larger values of n results in higher levels of rank collapse as well as performance degradation.

Original: In the top row of Figure 2, we show that larger values of n lead to a more aggressive drop in srank δ (Φ) (red vs .
Revised: In the top row of Figure 2, we show that larger values of n lead to a more aggressive drop in srank δ (Φ) (red vs .

Original: blue/orange lines), and that rank continues to decrease with more training.
Revised: blue/orange lines), and that rank continues to decrease with more training.

Original: Furthermore, the bottom row illustrates that larger values of n result in worse performance, corroborating Fu et al.
Revised: Furthermore, the bottom row illustrates that larger values of n result in worse performance, corroborating Fu et al.

Original: ; Fedus et al.
Revised: ; Fedus et al.

Original: (2020b).
Revised: (2020b).

Original: We ﬁnd similar results with the Rainbow algorithm (Hessel et al., 2018) (Appendix A.2).
Revised: We ﬁnd similar results with the Rainbow algorithm (Hessel et al., 2018) (Appendix A.2).

Original: As in the ofﬂine setting, directly regressing to Q ∗ via supervised learning does not cause rank collapse (black line in Figure 2), unlike when using bootstrapping.
Revised: As in the ofﬂine setting, directly regressing to Q ∗ via supervised learning does not cause rank collapse (black line in Figure 2), unlike when using bootstrapping.

Original: 3. U NDERSTANDING I MPLICIT U NDER - PARAMETERIZATION AND ITS I MPLICATIONS
Revised: 3. U NDERSTANDING I MPLICIT U NDER - PARAMETERIZATION AND ITS I MPLICATIONS

Original: How does implicit under-parameterization degrade performance?
Revised: How does implicit under-parameterization degrade performance?

Original: Having established the presence of rank collapse in data-efﬁcient RL, we now discuss how it can adversely affect performance.
Revised: Having established the presence of rank collapse in data-efﬁcient RL, we now discuss how it can adversely affect performance.

Original: As the effective rank of the network features Φ decreases, so does the network’s ability to ﬁt the subsequent target values, and eventually results in inability to ﬁt Q ∗ .
Revised: As the effective rank of the network features Φ decreases, so does the network’s ability to ﬁt the subsequent target values, and eventually results in inability to ﬁt Q ∗ .

Original: In the gridworld domain, we measure this loss of expressivity by measuring the error in ﬁtting oracle-computed Q ∗ values via a linear transformation of Φ .
Revised: In the gridworld domain, we measure this loss of expressivity by measuring the error in ﬁtting oracle-computed Q ∗ values via a linear transformation of Φ .

Original: When rank collapse occurs, the error in ﬁtting Q ∗ steadily increases during training, and the consequent network is not able to predict Q ∗ at all by the end of training (Figure 3a) – this entails a drop in performance.
Revised: When rank collapse occurs, the error in ﬁtting Q ∗ steadily increases during training, and the consequent network is not able to predict Q ∗ at all by the end of training (Figure 3a) – this entails a drop in performance.

Original: In Atari domains, we do not have access to Q ∗ , and so we instead measure TD error, that is, the error in ﬁtting the target value estimates, R + γP π Q k .
Revised: In Atari domains, we do not have access to Q ∗ , and so we instead measure TD error, that is, the error in ﬁtting the target value estimates, R + γP π Q k .

Original: In
Revised: In

Original: S EAQUEST , as rank decreases, the TD error increases (Figure 3b) and the value function is unable to ﬁt the target values, culminating in a performance plateau (Figure 2).
Revised: S EAQUEST , as rank decreases, the TD error increases (Figure 3b) and the value function is unable to ﬁt the target values, culminating in a performance plateau (Figure 2).

Original: This observation is consistent across other environments; we present further supporting evidence in Appendix A.4.
Revised: This observation is consistent across other environments; we present further supporting evidence in Appendix A.4.

Original: Does bootstrapping cause implicit under-parameterization?
Revised: Does bootstrapping cause implicit under-parameterization?

Original: We perform a number of controlled experiments in the gridworld and Atari environments to isolate the connection between rank collapse and bootstrapping.
Revised: We perform a number of controlled experiments in the gridworld and Atari environments to isolate the connection between rank collapse and bootstrapping.

Original: We ﬁrst remove confounding issues of poor network initialization (Fedus et al., 2020a) and non-stationarity
Revised: We ﬁrst remove confounding issues of poor network initialization (Fedus et al., 2020a) and non-stationarity

Original: (Igl et al., 2020) by showing that rank collapse occurs even when the
Revised: (Igl et al., 2020) by showing that rank collapse occurs even when the

Original: Q-network is re-initialized from scratch at the start of each ﬁtting iteration (Figure 3c).
Revised: Q-network is re-initialized from scratch at the start of each ﬁtting iteration (Figure 3c).

Original: To show that the problem is not isolated to the control setting, we show evidence of rank collapse in the policy evaluation setting as well.
Revised: To show that the problem is not isolated to the control setting, we show evidence of rank collapse in the policy evaluation setting as well.

Original: We trained a value network using ﬁtted Q-evaluation for a ﬁxed policy π ( i.e. , using the Bellman operator T π instead of T ), and found that rank drop still occurs (FQE in
Revised: We trained a value network using ﬁtted Q-evaluation for a ﬁxed policy π ( i.e. , using the Bellman operator T π instead of T ), and found that rank drop still occurs (FQE in

Original: Figure 3d).
Revised: Figure 3d).

Original: Finally, we show that by removing bootstrapped updates and instead regressing directly to Monte-Carlo (MC) estimates of the value, the effective rank does not collapse (MC Returns in
Revised: Finally, we show that by removing bootstrapped updates and instead regressing directly to Monte-Carlo (MC) estimates of the value, the effective rank does not collapse (MC Returns in

Original: Figure 3d).
Revised: Figure 3d).

Original: These results, along with similar ﬁndings on other Atari environments (Appendix A.3), our analysis indicates that bootstrapping is at the core of implicit under-parameterization.
Revised: These results, along with similar ﬁndings on other Atari environments (Appendix A.3), our analysis indicates that bootstrapping is at the core of implicit under-parameterization.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we formally analyze implicit under-parameterization and prove that training neural networks with bootstrapping reduces the effective rank of the Q-network, corroborating the empirical observations in the previous section.
Revised: In this section, we formally analyze implicit under-parameterization and prove that training neural networks with bootstrapping reduces the effective rank of the Q-network, corroborating the empirical observations in the previous section.

Original: We focus on policy evaluation (Figure 3d and Figure A.9), where we aim to learn a Q-function that satisﬁes Q = R + γP π Q for a ﬁxed π , for ease of analysis.
Revised: We focus on policy evaluation (Figure 3d and Figure A.9), where we aim to learn a Q-function that satisﬁes Q = R + γP π Q for a ﬁxed π , for ease of analysis.

Original: We also presume a ﬁxed dataset of transitions, D , to learn the Q-function.
Revised: We also presume a ﬁxed dataset of transitions, D , to learn the Q-function.

Original: 4. A NALYSIS VIA K ERNEL R EGRESSION
Revised: 4. A NALYSIS VIA K ERNEL R EGRESSION

Original: We ﬁrst study bootstrapping with neural networks through a mathematical abstraction that treats the
Revised: We ﬁrst study bootstrapping with neural networks through a mathematical abstraction that treats the

Original: Q-network as a kernel machine, following the neural tangent kernel (NTK) formalism (Jacot et al., 2018).
Revised: Q-network as a kernel machine, following the neural tangent kernel (NTK) formalism (Jacot et al., 2018).

Original: Building on prior analysis of self-distillation (Mobahi et al., 2020), we assume that each iteration of bootstrapping, the Q-function optimizes the squared TD error to target labels y k with a kernel regularizer.
Revised: Building on prior analysis of self-distillation (Mobahi et al., 2020), we assume that each iteration of bootstrapping, the Q-function optimizes the squared TD error to target labels y k with a kernel regularizer.

Original: Intuitively, this regularizer captures the inductive bias from gradient descent, resembling the regularization imposed by gradient descent under NTK (Mobahi et al., 2020).
Revised: Intuitively, this regularizer captures the inductive bias from gradient descent, resembling the regularization imposed by gradient descent under NTK (Mobahi et al., 2020).

Original: The error is computed on ( s i , a
Revised: The error is computed on ( s i , a

Original: i ) ∈
Revised: i ) ∈

Original: D whereas the regularization imposed by a universal kernel u with a coefﬁcient of c ≥ 0 is applied to the Q-values at all state-action pairs as shown in Equation 1.
Revised: D whereas the regularization imposed by a universal kernel u with a coefﬁcient of c ≥ 0 is applied to the Q-values at all state-action pairs as shown in Equation 1.

Original: The solution found by this kernel regression model for a choice of c > 0 can be alternatively interpreted as performing gradient descent on the TD error for a small number of iterations with early stopping (Suggala et al., 2018).
Revised: The solution found by this kernel regression model for a choice of c > 0 can be alternatively interpreted as performing gradient descent on the TD error for a small number of iterations with early stopping (Suggala et al., 2018).

Original: [Equation]
Revised: [Equation]

Original: The solution to Equation 1 can be expressed as Q k +1 ( s , a ) =
Revised: The solution to Equation 1 can be expressed as Q k +1 ( s , a ) =

Original: g T ( s , a ) ( c I + G ) − 1 y k , where G is the Gram matrix for a special positive-deﬁnite kernel (Duffy, 2015) and g ( s , a ) denotes the row of G corresponding to the input ( s , a ) (Mobahi et al., 2020, Proposition 1).
Revised: g T ( s , a ) ( c I + G ) − 1 y k , where G is the Gram matrix for a special positive-deﬁnite kernel (Duffy, 2015) and g ( s , a ) denotes the row of G corresponding to the input ( s , a ) (Mobahi et al., 2020, Proposition 1).

Original: A detailed proof is in Appendix C. When combined with the ﬁtted Q-iteration recursion, setting labels
Revised: A detailed proof is in Appendix C. When combined with the ﬁtted Q-iteration recursion, setting labels

Original: y k
Revised: y k

Original: = R + γP π Q k − 1 , we recover a recurrence that relates subsequent value function iterates
Revised: = R + γP π Q k − 1 , we recover a recurrence that relates subsequent value function iterates

Original: [Equation]
Revised: [Equation]

Original: Equation 2 follows from unrolling the recurrence and setting the algorithm-agnostic initial Q-value,
Revised: Equation 2 follows from unrolling the recurrence and setting the algorithm-agnostic initial Q-value,

Original: Q 0 , to be 0 .
Revised: Q 0 , to be 0 .

Original: We now show that the sparsity of singular values of the matrix M k generally increases over ﬁtting iterations, implying that the effective rank of M k diminishes with more iterations.
Revised: We now show that the sparsity of singular values of the matrix M k generally increases over ﬁtting iterations, implying that the effective rank of M k diminishes with more iterations.

Original: For this result, we assume that the matrix S is normal, i.e. , the norm of the (complex) eigenvalues of S is equal to its singular values.
Revised: For this result, we assume that the matrix S is normal, i.e. , the norm of the (complex) eigenvalues of S is equal to its singular values.

Original: We will discuss how this assumption can be relaxed after stating the result with the normality assumption.
Revised: We will discuss how this assumption can be relaxed after stating the result with the normality assumption.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let S be a shorthand for S =
Revised: Let S be a shorthand for S =

Original: γP π A
Revised: γP π A

Original: and assume S is a normal matrix.
Revised: and assume S is a normal matrix.

Original: Then there exists an inﬁnite, strictly increasing sequence of ﬁtting iterations, ( k l ) ∞ l =1 starting from k 1 = 0 , such that, for any two singular-values σ i ( S ) and σ j ( S ) of S with σ i ( S ) <
Revised: Then there exists an inﬁnite, strictly increasing sequence of ﬁtting iterations, ( k l ) ∞ l =1 starting from k 1 = 0 , such that, for any two singular-values σ i ( S ) and σ j ( S ) of S with σ i ( S ) <

Original: σ j ( S ) ,
Revised: σ j ( S ) ,

Original: [Equation]
Revised: [Equation]

Original: Hence, the effective rank of M k satisﬁes: srank δ ( M k l (cid:48) ) ≤ srank δ ( M k l ) .
Revised: Hence, the effective rank of M k satisﬁes: srank δ ( M k l (cid:48) ) ≤ srank δ ( M k l ) .

Original: Moreover, if S is positive semi-deﬁnite, then ( k l ) ∞ l =1 = N , i.e. , effective rank continuously decreases in each ﬁtting iteration.
Revised: Moreover, if S is positive semi-deﬁnite, then ( k l ) ∞ l =1 = N , i.e. , effective rank continuously decreases in each ﬁtting iteration.

Original: We provide a proof of the theorem above as well as present a stronger variant that shows a gradual decrease in the effective rank for ﬁtting iterations outside this inﬁnite sequence in Appendix C. As k increases along the sequence of iterations given by k =
Revised: We provide a proof of the theorem above as well as present a stronger variant that shows a gradual decrease in the effective rank for ﬁtting iterations outside this inﬁnite sequence in Appendix C. As k increases along the sequence of iterations given by k =

Original: ( k l ) ∞ l =1 , the effective rank of the matrix
Revised: ( k l ) ∞ l =1 , the effective rank of the matrix

Original: M k drops, leading to low expressivity of this matrix.
Revised: M k drops, leading to low expressivity of this matrix.

Original: Since M k linearly maps rewards to the Qfunction (Equation 2), drop in expressivity results of M k in the inability to model the actual Q π .
Revised: Since M k linearly maps rewards to the Qfunction (Equation 2), drop in expressivity results of M k in the inability to model the actual Q π .

Original: Extension to general matrices S.
Revised: Extension to general matrices S.

Original: We can relax the normality assumption on S in Theorem 4.1.
Revised: We can relax the normality assumption on S in Theorem 4.1.

Original: An analogous statement holds for non-normal matrices S for a slightly different notion of effective rank, denoted as srank δ,λ ( M k ) , that utilizes eigenvalue norms instead of singular values.
Revised: An analogous statement holds for non-normal matrices S for a slightly different notion of effective rank, denoted as srank δ,λ ( M k ) , that utilizes eigenvalue norms instead of singular values.

Original: Formally, let λ 1 ( M k ) , · · · , λ 2 ( M k ) , · · · be the (complex) eigenvalues of M k arranged in decreasing order of their norms, i.e. , , | λ 1 ( M k ) | ≥ | λ 2 ( M k ) | ≥ · · · , then,
Revised: Formally, let λ 1 ( M k ) , · · · , λ 2 ( M k ) , · · · be the (complex) eigenvalues of M k arranged in decreasing order of their norms, i.e. , , | λ 1 ( M k ) | ≥ | λ 2 ( M k ) | ≥ · · · , then,

Original: [Equation]
Revised: [Equation]

Original: A statement essentially analogous to Theorem 4.1 suggests that in this general case, srank δ,λ ( M k ) decreases for all (complex) diagonalizable matrices S , which is the set of almost all matrices of size dim ( S ) .
Revised: A statement essentially analogous to Theorem 4.1 suggests that in this general case, srank δ,λ ( M k ) decreases for all (complex) diagonalizable matrices S , which is the set of almost all matrices of size dim ( S ) .

Original: Now, if the square matrix S is approximately normal, i.e. , when | σ i ( S ) − | λ i ( S ) || is small, then the result in Theorem 4.1 also holds approximately as we discuss at the end of Appendix C.
Revised: Now, if the square matrix S is approximately normal, i.e. , when | σ i ( S ) − | λ i ( S ) || is small, then the result in Theorem 4.1 also holds approximately as we discuss at the end of Appendix C.

Original: In Appendix A.7, we experimentally demonstrate on Atari games and gridworld that for the feature matrix, Φ , which is the counterpart of S in practice, both the notions of effective rank closely follow each other indicating that the Theorem 4.1 explains our practical observations of rank collapse.
Revised: In Appendix A.7, we experimentally demonstrate on Atari games and gridworld that for the feature matrix, Φ , which is the counterpart of S in practice, both the notions of effective rank closely follow each other indicating that the Theorem 4.1 explains our practical observations of rank collapse.

Original: Summary of the proof.
Revised: Summary of the proof.

Original: Our analysis of the rank collapse effect due to bootstrapping and gradient descent from the view of regularized kernel regression suggests that rank drop happens as more training is performed.
Revised: Our analysis of the rank collapse effect due to bootstrapping and gradient descent from the view of regularized kernel regression suggests that rank drop happens as more training is performed.

Original: In contrast to self-distillation (Mobahi et al., 2020), rank drop may not happen in every consecutive iteration (and rank may increase between two consecutive iterations occasionally), but srank δ will exhibit a decreasing trend.
Revised: In contrast to self-distillation (Mobahi et al., 2020), rank drop may not happen in every consecutive iteration (and rank may increase between two consecutive iterations occasionally), but srank δ will exhibit a decreasing trend.

Revised: We can extend this proof to utilize the Bellman optimality operator under strong assumptions for the matrices

Revised: P π i A generated using greedy policies

Revised: π i used for backups in iteration i , and we leave it to future work.

Original: 4. A NALYSIS WITH D EEP L INEAR N ETWORKS UNDER G RADIENT D ESCENT
Revised: 4. A NALYSIS WITH D EEP L INEAR N ETWORKS UNDER G RADIENT D ESCENT

Original: While Section 4.1 demonstrates rank collapse will occur in a kernel-regression model of Q-learning, it does not illustrate when the rank collapse occurs.
Revised: While Section 4.1 demonstrates rank collapse will occur in a kernel-regression model of Q-learning, it does not illustrate when the rank collapse occurs.

Original: To better specify a point in training when rank collapse emerges, we present a complementary derivation motivated by implicit regularization of gradient descent over deep linear neural networks (Arora et al., 2019).
Revised: To better specify a point in training when rank collapse emerges, we present a complementary derivation motivated by implicit regularization of gradient descent over deep linear neural networks (Arora et al., 2019).

Original: Our analysis will show that rank collapse can emerge as the generated target values begin to approach the previous value estimate, in particular, when in the vicinity of the optimal Q-function.
Revised: Our analysis will show that rank collapse can emerge as the generated target values begin to approach the previous value estimate, in particular, when in the vicinity of the optimal Q-function.

Original: Proof strategy.
Revised: Proof strategy.

Original: Our proof is decomposed into two steps: (1) We show that the effective rank of the feature matrix decreases within one ﬁtting iteration (for a given target value) due to the implicit regularization effect (this proof is different from Arora et al. (2019), (2) We show that this effective rank drop is “compounded” as we train using a bootstrapped objective.
Revised: Our proof is decomposed into two steps: (1) We show that the effective rank of the feature matrix decreases within one ﬁtting iteration (for a given target value) due to the implicit regularization effect, (2) We show that this effective rank drop is “compounded” as we train using a bootstrapped objective.

Original: Proposition 4.1 corresponds to (1) and Theorem 4.2 corresponds to (2) .
Revised: Proposition 4.1 corresponds to (1) and Theorem 4.2 corresponds to (2) .

Original: Additional notation and assumptions.
Revised: Additional notation and assumptions.

Original: We represent the Q-function as a deep linear network with at ≥ 3 layers, such that Q ( s , a ) =
Revised: We represent the Q-function as a deep linear network with at ≥ 3 layers, such that Q ( s , a ) =

Original: W N W φ [ s ; a ] , where N ≥ 3 , W N ∈ R 1 × d N − 1 and
Revised: W N W φ [ s ; a ] , where N ≥ 3 , W N ∈ R 1 × d N − 1 and

Original: W φ = W N − 1 W N − 2 · · · W 1 with W i ∈ R d i × d i − 1 for i = 1 , . . .
Revised: W φ = W N − 1 W N − 2 · · · W 1 with W i ∈ R d i × d i − 1 for i = 1 , . . .

Original: , N − 1 .
Revised: , N − 1 .

Original: W φ maps an input [ s ; a ] to corresponding penultimate layer’ features Φ( s , a ) .
Revised: W φ maps an input [ s ; a ] to corresponding penultimate layer’ features Φ( s , a ) .

Original: Let W j ( k, t ) denotes the weight matrix W j at the t -th step of gradient descent during the k -th ﬁtting iteration (Algorithm 1).
Revised: Let W j ( k, t ) denotes the weight matrix W j at the t -th step of gradient descent during the k -th ﬁtting iteration (Algorithm 1).

Original: We deﬁne
Revised: We deﬁne

Original: W k,t = W N ( k, t ) W φ ( k, t ) and L N,k +1 ( W k,t ) as the TD error objective in the k -th ﬁtting iteration.
Revised: W k,t = W N ( k, t ) W φ ( k, t ) and L N,k +1 ( W k,t ) as the TD error objective in the k -th ﬁtting iteration.

Original: We study srank δ ( W φ ( k, t )) since the rank of features Φ = W φ ( k, t )
Revised: We study srank δ ( W φ ( k, t )) since the rank of features Φ = W φ ( k, t )

Original: [ S , A ] is equal to rank of
Revised: [ S , A ] is equal to rank of

Original: W φ ( k, t ) provided the state-action inputs have high rank.
Revised: W φ ( k, t ) provided the state-action inputs have high rank.

Original: We assume that the evolution of the weights is governed by a continuous-time differential equation (Arora et al., 2018) within each ﬁtting iteration k .
Revised: We assume that the evolution of the weights is governed by a continuous-time differential equation (Arora et al., 2018) within each ﬁtting iteration k .

Original: We assume that all except the last-layer weights follow a “balancedness” property, which suggests that the weight matrices in the consecutive layers in the deep linear network share the same singular values (but with different multiplicities and permutations).
Revised: We assume that all except the last-layer weights follow a “balancedness” property (mathematically shown in Equation D.4), which suggests that the weight matrices in the consecutive layers in the deep linear network share the same singular values (but with different multiplicities and permutations).

Original: This is different from Arora et al.
Revised: This is different from Arora et al.

Original: (2018; 2019) in that we do not assume balancedness for the last layer, which will trivially give us rank-1 features.
Revised: (2018; 2019) in that we do not assume balancedness for the last layer, which will trivially give us rankfeatures.

Original: In this model, we can characterize the evolution of the singular values of the feature matrix W φ ( k, t ) , using techniques analogous to Arora et al.
Revised: In this model, we can characterize the evolution of the singular values of the feature matrix W φ ( k, t ) , using techniques analogous to Arora et al.

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The singular values of the feature matrix W φ ( k, t ) evolve according to:
Revised: The singular values of the feature matrix W φ ( k, t ) evolve according to:

Original: [Equation]
Revised: [Equation]

Original: for r = 1 , · · · , min N − 1 i =1 d i , where u r ( k, t ) and v r ( k, t ) denote the left and right singular vectors of the feature matrix, W φ ( k, t ) , respectively.
Revised: for r = 1 , · · · , min N − 1 i =1 d i , where u r ( k, t ) and v r ( k, t ) denote the left and right singular vectors of the feature matrix, W φ ( k, t ) , respectively.

Original: [Figure]
Revised: [Figure]

Original: 1010101010s
Revised: 1010101010s

Original: i n g u l a r v a l u e s ( l o g s c a l e )
Revised: i n g u l a r v a l u e s ( l o g s c a l e )

Original: Seaquest (Singular values) σ max σ 2 σ
Revised: Seaquest (Singular values) σ max σ 2 σ

Original: [Figure]
Revised: [Figure]

Original: Solving the differential equation (4) tells us that larger singular values will evolve at a exponentially faster rate than smaller singular values (as we also formally show in Appendix D.1) and the difference in their magnitudes disproportionately increase with increasing t .
Revised: Solving the differential equation (4) tells us that larger singular values will evolve at a exponentially faster rate than smaller singular values (as we also formally show in Appendix D.1) and the difference in their magnitudes disproportionately increase with increasing t .

Original: This behavior also occurs empirically, illustrated in the ﬁgure on the right (also see Figure D.1), where larger singular values are orders of magnitude larger than smaller singular values.
Revised: This behavior also occurs empirically, illustrated in the ﬁgure on the right (also see Figure D.1), where larger singular values are orders of magnitude larger than smaller singular values.

Original: Hence, the effective rank, srank δ ( W φ ( k, t )) , will decrease with more gradient steps within a ﬁtting iteration k .
Revised: Hence, the effective rank, srank δ ( W φ ( k, t )) , will decrease with more gradient steps within a ﬁtting iteration k .

Original: Rank drop within a ﬁtting iteration “compounds” due to bootstrapping.
Revised: Rank drop within a ﬁtting iteration “compounds” due to bootstrapping.

Original: So far, we have shown that gradient descent within a ﬁtting iteration has an implicit bias towards solutions with low effective rank.
Revised: So far, we have shown that gradient descent within a ﬁtting iteration has an implicit bias towards solutions with low effective rank.

Original: Now, we aim to understand how this afﬁnity towards producing low rank solutions compounds as bootstrapping is performed.
Revised: Now, we aim to understand how this afﬁnity towards producing low rank solutions compounds as bootstrapping is performed.

Original: To begin, we express the solution obtained at the end of a ﬁtting iteration
Revised: To begin, we express the solution obtained at the end of a ﬁtting iteration

Original: k
Revised: k

Original: (Algorithm 1), using a sort of transformation between gradient descent and regularization paths (Suggala et al., 2018) as (with λ k > 0 ):
Revised: (Algorithm 1), using a sort of transformation between gradient descent and regularization paths (Suggala et al., 2018) as (with λ k > 0 ):

Original: [Equation]
Revised: [Equation]

Original: This provides an abstraction to analyze the evolution of srank δ ( W φ ) across ﬁtting iterations.
Revised: This provides an abstraction to analyze the evolution of srank δ ( W φ ) across ﬁtting iterations.

Original: Before considering the full bootstrapping setting, we analyze the self-training or self-distillation setting to build intuition, where the target values are equal to the previous Q-values, y k ( s , a ) =
Revised: Before considering the full bootstrapping setting, we analyze the self-training or self-distillation setting to build intuition, where the target values are equal to the previous Q-values, y k ( s , a ) =

Original: Q k − 1 ( s , a ) =
Revised: Q k − 1 ( s , a ) =

Original: W N ( k − 1 , T ) W φ ( k − 1 , T )[ s ; a ] .
Revised: W N ( k − 1 , T ) W φ ( k − 1 , T )[ s ; a ] .

Original: To be able to compare the values of effective ranks of features at iterations k and k − 1 , srank δ ( W φ ( k, T )) and srank δ ( W φ ( k − 1 , T )) , when obtaining weights by solving Equation 5, we ﬁrst note that just copying over weights from iteration k − 1 to k is a feasible point in solving Equation 5.
Revised: To be able to compare the values of effective ranks of features at iterations k and k − 1 , srank δ ( W φ ( k, T )) and srank δ ( W φ ( k − 1 , T )) , when obtaining weights by solving Equation 5, we ﬁrst note that just copying over weights from iteration k − 1 to k is a feasible point in solving Equation 5.

Original: This approach attains zero TD error without increasing effective rank.
Revised: This approach attains zero TD error without increasing effective rank.

Original: Since the optimal solution W φ ( k, T ) attains a smaller (or equal) value of the objective in
Revised: Since the optimal solution W φ ( k, T ) attains a smaller (or equal) value of the objective in

Original: Equation 5 compared to any feasible point, it can only do so by obtaining a smaller (or equal) value of srank δ ( W φ ( k, T )) at the expense of non-zero (or zero respectively) TD error.
Revised: Equation 5 compared to any feasible point, it can only do so by obtaining a smaller (or equal) value of srank δ ( W φ ( k, T )) at the expense of non-zero (or zero respectively) TD error.

Original: This tradeoff between TD error and srank δ ( W φ ) is also generally observed in practice ( c.f. Figure 2 (middle) and Figure 3a).
Revised: This tradeoff between TD error and srank δ ( W φ ) is also generally observed in practice ( c.f. Figure 2 (middle) and Figure 3a).

Original: Applying this argument inductively implies that srank δ ( W φ ( k, T )) decreases with increasing k , right from the start ( i.e. , k = 0 ) with self-training or self-distillation.
Revised: Applying this argument inductively implies that srank δ ( W φ ( k, T )) decreases with increasing k , right from the start ( i.e. , k = 0 ) with self-training or self-distillation.

Original: Implicit under-parameterization with complete bootstrapped updates in RL.
Revised: Implicit under-parameterization with complete bootstrapped updates in RL.

Original: In the complete
Revised: In the complete

Original: RL setting, the target values are now given by y k ( s , a ) =
Revised: RL setting, the target values are now given by y k ( s , a ) =

Original: r ( s , a ) + γP π Q k − 1 ( s , a ) .
Revised: r ( s , a ) + γP π Q k − 1 ( s , a ) .

Original: Unlike the self-training setting, y k ( s , a ) is not directly expressible as a function of the previous W φ ( k, T ) due to additional reward and dynamics transformations, and the rank of the solution to Equation 5 may increase initially.
Revised: Unlike the self-training setting, y k ( s , a ) is not directly expressible as a function of the previous W φ ( k, T ) due to additional reward and dynamics transformations, and the rank of the solution to Equation 5 may increase initially.

Original: We investigate the emergence of under-parameterization as the value function gets closer to the Bellman ﬁxed point, which is a favourable initialization for the Q-function.
Revised: We investigate the emergence of under-parameterization as the value function gets closer to the Bellman ﬁxed point, which is a favourable initialization for the Q-function.

Original: In this case, the learning dynamics begins to resemble the self-training regime, since the target values approach the previous value iterate y k ≈ Q k − 1 .
Revised: In this case, the learning dynamics begins to resemble the self-training regime, since the target values approach the previous value iterate y k ≈ Q k − 1 .

Original: As learning approaches this regime, the rank decrease phenomenon discussed above begins to occur and under-parameterization emerges.
Revised: As learning approaches this regime, the rank decrease phenomenon discussed above begins to occur and under-parameterization emerges.

Original: This insight is captured by our next result, which shows rank decrease when target values are sufﬁciently close to the previous value estimate ( e.g. , near convergence).
Revised: This insight is captured by our next result, which shows rank decrease when target values are sufﬁciently close to the previous value estimate ( e.g. , near convergence).

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Suppose target values
Revised: Suppose target values

Original: y k
Revised: y k

Original: = R + γP π Q k − 1 are close to the previous value estimate
Revised: = R + γP π Q k − 1 are close to the previous value estimate

Original: Q k − 1 , i.e. ∀ s , a , y k
Revised: Q k − 1 , i.e. ∀ s , a , y k

Original: ( s , a ) =
Revised: ( s , a ) =

Original: Q k − 1 ( s , a ) + ε ( s , a ) , with | ε ( s , a ) | (cid:28) | Q k − 1 ( s , a ) | .
Revised: Q k − 1 ( s , a ) + ε ( s , a ) , with | ε ( s , a ) | (cid:28) | Q k − 1 ( s , a ) | .

Original: Then, there is a constant (cid:15) 

Revised: Then, there is a constant (cid:15)

Original: such that for all (cid:107) ε (cid:107) < ε 0 , the solution to Equation 5, Q k , has lower srank than Q k − 1 :
Revised: such that for all (cid:107) ε (cid:107) < ε 0 , the solution to Equation 5, Q k , has lower srank than Q k − 1 :

Original: [Equation]
Revised: [Equation]

Original: The constant (cid:15) 0 depends on the magnitude of W N and W Φ as well as the gap between singular values of W φ ; we provide the complete form in Appendix D.2.
Revised: The constant (cid:15) 0 depends on the magnitude of W N and W Φ as well as the gap between singular values of W φ ; we provide the complete form in Appendix D.2.

Original: Consequence of this theorem.
Revised: Consequence of this theorem.

Original: One important practical consequence of this theorem is that rank decrease occurs when the value function is near (but not at) the ﬁxed point, since gradient descent will preferentially choose solutions with decreased effective rank, which can also increase TD error in many MDPs.
Revised: One important practical consequence of this theorem is that rank decrease occurs when the value function is near (but not at) the ﬁxed point, since gradient descent will preferentially choose solutions with decreased effective rank, which can also increase TD error in many MDPs.

Original: Increased TD error values will then prevent convergence to the ﬁxed point even the
Revised: Increased TD error values will then prevent convergence to the ﬁxed point even the

Original: TD learning process starts from the vicinity of this ﬁxed point, which is a “favorable” initialization for the Q-function training.
Revised: TD learning process starts from the vicinity of this ﬁxed point, which is a “favorable” initialization for the Q-function training.

Original: This is consistent with the empirical evidence in Figure 1, where rank collapse happens right after a peak in performance is achieved.
Revised: This is consistent with the empirical evidence in Figure 1, where rank collapse happens right after a peak in performance is achieved.

Original: To empirically show the consequence of Theorem 4.2 that a decrease in srank δ ( W φ ) values can lead to an increase in the distance to the ﬁxed point in a neighborhood around the ﬁxed point, we performed a controlled experiment on a deep linear net shown in Figure 4 that measures the relationship between of srank δ (Φ) and the error to the projected TD ﬁxed point | Q − Q ∗ | .
Revised: To empirically show the consequence of Theorem 4.2 that a decrease in srank δ ( W φ ) values can lead to an increase in the distance to the ﬁxed point in a neighborhood around the ﬁxed point, we performed a controlled experiment on a deep linear net shown in Figure 4 that measures the relationship between of srank δ (Φ) and the error to the projected TD ﬁxed point | Q − Q ∗ | .

Original: Note that a drop in srank δ (Φ) corresponds to a increased value of | Q − Q ∗ | indicating that rank drop when Q get close to a ﬁxed point can affect convergence to it.
Revised: Note that a drop in srank δ (Φ) corresponds to a increased value of | Q − Q ∗ | indicating that rank drop when Q get close to a ﬁxed point can affect convergence to it.

Original:  M ITIGATING U NDER -P ARAMETRIZATION I MPROVES D EEP Q-L EARNING
Revised:  M ITIGATING U NDER -P ARAMETRIZATION I MPROVES D EEP Q-L EARNING

Original: We now show that mitigating implicit under-parameterization by preventing rank collapse can improve performance.
Revised: We now show that mitigating implicit under-parameterization by preventing rank collapse can improve performance.

Original: We place special emphasis on the ofﬂine RL setting in this section, since it is particularly vulnerable to the adverse effects of rank collapse.
Revised: We place special emphasis on the ofﬂine RL setting in this section, since it is particularly vulnerable to the adverse effects of rank collapse.

Original: [Figure]
Revised: [Figure]

Original: We devise a penalty (or a regularizer)
Revised: We devise a penalty (or a regularizer)

Original: L p (Φ) that encourages higher effective rank of the learned features, srank δ (Φ) , to prevent rank collapse.
Revised: L p (Φ) that encourages higher effective rank of the learned features, srank δ (Φ) , to prevent rank collapse.

Original: The effective rank function srank δ (Φ) is non-differentiable, so we choose a simple surrogate that can be optimized over deep networks.
Revised: The effective rank function srank δ (Φ) is non-differentiable, so we choose a simple surrogate that can be optimized over deep networks.

Original: Since effective rank is maximized when the magnitude of the singular values is roughly balanced, one way to increase effective rank is to minimize the largest singular value of Φ , σ max (Φ) , while simultaneously maximizing the smallest singular value, σ min (Φ) .
Revised: Since effective rank is maximized when the magnitude of the singular values is roughly balanced, one way to increase effective rank is to minimize the largest singular value of Φ , σ max (Φ) , while simultaneously maximizing the smallest singular value, σ min (Φ) .

Original: We construct a simple penalty
Revised: We construct a simple penalty

Original: L p (Φ) derived from this intuition, formally given by:
Revised: L p (Φ) derived from this intuition, formally given by:

Original: [Equation]
Revised: [Equation]

Original: L p
Revised: L p

Original: [Equation]
Revised: [Equation]

Original: Φ) can be computed by invoking the singular value decomposition subroutines in standard automatic differentiation frameworks (Abadi et al., 2016; Paszke et al., 2019).
Revised: Φ) can be computed by invoking the singular value decomposition subroutines in standard automatic differentiation frameworks (Abadi et al., 2016; Paszke et al., 2019).

Original: We estimate the singular values over the feature matrix computed over a minibatch, and add the resulting value of L p as a penalty to the TD error objective, with a tradeoff factor α = 0 .
Revised: We estimate the singular values over the feature matrix computed over a minibatch, and add the resulting value of L p as a penalty to the TD error objective, with a tradeoff factor α = 0 .

Original: Does L p (Φ) address rank collapse?
Revised: Does L p (Φ) address rank collapse?

Original: We ﬁrst verify whether controlling the minimum and maximum singular values using L p (Φ) actually prevents rank collapse.
Revised: We ﬁrst verify whether controlling the minimum and maximum singular values using L p (Φ) actually prevents rank collapse.

Original: When using this penalty on the gridworld problem (Figure 5a), the effective rank does not collapse, instead gradually decreasing at the onset and then plateauing, akin to the evolution of effective rank in supervised learning.
Revised: When using this penalty on the gridworld problem (Figure 5a), the effective rank does not collapse, instead gradually decreasing at the onset and then plateauing, akin to the evolution of effective rank in supervised learning.

Original: In
Revised: In

Original: Figure 5b, we plot the evolution of stable rank on two Atari games in the ofﬂine setting, S EAQUEST and B REAKOUT (all games in Appendix A.5), and observe that the addition of L p also generally leads to increasing rank values.
Revised: Figure 5b, we plot the evolution of stable rank on two Atari games in the ofﬂine setting, S EAQUEST and B REAKOUT (all games in Appendix A.5), and observe that the addition of L p also generally leads to increasing rank values.

Original: Does mitigating rank collapse improve performance?
Revised: Does mitigating rank collapse improve performance?

Original: We now evaluate the performance of the penalty using DQN (Mnih et al., 2015) and CQL (Kumar et al., 2020b) on Atari dataset from Agarwal et al.
Revised: We now evaluate the performance of the penalty using DQN (Mnih et al., 2015) and CQL (Kumar et al., 2020b) on Atari dataset from Agarwal et al.

Original: (2020) (5% replay data), used in Section 3.
Revised: (2020) (5% replay data), used in Section 3.

Original: Figure 6 summarizes the relative improvement from using the penalty for 16 Atari games.
Revised: Figure 6 summarizes the relative improvement from using the penalty for 16 Atari games.

Original: Adding the penalty to DQN improves performance on all 16/16 games with a median improvement of 74.5% ; adding it to CQL, a state-of-the-art ofﬂine algorithm, improves performance on 11/16 games with median improvement of 14.1% .
Revised: Adding the penalty to DQN improves performance on all 16/16 games with a median improvement of 74.5% ; adding it to CQL, a state-of-the-art ofﬂine algorithm, improves performance on 11/16 games with median improvement of 14.1% .

Original: Prior work has discussed that standard Q-learning methods designed for the online setting, such as DQN, are generally ineffective with small ofﬂine datasets (Kumar et al., 2020b; Agarwal et al., 2020).
Revised: Prior work has discussed that standard Q-learning methods designed for the online setting, such as DQN, are generally ineffective with small ofﬂine datasets (Kumar et al., 2020b; Agarwal et al., 2020).

Original: Our results show that mitigating rank collapse makes even such simple methods substantially more effective in this setting, suggesting that rank collapse and the resulting implicit under-parameterization may be an crucial piece of the puzzle in explaining the challenges of ofﬂine RL.
Revised: Our results show that mitigating rank collapse makes even such simple methods substantially more effective in this setting, suggesting that rank collapse and the resulting implicit under-parameterization may be an crucial piece of the puzzle in explaining the challenges of ofﬂine RL.

Original: [Figure]
Revised: [Figure]

Original: We also evaluated the regularizer L p (Φ) in the dataefﬁcient online RL setting, with results in Appendix A.6.
Revised: We also evaluated the regularizer L p (Φ) in the dataefﬁcient online RL setting, with results in Appendix A.6.

Original: This variant achieved median improvement of 20.6% performance with Rainbow (Hes
Revised: This variant achieved median improvement of 20.6% performance with Rainbow (Hes

Original: - sel et al., 2018), however performed poorly with DQN, where it reduced median performance by 11.5%.
Revised: - sel et al., 2018), however performed poorly with DQN, where it reduced median performance by 11.5%.

Original: Thus,
Revised: Thus,

Original: while our proposed penalty is effective in many cases spanning both ofﬂine and data-efﬁcient online settings, it does not solve the problem fully, and a more sophisticated solution may better prevent the issues with implicit under-parameterization.
Revised: while our proposed penalty is effective in many cases spanning both ofﬂine and data-efﬁcient online settings, it does not solve the problem fully, and a more sophisticated solution may better prevent the issues with implicit under-parameterization.

Original: Nevertheless, our results suggest that mitigation of implicit under-parameterization can improve performance of data-efﬁcient RL.
Revised: Nevertheless, our results suggest that mitigation of implicit under-parameterization can improve performance of data-efﬁcient RL.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior work has extensively studied the learning dynamics of Q-learning with tabular and linear function approximation, to study error propagation (Munos, 2003; Farahmand et al., 2010) and to prevent divergence (De Farias, 2002; Maei et al., 2009;
Revised: Prior work has extensively studied the learning dynamics of Q-learning with tabular and linear function approximation, to study error propagation (Munos, 2003; Farahmand et al., 2010) and to prevent divergence (De Farias, 2002; Maei et al., 2009;

Original: Sutton et al., 2009; Dai et al., 2018), as opposed to deep Q-learning analyzed in this work.
Revised: Sutton et al., 2009; Dai et al., 2018), as opposed to deep Q-learning analyzed in this work.

Original: Q-learning has been shown to have favorable optimization properties with certain classes of features (Ghosh & Bellemare, 2020), but our work shows that the features learned by a neural net when minimizing TD error do not enjoy such guarantees, and instead suffer from rank collapse.
Revised: Q-learning has been shown to have favorable optimization properties with certain classes of features (Ghosh & Bellemare, 2020), but our work shows that the features learned by a neural net when minimizing TD error do not enjoy such guarantees, and instead suffer from rank collapse.

Original: Recent theoretical analyses of deep Q-learning have shown convergence under restrictive assumptions (Yang et al., 2020;
Revised: Recent theoretical analyses of deep Q-learning have shown convergence under restrictive assumptions (Yang et al., 2020;

Original: Cai et al., 2019; Zhang et al., 2020; Xu &
Revised: Cai et al., 2019; Zhang et al., 2020; Xu &

Original: Gu, 2019), but Theorem 4.2 shows that implicit under-parameterization appears when the estimates of the value function approach the optimum, potentially preventing convergence.
Revised: Gu, 2019), but Theorem 4.2 shows that implicit under-parameterization appears when the estimates of the value function approach the optimum, potentially preventing convergence.

Original: Xu et al.
Revised: Xu et al.

Original: (2005; 2007) present variants of LSTD (Boyan, 1999), which model the Q-function as a kernel-machine but do not take into account the regularization from gradient descent, as done in Equation 1, which is essential for implicit under-parameterization.
Revised: (2005; 2007) present variants of LSTD (Boyan, 1999), which model the Q-function as a kernel-machine but do not take into account the regularization from gradient descent, as done in Equation 1, which is essential for implicit under-parameterization.

Original: Igl et al.
Revised: Igl et al.

Original: (2020); Fedus et al.
Revised: (2020); Fedus et al.

Original: (2020a) argue that non-stationarity arising from distribution shift hinders generalization and recommend periodic network re-initialization.
Revised: (2020a) argue that non-stationarity arising from distribution shift hinders generalization and recommend periodic network re-initialization.

Original: Under-parameterization is not caused by this distribution shift, and we ﬁnd that network re-initialization does little to prevent rank collapse (Figure 3c).
Revised: Under-parameterization is not caused by this distribution shift, and we ﬁnd that network re-initialization does little to prevent rank collapse (Figure 3c).

Original: Luo et al.
Revised: Luo et al.

Original: (2020) proposes a regularization similar to ours, but in a different setting, ﬁnding that more expressive features increases performance of on-policy RL methods.
Revised: (2020) proposes a regularization similar to ours, but in a different setting, ﬁnding that more expressive features increases performance of on-policy RL methods.

Original: Finally, Yang et al.
Revised: Finally, Yang et al.

Original: (2019) study the effective rank of the Q ∗ -values when expressed as a |S| × |A| matrix in online RL and ﬁnd that low ranks for this Q ∗ -matrix are preferable.
Revised: (2019) study the effective rank of the Q ∗ -values when expressed as a |S| × |A| matrix in online RL and ﬁnd that low ranks for this Q ∗ -matrix are preferable.

Original: We analyze a fundamentally different object: the learned features (and illustrate that a rank-collapse of features can hurt), not the Q ∗ -matrix, whose rank is upper-bounded by the number of actions ( e.g. , 24 for Atari).
Revised: We analyze a fundamentally different object: the learned features (and illustrate that a rank-collapse of features can hurt), not the Q ∗ -matrix, whose rank is upper-bounded by the number of actions ( e.g. , 24 for Atari).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We identiﬁed an implicit under-parameterization phenomenon in deep RL algorithms that use bootstrapping, where gradient-based optimization of a bootstrapped objective can lead to a reduction in the expressive power of the value network.
Revised: We identiﬁed an implicit under-parameterization phenomenon in deep RL algorithms that use bootstrapping, where gradient-based optimization of a bootstrapped objective can lead to a reduction in the expressive power of the value network.

Original: This effect manifests as a collapse of the rank of the features learned by the value network, causing aliasing across states and often leading to poor performance.
Revised: This effect manifests as a collapse of the rank of the features learned by the value network, causing aliasing across states and often leading to poor performance.

Original: Our analysis reveals that this phenomenon is caused by the implicit regularization due to gradient descent on bootstrapped objectives.
Revised: Our analysis reveals that this phenomenon is caused by the implicit regularization due to gradient descent on bootstrapped objectives.

Original: We observed that mitigating this problem by means of a simple regularization scheme improves performance of deep Q-learning methods.
Revised: We observed that mitigating this problem by means of a simple regularization scheme improves performance of deep Q-learning methods.

Original: While our proposed regularization provides some improvement, devising better mitigation strategies for implicit under-parameterization remains an exciting direction for future work.
Revised: While our proposed regularization provides some improvement, devising better mitigation strategies for implicit under-parameterization remains an exciting direction for future work.

Original: Our method explicitly attempts to prevent rank collapse, but relies on the emergence of useful features solely through the bootstrapped signal.
Revised: Our method explicitly attempts to prevent rank collapse, but relies on the emergence of useful features solely through the bootstrapped signal.

Original: An alternative path may be to develop new auxiliary losses ( e.g. ,
Revised: An alternative path may be to develop new auxiliary losses ( e.g. ,

Original: Jaderberg et al., 2016) that learn useful features while passively preventing underparameterization.
Revised: Jaderberg et al., 2016) that learn useful features while passively preventing underparameterization.

Original: More broadly, understanding the effects of neural nets and associated factors such as initialization, choice of optimizer, etc.
Revised: More broadly, understanding the effects of neural nets and associated factors such as initialization, choice of optimizer, etc.

Original: on the learning dynamics of deep RL algorithms, using tools from deep learning theory, is likely to be key towards developing robust and data-efﬁcient deep RL algorithms.
Revised: on the learning dynamics of deep RL algorithms, using tools from deep learning theory, is likely to be key towards developing robust and data-efﬁcient deep RL algorithms.

Original: Appendices A A DDITIONAL E VIDENCE FOR I MPLICIT U NDER -P ARAMETERIZATION
Revised: Appendices A A DDITIONAL E VIDENCE FOR I MPLICIT U NDER -P ARAMETERIZATION

Original: In this section, we present additional evidence that demonstrates the existence of the implicit underparameterization phenomenon from Section 3.
Revised: In this section, we present additional evidence that demonstrates the existence of the implicit underparameterization phenomenon from Section 3.

Original: In all cases, we plot the values of srank δ (Φ) computed on a batch size of 

Revised: In all cases, we plot the values of srank δ (Φ) computed on a batch size of

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d. sampled transitions from the dataset.
Revised: d. sampled transitions from the dataset.

Original: A. 
Revised: A.

Original: [Figure]
Revised: [Figure]

Original: v e r a g e
Revised: v e r a g e

Original: R e t u r
Revised: R e t u r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. D ATA E FFICIENT O NLINE RL
Revised: A. D ATA E FFICIENT O NLINE RL

Original: In the data-efﬁcient online RL setting
Revised: In the data-efﬁcient online RL setting

Original: [Figure]
Revised: [Figure]

Original: we verify the presence of implicit under-parameterization on both DQN and Rainbow (Hessel et al., 2018) algorithms when larger number of gradient updates are made per environment step.
Revised: we verify the presence of implicit under-parameterization on both DQN and Rainbow (Hessel et al., 2018) algorithms when larger number of gradient updates are made per environment step.

Original: In these settings we ﬁnd that more gradient updates per environment step lead to a larger decrease in effective rank
Revised: In these settings we ﬁnd that more gradient updates per environment step lead to a larger decrease in effective rank

Original: [Figure]
Revised: [Figure]

Original: whereas effective rank can increase when the amount of data re-use is reduced by taking fewer gradient steps
Revised: whereas effective rank can increase when the amount of data re-use is reduced by taking fewer gradient steps

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e R e t u r n
Revised: A v e r a g e R e t u r n

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. D OES B OOTSTRAPPING C AUSE I MPLICIT U NDER -P ARAMETERIZATION ?
Revised: A. D OES B OOTSTRAPPING C AUSE I MPLICIT U NDER -P ARAMETERIZATION ?

Original: In this section, we provide additional evidence to support our claim from Section 3 that suggests that bootstrapping-based updates are a key component behind the existence of implicit underparameterization.
Revised: In this section, we provide additional evidence to support our claim from Section 3 that suggests that bootstrapping-based updates are a key component behind the existence of implicit underparameterization.

Original: To do so, we empirically demonstrate the following points empirically:
Revised: To do so, we empirically demonstrate the following points empirically:

Original: • Implicit under-parameterization occurs even when the form of the bootstrapping update is changed from Q-learning that utilizes a max a (cid:48) backup operator to a policy evaluation (ﬁtted Q-evaluation) backup operator, that computes an expectation of the target Q-values under the distributions speciﬁed by a different policy.
Revised: • Implicit under-parameterization occurs even when the form of the bootstrapping update is changed from Q-learning that utilizes a max a (cid:48) backup operator to a policy evaluation (ﬁtted Q-evaluation) backup operator, that computes an expectation of the target Q-values under the distributions speciﬁed by a different policy.

Original: Thus, with different bootstrapped updates, the phenomenon still appears.
Revised: Thus, with different bootstrapped updates, the phenomenon still appears.

Original: [Figure]
Revised: [Figure]

Original: • Implicit under-parameterization does not occur when Monte-Carlo regression targets - that compute regression targets for the Q-function by computing a non-parameteric estimate the future trajectory return, i.e. , , y k
Revised: • Implicit under-parameterization does not occur when Monte-Carlo regression targets - that compute regression targets for the Q-function by computing a non-parameteric estimate the future trajectory return, i.e. , , y k

Original: ( s t , a t ) = (cid:80) ∞
Revised: ( s t , a t ) = (cid:80) ∞

Original: t (cid:48)
Revised: t (cid:48)

Original: =
Revised: =

Original: t γ t r ( s t (cid:48) , a t (cid:48) ) and do not use bootstrapping.
Revised: t γ t r ( s t (cid:48) , a t (cid:48) ) and do not use bootstrapping.

Original: In this setting, we ﬁnd that
Revised: In this setting, we ﬁnd that

Original: the values of effective rank actually increase over time and stabilize, unlike the corresponding case for bootstrapped updates.
Revised: the values of effective rank actually increase over time and stabilize, unlike the corresponding case for bootstrapped updates.

Original: Thus, other factors kept identically the same, implicit under-parameterization happens only when bootstrapped updates are used.
Revised: Thus, other factors kept identically the same, implicit under-parameterization happens only when bootstrapped updates are used.

Original: [Figure]
Revised: [Figure]

Original: A. H OW D OES I MPLICIT R EGULARIZATION I NHIBIT D ATA -E FFICIENT RL?
Revised: A. H OW D OES I MPLICIT R EGULARIZATION I NHIBIT D ATA -E FFICIENT RL?

Original: [Figure]
Revised: [Figure]

Original: Implicit under-parameterization leads to a trade-off between minimizing the TD error vs .
Revised: Implicit under-parameterization leads to a trade-off between minimizing the TD error vs .

Original: encouraging low rank features as shown in Figure 3b.
Revised: encouraging low rank features as shown in Figure 3b.

Original: This trade-off often results in decrease in effective rank, at the expense of increase in TD error, resulting in lower performance.
Revised: This trade-off often results in decrease in effective rank, at the expense of increase in TD error, resulting in lower performance.

Original: Here we present additional evidence to support this.
Revised: Here we present additional evidence to support this.

Original: Figure A.11 shows a gridworld problem with one-hot features, which naturally leads to reduced state-aliasing.
Revised: Figure A.11 shows a gridworld problem with one-hot features, which naturally leads to reduced state-aliasing.

Original: In this setting, we ﬁnd that the amount of rank drop with respect to the supervised projection of oracle computed Q ∗ values is quite small and the regression error to Q ∗ actually decreases unlike the case in Figure 3a, where it remains same or even increases.
Revised: In this setting, we ﬁnd that the amount of rank drop with respect to the supervised projection of oracle computed Q ∗ values is quite small and the regression error to Q ∗ actually decreases unlike the case in Figure 3a, where it remains same or even increases.

Original: The method is able to learn policies that attain good performance as well.
Revised: The method is able to learn policies that attain good performance as well.

Original: Hence, this justiﬁes that when there’s very little rank drop, for example, 5 rank units in the example on the right, FQI methods are generally able to learn Φ that is able to ﬁt Q ∗ .
Revised: Hence, this justiﬁes that when there’s very little rank drop, for example, 5 rank units in the example on the right, FQI methods are generally able to learn Φ that is able to ﬁt Q ∗ .

Original: This provides evidence showing that typical Q-networks learn Φ that can ﬁt the optimal
Revised: This provides evidence showing that typical Q-networks learn Φ that can ﬁt the optimal

Original: Q-function when rank collapse does not occur.
Revised: Q-function when rank collapse does not occur.

Original: In Atari, we do not have access to Q ∗ , and so we instead measure the error in ﬁtting the target value estimates, R + γP π Q k .
Revised: In Atari, we do not have access to Q ∗ , and so we instead measure the error in ﬁtting the target value estimates, R + γP π Q k .

Original: As rank de- creases, the TD error increases (Figure A.12) and the value function is unable to ﬁt the target values, culminating in a performance plateau (Figure A.6).
Revised: As rank de- creases, the TD error increases (Figure A.12) and the value function is unable to ﬁt the target values, culminating in a performance plateau (Figure A.6).

Original: [Figure]
Revised: [Figure]

Original: A. T RENDS IN V ALUES OF E FFECTIVE R ANK W ITH P ENALTY .
Revised: A. T RENDS IN V ALUES OF E FFECTIVE R ANK W ITH P ENALTY .

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we present the trend in the values of the effective rank when the penalty L p (Φ) is added.
Revised: we present the trend in the values of the effective rank when the penalty L p (Φ) is added.

Original: In each plot below, we present the value of srank δ (Φ) with and without penalty respectively.
Revised: In each plot below, we present the value of srank δ (Φ) with and without penalty respectively.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: P o n g
Revised: P o n g

Original: E n d u r o Z a
Revised: E n d u r o Z a

Original: [Figure]
Revised: [Figure]

Original: o n Q b e r
Revised: o n Q b e r

Original: [Figure]
Revised: [Figure]

Original: Y a r s R e v e n g e
Revised: Y a r s R e v e n g e

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: A s t e r i x
Revised: A s t e r i x

Original: M s P a c m a n
Revised: M s P a c m a n

Original: B e a m
Revised: B e a m

Original: R i d e r
Revised: R i d e r

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: S e a q u e s t
Revised: S e a q u e s t

Original: D o u b l e
Revised: D o u b l e

Original: D un k
Revised: D un k

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: D e m o n A tt a c
Revised: D e m o n A tt a c

Original: [Figure]
Revised: [Figure]

Original: J
Revised: J

Original: [Figure]
Revised: [Figure]

Original: m e s
Revised: m e s

Original: [Figure]
Revised: [Figure]

Original: o n d
Revised: o n d

Original: D o u b l e
Revised: D o u b l e

Original: D un k Q b e r t
Revised: D un k Q b e r t

Original: E n d u r o
Revised: E n d u r o

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: R o a d R unn e r P o n g
Revised: R o a d R unn e r P o n g

Original: Y a r s R e v e n g e
Revised: Y a r s R e v e n g e

Original: M s P a c m a n
Revised: M s P a c m a n

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: A s t e r i x
Revised: A s t e r i x

Original: S e a q u e s t
Revised: S e a q u e s t

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: Z a
Revised: Z a

Original: [Figure]
Revised: [Figure]

Original: o n
Revised: o n

Original: B e a m
Revised: B e a m

Original: R i d e r
Revised: R i d e r

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) ofﬂine with the on dataset, i e., the dataset referred to as 4x large in Figure
Revised: ) ofﬂine with the on dataset, i e., the dataset referred to as 4x large in Figure

Original: A.5. O FFLINE RL: CQL W ITH L p (Φ) P ENALTY
Revised: A.5. O FFLINE RL: CQL W ITH L p (Φ) P ENALTY

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A.6. R AINBOW W ITH L p (Φ) P ENALTY : P ERFORMANCE
Revised: A.6. R AINBOW W ITH L p (Φ) P ENALTY : P ERFORMANCE

Original: D e m o n
Revised: D e m o n

Original: A tt a c k
Revised: A tt a c k

Original: B e a m R i
Revised: B e a m R i

Original: d
Revised: d

Original: e r
Revised: e r

Original: Y a r s ' s R e v e n g e P o n g
Revised: Y a r s ' s R e v e n g e P o n g

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: J a m e s B o n d Q * B e r t
Revised: J a m e s B o n d Q * B e r t

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r A s t e r i x
Revised: O f W o r A s t e r i x

Original: M s
Revised: M s

Original: [Figure]
Revised: [Figure]

Original: P a c - M a n
Revised: P a c - M a n

Original: Z a xx o n E n d u r o
Revised: Z a xx o n E n d u r o

Original: [Figure]
Revised: [Figure]

Original: a c e I n v a d e r s
Revised: a c e I n v a d e r s

Original: S e
Revised: S e

Original: [Figure]
Revised: [Figure]

Original: q u e s t
Revised: q u e s t

Original: I c e
Revised: I c e

Original: [Figure]
Revised: [Figure]

Original: c k e y
Revised: c k e y

Original: [Figure]
Revised: [Figure]

Original: t (
Revised: t (

Original: [Figure]
Revised: [Figure]

Original: In this section, we present additional results for supporting the hypothesis that preventing rank-collapse leads to better performance.
Revised: In this section, we present additional results for supporting the hypothesis that preventing rank-collapse leads to better performance.

Original: In the ﬁrst set of experiments, we apply the proposed L p penalty to Rainbow in the data-efﬁcient online RL setting ( n = 4) .
Revised: In the ﬁrst set of experiments, we apply the proposed L p penalty to Rainbow in the data-efﬁcient online RL setting ( n = 4) .

Original: In the second set of experiments, we present evidence for prevention of rank collapse by comparing rank values for different runs.
Revised: In the second set of experiments, we present evidence for prevention of rank collapse by comparing rank values for different runs.

Original: As we will show in the next section, the state-of-the-art Rainbow (Hessel et al., 2018) algorithm also suffers form rank collapse in the data-efﬁcient online RL setting when more updates are performed per gradient step.
Revised: As we will show in the next section, the state-of-the-art Rainbow (Hessel et al., 2018) algorithm also suffers form rank collapse in the data-efﬁcient online RL setting when more updates are performed per gradient step.

Original: In this section, we applied our penalty L p to Rainbow with n =
Revised: In this section, we applied our penalty L p to Rainbow with n =

Original: [Figure]
Revised: [Figure]

Original: and obtained a median 20.66% improvement on top of the base method.
Revised: and obtained a median 20.66% improvement on top of the base method.

Original: This result is summarized below
Revised: This result is summarized below

Original: [Figure]
Revised: [Figure]

Original: A. E FFECTIVE R ANK C OMPUTED U SING E IGENVALUE N ORMS
Revised: A. E FFECTIVE R ANK C OMPUTED U SING E IGENVALUE N ORMS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we provide empirical evidence showing that the trend in the values of effective rank computed using singular values, srank δ (Φ) is almost identical to the trend in the effective rank computed using normalized eigenvalues, srank δ,λ (Φ) .
Revised: we provide empirical evidence showing that the trend in the values of effective rank computed using singular values, srank δ (Φ) is almost identical to the trend in the effective rank computed using normalized eigenvalues, srank δ,λ (Φ) .

Original: Since eigenvalues are only deﬁned for a square matrix Φ , in practice
Revised: Since eigenvalues are only deﬁned for a square matrix Φ , in practice

Original: [Figure]
Revised: [Figure]

Original: we use a batch of d = dim ( φ ( s , a )) state-action pairs for computing the eigenvalue rank and compare to the corresponding singular value rank in Figures A.20 and A.
Revised: we use a batch of d = dim ( φ ( s , a )) state-action pairs for computing the eigenvalue rank and compare to the corresponding singular value rank in Figures A.20 and A.

Original: [Figure]
Revised: [Figure]

Original: Connection to Theorem 4.1.
Revised: Connection to Theorem 4.1.

Original: We computed the effective rank of Φ instead of S , since S is a theoretical abstraction that cannot be computed in practice as it depends on the Green’s kernel (Duffy, 2015) obtained by assuming that the neural network behaves as a kernel regressor.
Revised: We computed the effective rank of Φ instead of S , since S is a theoretical abstraction that cannot be computed in practice as it depends on the Green’s kernel (Duffy, 2015) obtained by assuming that the neural network behaves as a kernel regressor.

Original: Instead, we compare the different notions of ranks of Φ since Φ is the practical counterpart for the matrix, S , when using neural networks (as also indicated by the analysis in Section 4.2).
Revised: Instead, we compare the different notions of ranks of Φ since Φ is the practical counterpart for the matrix, S , when using neural networks (as also indicated by the analysis in Section 4.2).

Original: In fact, on the gridworld (Figure A.21), we experiment with a feature Φ with dimension equal to the number of state-action pairs, i.e. , dim ( φ ( s , a )) =
Revised: In fact, on the gridworld (Figure A.21), we experiment with a feature Φ with dimension equal to the number of state-action pairs, i.e. , dim ( φ ( s , a )) =

Original: |S||A| , with the same number of parameters as a kernel parameterization of the Q-function: Q ( s , a ) =
Revised: |S||A| , with the same number of parameters as a kernel parameterization of the Q-function: Q ( s , a ) =

Original: (cid:80) s (cid:48) , a (cid:48)
Revised: (cid:80) s (cid:48) , a (cid:48)

Original: w ( s (cid:48) , a (cid:48) )
Revised: w ( s (cid:48) , a (cid:48) )

Original: k ( s , a , s (cid:48) , a (cid:48) ) .
Revised: k ( s , a , s (cid:48) , a (cid:48) ) .

Original: This can also be considered as performing gradient descent on a “wide” linear network , and we measure the feature rank while observing similar rank trends.
Revised: This can also be considered as performing gradient descent on a “wide” linear network , and we measure the feature rank while observing similar rank trends.

Original: Since we do not require the assumption that S is normal in Theorem 4.1 to obtain a decreasing trend in srank δ,λ (Φ) , and we ﬁnd that in practical scenarios (Figures A.20 and A.21), srank δ (Φ) ≈ srank δ,λ (Φ) with an extremely similar qualitative trend we believe that Theorem 4.1 still explains the rank-collapse practically observed in deep Q-learning and is not vacuous.
Revised: Since we do not require the assumption that S is normal in Theorem 4.1 to obtain a decreasing trend in srank δ,λ (Φ) , and we ﬁnd that in practical scenarios (Figures A.20 and A.21), srank δ (Φ) ≈ srank δ,λ (Φ) with an extremely similar qualitative trend we believe that Theorem 4.1 still explains the rank-collapse practically observed in deep Q-learning and is not vacuous.

Original: [Figure]
Revised: [Figure]

Original: A. N ORMALIZED P LOTS FOR F IGURE 2/ F IGURE A. 
Revised: A. N ORMALIZED P LOTS FOR F IGURE 2/ F IGURE A.

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we provide a set of normalized srank and performance trends for Atari games (the corresponding unnormalized plots are found in Figure A.6).
Revised: we provide a set of normalized srank and performance trends for Atari games (the corresponding unnormalized plots are found in Figure A.6).

Original: In these plots
Revised: In these plots

Original: [Figure]
Revised: [Figure]

Original: each unit on the x-axis is equivalent to one gradient update
Revised: each unit on the x-axis is equivalent to one gradient update

Original: [Figure]
Revised: [Figure]

Original: and so since n = 8 prescribes 8 × many updates as compared to n = 1 , it it runs for 8 × as long as n = 1 .
Revised: and so since n = 8 prescribes 8 × many updates as compared to n = 1 , it it runs for 8 × as long as n = 1 .

Original: These plots are in Figure A.
Revised: These plots are in Figure A.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Note that the trend that effective rank decreases with larger n values also persists when rescaling the x-axis to account for the number of gradient steps
Revised: Note that the trend that effective rank decreases with larger n values also persists when rescaling the x-axis to account for the number of gradient steps

Original: [Figure]
Revised: [Figure]

Original: in all but one game.
Revised: in all but one game.

Original: This is expected since it tells us that performing bootstrapping based updates in the data-efﬁcient setting (larger n values) still leads to more aggressive rank drop as updates are being performed on a relatively more static dataset for larger values of n .
Revised: This is expected since it tells us that performing bootstrapping based updates in the data-efﬁcient setting (larger n values) still leads to more aggressive rank drop as updates are being performed on a relatively more static dataset for larger values of n .

Original: [Table]
Revised: [Table]

Original: B H YPERPARAMETERS & E XPERIMENT D ETAILS B. A TARI E XPERIMENTS
Revised: B H YPERPARAMETERS & E XPERIMENT D ETAILS B. A TARI E XPERIMENTS

Original: We follow the experiment protocol from Agarwal et al.
Revised: We follow the experiment protocol from Agarwal et al.

Original: (2020) for all our experiments including hyperparameters and agent architectures provided in Dopamine
Revised: (2020) for all our experiments including hyperparameters and agent architectures provided in Dopamine

Original: and report them for completeness and ease of reproducibility in Table B.1.
Revised: and report them for completeness and ease of reproducibility in Table B.1.

Original: We only use hyperparameter selection over the regularization experiment α p based on results from 5 Atari games (Asterix, Seaquest, Pong, Breakout and
Revised: We only use hyperparameter selection over the regularization experiment α p based on results from 5 Atari games (Asterix, Seaquest, Pong, Breakout and

Original: Seaquest).
Revised: Seaquest).

Original: We will also open source our code to further aid in reproducing our results
Revised: We will also open source our code to further aid in reproducing our results

Original: [Table]
Revised: [Table]

Original: Evaluation Protocol .
Revised: Evaluation Protocol .

Original: Following Agarwal et al.
Revised: Following Agarwal et al.

Original: (2020), the Atari environments used in our experiments are stochastic due to sticky actions, i.e. , there is 25% chance at every time step that the environment will execute the agent’s previous action again, instead of the agent’s new action.
Revised: (2020), the Atari environments used in our experiments are stochastic due to sticky actions, i.e. , there is 25% chance at every time step that the environment will execute the agent’s previous action again, instead of the agent’s new action.

Original: All agents (online or ofﬂine) are compared using the best evaluation score (averaged over 5 runs) achieved during training where the evaluation is done online every training iteration using a (cid:15) -greedy policy with (cid:15)
Revised: All agents (online or ofﬂine) are compared using the best evaluation score (averaged over 5 runs) achieved during training where the evaluation is done online every training iteration using a (cid:15) -greedy policy with (cid:15)

Original: = 0 .
Revised: = 0 .

Original: We report ofﬂine training results with same hyperparameters over 5 random seeds of the DQN replay data collection, game simulator and network initialization.
Revised: We report ofﬂine training results with same hyperparameters over 5 random seeds of the DQN replay data collection, game simulator and network initialization.

Original: Ofﬂine Dataset .
Revised: Ofﬂine Dataset .

Original: As suggested by Agarwal et al.
Revised: As suggested by Agarwal et al.

Original: (2020), we randomly subsample the DQN Replay dataset containing 50 millions transitions to create smaller ofﬂine datasets with the same data distribution as the original dataset.
Revised: (2020), we randomly subsample the DQN Replay dataset containing 50 millions transitions to create smaller ofﬂine datasets with the same data distribution as the original dataset.

Original: We use the 5% DQN replay dataset for most of our experiments.
Revised: We use the 5% DQN replay dataset for most of our experiments.

Original: We also report results using the 20% dataset setting (4x larger) to show that our claims are also valid even when we have higher coverage over the state space.
Revised: We also report results using the 20% dataset setting (4x larger) to show that our claims are also valid even when we have higher coverage over the state space.

Original: Optimizer related hyperparameters .
Revised: Optimizer related hyperparameters .

Original: For existing off-policy agents, step size and optimizer were taken as published.
Revised: For existing off-policy agents, step size and optimizer were taken as published.

Original: We used the DQN (Adam) algorithm for all our experiments, given its superior performance over the DQN (Nature) which uses RMSProp, as reported by Agarwal et al.
Revised: We used the DQN (Adam) algorithm for all our experiments, given its superior performance over the DQN (Nature) which uses RMSProp, as reported by Agarwal et al.

Original: Atari 2600 games used .
Revised: Atari 2600 games used .

Original: For all our experiments in Section 3, we used the same set of 5 games as utilized by Agarwal et al.
Revised: For all our experiments in Section 3, we used the same set of 5 games as utilized by Agarwal et al.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2017) to present analytical results.
Revised: (2017) to present analytical results.

Original: For our empirical evaluation in Appendix A.5, we use the set of games employed by Fedus et al.
Revised: For our empirical evaluation in Appendix A.5, we use the set of games employed by Fedus et al.

Original: (2020b) which are deemed suitable for ofﬂine RL by Gulcehre et al.
Revised: (2020b) which are deemed suitable for ofﬂine RL by Gulcehre et al.

Original: Similar in spirit to Gulcehre et al.
Revised: Similar in spirit to Gulcehre et al.

Original: (2020), we use the set of 5 games used for analysis for hyperparameter tuning for ofﬂine RL methods.
Revised: (2020), we use the set of 5 games used for analysis for hyperparameter tuning for ofﬂine RL methods.

Original: 5 games subset : A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT 16 game
Revised: 5 games subset : A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT 16 game

Original: subset : In addition to 5 games above, the following 11 games
Revised: subset : In addition to 5 games above, the following 11 games

Original: G
Revised: G

Original: RAVITAR ,
Revised: RAVITAR ,

Original: J
Revised: J

Original: AMES
Revised: AMES

Original: B OND
Revised: B OND

Original: , M S . P ACMAN , S PACE I NVADERS , Z AXXON , W IZARD OF W OR , Y ARS ’ R EVENGE , E NDURO , R OAD R UNNER , B EAM R IDER , D EMON A TTACK B. G RIDWORLD E XPERIMENTS
Revised: , M S . P ACMAN , S PACE I NVADERS , Z AXXON , W IZARD OF W OR , Y ARS ’ R EVENGE , E NDURO , R OAD R UNNER , B EAM R IDER , D EMON A TTACK B. G RIDWORLD E XPERIMENTS

Original: We use the gridworld suite from Fu et al.
Revised: We use the gridworld suite from Fu et al.

Original: (2019) to obtain gridworlds for our experiments.
Revised: (2019) to obtain gridworlds for our experiments.

Original: All of our gridworld results are computed using the 16 × 16 G RID 16 SMOOTHOBS environment, which consists of a 256-cell grid, with walls arising randomly with a probability of 0.2.
Revised: All of our gridworld results are computed using the 16 × 16 G RID 16 SMOOTHOBS environment, which consists of a 256-cell grid, with walls arising randomly with a probability of 0.2.

Original: Each state allowsdifferent actions (subject to hitting the boundary of the grid): move left, move right, move up, move down and no op.
Revised: Each state allowsdifferent actions (subject to hitting the boundary of the grid): move left, move right, move up, move down and no op.

Original: The goal in this environment is to minimize the cumulative discounted distance to a ﬁxed goal location where the discount factor is given by γ = 0 .
Revised: The goal in this environment is to minimize the cumulative discounted distance to a ﬁxed goal location where the discount factor is given by γ = 0 .

Original: The features for this Q-function are given by randomly chosen vectors which are smoothened spatially in a local neighborhood of a grid cell (
Revised: The features for this Q-function are given by randomly chosen vectors which are smoothened spatially in a local neighborhood of a grid cell (

Original: x, y )
Revised: x, y )

Original: We use a deep Q-network with two hidden layers of size (64 , 64) , and train it using soft Q-learning with entropy coefﬁcient of 0.1, following the code provided by authors of Fu et al.
Revised: We use a deep Q-network with two hidden layers of size (64 , 64) , and train it using soft Q-learning with entropy coefﬁcient of 0.1, following the code provided by authors of Fu et al.

Original: We use a ﬁrst-in-ﬁrst out replay buffer of size 10000 to store past transitions.
Revised: We use a ﬁrst-in-ﬁrst out replay buffer of size 10000 to store past transitions.

Original: C P ROOFS FOR S ECTION 4. 
Revised: C P ROOFS FOR S ECTION 4.

Original: In this section, we provide the technical proofs from Section 4.1.
Revised: In this section, we provide the technical proofs from Section 4.1.

Original: We ﬁrst derive a solution to optimization problem Equation 1 and show that it indeed comes out to have the form described in
Revised: We ﬁrst derive a solution to optimization problem Equation 1 and show that it indeed comes out to have the form described in

Original: Equation 2.
Revised: Equation 2.

Original: We ﬁrst introduce some notation, including deﬁnition of the kernel G which was used for this proof.
Revised: We ﬁrst introduce some notation, including deﬁnition of the kernel G which was used for this proof.

Original: This proof closely follows the proof from Mobahi et al.
Revised: This proof closely follows the proof from Mobahi et al.

Original: Deﬁnitions.
Revised: Deﬁnitions.

Original: For any universal kernel u , the Green’s function (Duffy, 2015) of the linear kernel operator L given by: [ L Q ] ( s , a ) := (cid:80) ( s (cid:48) , a (cid:48) )
Revised: For any universal kernel u , the Green’s function (Duffy, 2015) of the linear kernel operator L given by: [ L Q ] ( s , a ) := (cid:80) ( s (cid:48) , a (cid:48) )

Original: u (( s , a ) , ( s (cid:48) , a (cid:48) ))
Revised: u (( s , a ) , ( s (cid:48) , a (cid:48) ))

Original: Q ( s (cid:48) , a (cid:48) ) is given by the function g (( s , a ) , ( s (cid:48) , a (cid:48) ))
Revised: Q ( s (cid:48) , a (cid:48) ) is given by the function g (( s , a ) , ( s (cid:48) , a (cid:48) ))

Original: that satisﬁes:
Revised: that satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where δ is the Dirac-delta function.
Revised: where δ is the Dirac-delta function.

Original: Thus, Green’s function can be understood as a kernel that “inverts” the universal kernel u to the idehtity (Dirac-delta) matrix.
Revised: Thus, Green’s function can be understood as a kernel that “inverts” the universal kernel u to the idehtity (Dirac-delta) matrix.

Original: We can then deﬁne the matrix G as the matrix of vectors g ( s , a ) evaluated on the training dataset, D , however note that the functional g ( s , a ) can be evaluated for other state-action tuples,
Revised: We can then deﬁne the matrix G as the matrix of vectors g ( s , a ) evaluated on the training dataset, D , however note that the functional g ( s , a ) can be evaluated for other state-action tuples,

Original: not present in D .
Revised: not present in D .

Original: G (( s i , a i ) , ( s j , a j )) :
Revised: G (( s i , a i ) , ( s j , a j )) :

Original: = g
Revised: = g

Original: (( s i , a i ) , ( s j , a j )) and g ( s , a )
Revised: (( s i , a i ) , ( s j , a j )) and g ( s , a )

Original: [ i ] =
Revised: [ i ] =

Original: g
Revised: g

Original: (( s , a ) , ( s i , a i ))
Revised: (( s , a ) , ( s i , a i ))

Original: ∀ ( s i , a i ) ∈ D .
Revised: ∀ ( s i , a i ) ∈ D .

Original: (C.2)
Revised: (C.2)

Original: Lemma C.0.1.
Revised: Lemma C.0.1.

Original: The solution to Equation 1 is given by Equation 2.
Revised: The solution to Equation 1 is given by Equation 2.

Original: Proof.
Revised: Proof.

Original: This proof closely follows the proof of Proposition 1 from (Mobahi et al., 2020).
Revised: This proof closely follows the proof of Proposition 1 from (Mobahi et al., 2020).

Original: We revisit key aspects the key parts of this proof here.
Revised: We revisit key aspects the key parts of this proof here.

Original: We restate the optimization problem below, and solve for the optimum Q k to this equation by applying the functional derivative principle.
Revised: We restate the optimization problem below, and solve for the optimum Q k to this equation by applying the functional derivative principle.

Original: [Equation]
Revised: [Equation]

Original: The functional derivative principle would say that the optimal Q k to this problem would satisfy, for any other function f and for a small enough ε → 0 ,
Revised: The functional derivative principle would say that the optimal Q k to this problem would satisfy, for any other function f and for a small enough ε → 0 ,

Original: [Equation]
Revised: [Equation]

Original: By setting the gradient of the above expression to
Revised: By setting the gradient of the above expression to

Original: [Equation]
Revised: [Equation]

Original: we obtain the following stationarity conditions on Q k (also denoting ( s i , a i ) :
Revised: we obtain the following stationarity conditions on Q k (also denoting ( s i , a i ) :

Original: = x i ) for brevity:
Revised: = x i ) for brevity:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we invoke the deﬁnition of the Green’s function discussed above and utilize the fact that the Dirac-delta function can be expressed in terms of the Green’s function, we obtain a simpliﬁed version of the above relation:
Revised: we invoke the deﬁnition of the Green’s function discussed above and utilize the fact that the Dirac-delta function can be expressed in terms of the Green’s function, we obtain a simpliﬁed version of the above relation:

Original: [Equation]
Revised: [Equation]

Original: Since the kernel u ( x , x (cid:48) ) os universal and positive deﬁnite, the optimal solution Q k ( x ) is given by:
Revised: Since the kernel u ( x , x (cid:48) ) os universal and positive deﬁnite, the optimal solution Q k ( x ) is given by:

Original: [Equation]
Revised: [Equation]

Original: Finally we can replace the expression for residual error, Q k ( s i , a i )
Revised: Finally we can replace the expression for residual error, Q k ( s i , a i )

Original: − y k
Revised: − y k

Original: ( s i , a i ) using the green’s kernel on the training data by solving for it in closed form, which gives us the solution in Equation 2.
Revised: ( s i , a i ) using the green’s kernel on the training data by solving for it in closed form, which gives us the solution in Equation 2.

Original: [Equation]
Revised: [Equation]

Original: Next, we now state and prove a slightly stronger version of Theorem 4.1 that immediately implies the original theorem.
Revised: Next, we now state and prove a slightly stronger version of Theorem 4.1 that immediately implies the original theorem.

Original: Theorem C.1.
Revised: Theorem C.1.

Original: Let S be a shorthand for S =
Revised: Let S be a shorthand for S =

Original: γP
Revised: γP

Original: π A and assume S is a normal matrix.
Revised: π A and assume S is a normal matrix.

Original: Then there exists an inﬁnite, strictly increasing sequence of ﬁtting iterations, ( k l ) ∞ l =1 starting from k 1 = 0 , such that, for any two singular-values σ i ( S ) and σ j ( S ) of S with σ i ( S ) ≤ σ j ( S ) , ∀ l ∈ N and l (cid:48) ≥ l, σ i ( M k l (cid:48) )
Revised: Then there exists an inﬁnite, strictly increasing sequence of ﬁtting iterations, ( k l ) ∞ l =1 starting from k 1 = 0 , such that, for any two singular-values σ i ( S ) and σ j ( S ) of S with σ i ( S ) ≤ σ j ( S ) , ∀ l ∈ N and l (cid:48) ≥ l, σ i ( M k l (cid:48) )

Original: σ j ( M k l (cid:48) ) <
Revised: σ j ( M k l (cid:48) ) <

Original: σ i ( M k l
Revised: σ i ( M k l

Original: [Equation]
Revised: [Equation]

Original: σ j ( M k l ) ≤ σ i ( S
Revised: σ j ( M k l ) ≤ σ i ( S

Original: [Equation]
Revised: [Equation]

Original: σ j ( S ) .
Revised: σ j ( S ) .

Original: (C.8)
Revised: (C.8)

Original: Therefore, the effective rank of M k satisﬁes: srank δ ( M k l (cid:48) ) ≤ srank δ ( M k l ) .
Revised: Therefore, the effective rank of M k satisﬁes: srank δ ( M k l (cid:48) ) ≤ srank δ ( M k l ) .

Original: Furthermore, ∀ l ∈ N and t ≥ k l , σ i ( M t ) σ j ( M t ) <
Revised: Furthermore, ∀ l ∈ N and t ≥ k l , σ i ( M t ) σ j ( M t ) <

Original: σ i ( M k l ) σ j (
Revised: σ i ( M k l ) σ j (

Original: [Equation]
Revised: [Equation]

Original: Therefore, the effective rank of M t , srank δ ( M t ) , outside the chosen subsequence is also controlled above by the effective rank on the subsequence ( srank δ ( M k l ))
Revised: Therefore, the effective rank of M t , srank δ ( M t ) , outside the chosen subsequence is also controlled above by the effective rank on the subsequence ( srank δ ( M k l ))

Original: ∞ l =1 .
Revised: ∞ l =1 .

Original: To prove this theorem, we ﬁrst show that for any two ﬁtting iterations, t < t (cid:48) , if S t and S t (cid:48) are positive semi-deﬁnite, the ratio of singular values and the effective rank decreases from t to t (cid:48) .
Revised: To prove this theorem, we ﬁrst show that for any two ﬁtting iterations, t < t (cid:48) , if S t and S t (cid:48) are positive semi-deﬁnite, the ratio of singular values and the effective rank decreases from t to t (cid:48) .

Original: As an immediate consequence, this shows that when S is positive semi-deﬁnite, the effective rank decreases at every iteration, i.e. , by setting k l = l (Corollary C.1.1).
Revised: As an immediate consequence, this shows that when S is positive semi-deﬁnite, the effective rank decreases at every iteration, i.e. , by setting k l = l (Corollary C.1.1).

Original: To extend the proof to arbitrary normal matrices, we show that for any S , a sequence of ﬁtting iterations ( k l ) ∞ l =1 can be chosen such that S k l is (approximately) positive semi-deﬁnite.
Revised: To extend the proof to arbitrary normal matrices, we show that for any S , a sequence of ﬁtting iterations ( k l ) ∞ l =1 can be chosen such that S k l is (approximately) positive semi-deﬁnite.

Original: For this subsequence of ﬁtting iterations, the ratio of singular values and effective rank also decreases.
Revised: For this subsequence of ﬁtting iterations, the ratio of singular values and effective rank also decreases.

Original: Finally, to control the ratio and effective rank on ﬁtting iterations t outside this subsequence, we construct an upper bound on the ratio f ( t ) : σ i ( M t ) σ j ( M t ) < f ( t ) , and relate this bound to the ratio of singular values on the chosen subsequence.
Revised: Finally, to control the ratio and effective rank on ﬁtting iterations t outside this subsequence, we construct an upper bound on the ratio f ( t ) : σ i ( M t ) σ j ( M t ) < f ( t ) , and relate this bound to the ratio of singular values on the chosen subsequence.

Original: Lemma C.1.1 (srank δ ( M k ) decreases when S k is PSD.) .
Revised: Lemma C.1.1 (srank δ ( M k ) decreases when S k is PSD.) .

Original: Let S be a shorthand for S =
Revised: Let S be a shorthand for S =

Original: γP π A
Revised: γP π A

Original: and assume S is a normal matrix.
Revised: and assume S is a normal matrix.

Original: Choose any t, t (cid:48) ∈ N such that t < t (cid:48) .
Revised: Choose any t, t (cid:48) ∈ N such that t < t (cid:48) .

Original: If S t and S t (cid:48) are positive semi-deﬁnite, then for any two singular-values σ i ( S ) and σ j ( S ) of S , such that 0 < σ i ( S ) <
Revised: If S t and S t (cid:48) are positive semi-deﬁnite, then for any two singular-values σ i ( S ) and σ j ( S ) of S , such that 0 < σ i ( S ) <

Original: σ j ( S ) :
Revised: σ j ( S ) :

Original: [Equation]
Revised: [Equation]

Original: i ( M t (cid:48)
Revised: i ( M t (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Hence, the effective rank of M k decreases from t to t (cid:48) : srank δ ( M t (cid:48) )
Revised: Hence, the effective rank of M k decreases from t to t (cid:48) : srank δ ( M t (cid:48) )

Original: ≤ srank δ ( M t ) .
Revised: ≤ srank δ ( M t ) .

Original: Proof.
Revised: Proof.

Original: First note that M k is given by:
Revised: First note that M k is given by:

Original: [Equation]
Revised: [Equation]

Original: From hereon, we omit the leading γ term since it is a constant scaling factor that does not affect ratio or effective rank.
Revised: From hereon, we omit the leading γ term since it is a constant scaling factor that does not affect ratio or effective rank.

Original: Almost every matrix S admits a complex orthogonal eigendecomposition.
Revised: Almost every matrix S admits a complex orthogonal eigendecomposition.

Original: Thus, we can write S :
Revised: Thus, we can write S :

Original: = U λ ( S ) U H .
Revised: = U λ ( S ) U H .

Original: And any power of S , i.e. , , S i can be expressed as: S i =
Revised: And any power of S , i.e. , , S i can be expressed as: S i =

Original: U λ ( S )
Revised: U λ ( S )

Original: i U H , and hence, we can express M k as:
Revised: i U H , and hence, we can express M k as:

Original: [Equation]
Revised: [Equation]

Original: Since S is normal, its eigenvalues and singular values are further related as σ k ( S ) =
Revised: Since S is normal, its eigenvalues and singular values are further related as σ k ( S ) =

Original: | λ k ( S ) | .
Revised: | λ k ( S ) | .

Original: And this also means that M k is normal, indicating that σ i ( M k ) =
Revised: And this also means that M k is normal, indicating that σ i ( M k ) =

Original: | λ i ( M k ) | .
Revised: | λ i ( M k ) | .

Original: Thus, the singular values of M k can be expressed as
Revised: Thus, the singular values of M k can be expressed as

Original: [Equation]
Revised: [Equation]

Original: When S k is positive semi-deﬁnite, λ i ( S ) k
Revised: When S k is positive semi-deﬁnite, λ i ( S ) k

Original: = σ i ( S ) k , enabling the following simpliﬁcation:
Revised: = σ i ( S ) k , enabling the following simpliﬁcation:

Original: [Equation]
Revised: [Equation]

Original: To show that the ratio of singular values decreases from t to t (cid:48) , we need to show that f ( σ ) = | 1 − σ t (cid:48) | | 1 − σ t | is an increasing function of σ when t (cid:48) > t .
Revised: To show that the ratio of singular values decreases from t to t (cid:48) , we need to show that f ( σ ) = | 1 − σ t (cid:48) | | 1 − σ t | is an increasing function of σ when t (cid:48) > t .

Original: It can be seen that this is the case, which implies the desired result.
Revised: It can be seen that this is the case, which implies the desired result.

Original: To further show that srank δ ( M t ) ≥ srank δ ( M t (cid:48) ) , we can simply show that ∀ i ∈ [1 , · · · , n ] , h k ( i ) :
Revised: To further show that srank δ ( M t ) ≥ srank δ ( M t (cid:48) ) , we can simply show that ∀ i ∈ [1 , · · · , n ] , h k ( i ) :

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ij =1 σ j ( M k ) (cid:80) nj =1 σ j ( M k ) increases with k , and this would imply that the srank δ ( M k ) cannot increase from k = t to k = t (cid:48) .
Revised: ij =1 σ j ( M k ) (cid:80) nj =1 σ j ( M k ) increases with k , and this would imply that the srank δ ( M k ) cannot increase from k = t to k = t (cid:48) .

Original: We can decompose h k ( i ) as:
Revised: We can decompose h k ( i ) as:

Original: [Equation]
Revised: [Equation]

Original: Since σ j ( M k ) /σ
Revised: Since σ j ( M k ) /σ

Original: l ( M i ) decreases over time k for all j, l if σ j ( S ) ≤ σ l ( S ) , the ratio in the denominator of h k ( i ) decreases with increasing k implying that h k ( i ) increases from t to t (cid:48) .
Revised: l ( M i ) decreases over time k for all j, l if σ j ( S ) ≤ σ l ( S ) , the ratio in the denominator of h k ( i ) decreases with increasing k implying that h k ( i ) increases from t to t (cid:48) .

Original: Corollary C.1.1 (srank δ ( M k ) decreases for PSD S matrices.) .
Revised: Corollary C.1.1 (srank δ ( M k ) decreases for PSD S matrices.) .

Original: Let S be a shorthand for S =
Revised: Let S be a shorthand for S =

Original: γP π A .
Revised: γP π A .

Original: Assuming that S is positive semi-deﬁnite, for any k, t ∈ N , such that t > k and that for any two singular-values σ i ( S ) and σ j ( S ) of S , such that σ i ( S ) <
Revised: Assuming that S is positive semi-deﬁnite, for any k, t ∈ N , such that t > k and that for any two singular-values σ i ( S ) and σ j ( S ) of S , such that σ i ( S ) <

Original: σ j 
Revised: σ j (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, the effective rank of M k decreases with more ﬁtting iterations: srank δ ( M t ) ≤ srank δ ( M k ) .
Revised: Hence, the effective rank of M k decreases with more ﬁtting iterations: srank δ ( M t ) ≤ srank δ ( M k ) .

Original: In order to now extend the result to arbitrary normal matrices, we must construct a subsequence of ﬁtting iterations ( k l ) ∞ l =1 where S k l is (approximately) positive semi-deﬁnite.
Revised: In order to now extend the result to arbitrary normal matrices, we must construct a subsequence of ﬁtting iterations ( k l ) ∞ l =1 where S k l is (approximately) positive semi-deﬁnite.

Original: To do so, we ﬁrst prove a technical lemma that shows that rational numbers, i.e. , numbers that can be expressed as r = pq , for integers p, q ∈ Z are “dense” in the space of real numbers.
Revised: To do so, we ﬁrst prove a technical lemma that shows that rational numbers, i.e. , numbers that can be expressed as r = pq , for integers p, q ∈ Z are “dense” in the space of real numbers.

Original: Lemma C.1.2 (Rational numbers are dense in the real space.) .
Revised: Lemma C.1.2 (Rational numbers are dense in the real space.) .

Original: For any real number α , there exist inﬁnitely many rational numbers pq such that α can be approximated by pq upto 1 q 2 accuracy.
Revised: For any real number α , there exist inﬁnitely many rational numbers pq such that α can be approximated by pq upto 1 q 2 accuracy.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst use Dirichlet’s approximation theorem (see Hlawka et al.
Revised: We ﬁrst use Dirichlet’s approximation theorem (see Hlawka et al.

Original: for a proof of this result using a pigeonhole argument and extensions) to obtain that for any real numbers α and N ≥ 1 , there exist integers p and q such that 1 ≤ q ≤ N and,
Revised: for a proof of this result using a pigeonhole argument and extensions) to obtain that for any real numbers α and N ≥ 1 , there exist integers p and q such that 1 ≤ q ≤ N and,

Original: [Equation]
Revised: [Equation]

Original: Now, since q ≥ 1 > 0 , we can divide both sides by q , to obtain:
Revised: Now, since q ≥ 1 > 0 , we can divide both sides by q , to obtain:

Original: [Equation]
Revised: [Equation]

Original: To obtain inﬁnitely many choices for pq , we observe that Dirichlet’s lemma is valid only for all values of N that satisfy N ≤ 1 | qα − p | .
Revised: To obtain inﬁnitely many choices for pq , we observe that Dirichlet’s lemma is valid only for all values of N that satisfy N ≤ 1 | qα − p | .

Original: Thus if we choose an N (cid:48) such that N (cid:48) ≥ N max where N max is deﬁned as:
Revised: Thus if we choose an N (cid:48) such that N (cid:48) ≥ N max where N max is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Equation C.20 essentially ﬁnds a new value of N , such that the current choices of p and q , which were valid for the ﬁrst value of N do not satisfy the approximation error bound.
Revised: Equation C.20 essentially ﬁnds a new value of N , such that the current choices of p and q , which were valid for the ﬁrst value of N do not satisfy the approximation error bound.

Original: Applying Dirichlet’s lemma to this new value of N (cid:48) hence gives us a new set of p (cid:48) and q (cid:48) which satisfy the 1 q (cid:48) 2 approximation error bound.
Revised: Applying Dirichlet’s lemma to this new value of N (cid:48) hence gives us a new set of p (cid:48) and q (cid:48) which satisfy the 1 q (cid:48) 2 approximation error bound.

Original: Repeating this process gives us countably many choices of ( p, q ) pairs that satisfy the approximation error bound.
Revised: Repeating this process gives us countably many choices of ( p, q ) pairs that satisfy the approximation error bound.

Original: As a result, rational numbers are dense in the space of real numbers, since for any arbitrarily chosen approximation accuracy given by 1 q 2 , we can obtain atleast one rational number, pq which is closer to α than 1 q 2 .
Revised: As a result, rational numbers are dense in the space of real numbers, since for any arbitrarily chosen approximation accuracy given by 1 q 2 , we can obtain atleast one rational number, pq which is closer to α than 1 q 2 .

Original: This proof is based on Johnson (2016).
Revised: This proof is based on Johnson (2016).

Original: Now we utilize Lemmas C.1.1 and C.1.2 to prove Proposition 4.1.
Revised: Now we utilize Lemmas C.1.1 and C.1.2 to prove Proposition 4.1.

Original: Proof of Proposition 4.1 and Theorem C.1 Recall from the proof of Lemma C.1.1 that the singular values of M k are given by:
Revised: Proof of Proposition 4.1 and Theorem C.1 Recall from the proof of Lemma C.1.1 that the singular values of M k are given by:

Original: [Equation]
Revised: [Equation]

Original: Bound on Singular Value Ratio: The ratio between σ i ( M k ) and σ j ( M k ) can be expressed as
Revised: Bound on Singular Value Ratio: The ratio between σ i ( M k ) and σ j ( M k ) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: For a normal matrix S , σ i ( S ) =
Revised: For a normal matrix S , σ i ( S ) =

Original: | λ i ( S ) | , so this ratio can be bounded above as
Revised: | λ i ( S ) | , so this ratio can be bounded above as

Original: [Equation]
Revised: [Equation]

Original: Deﬁning f
Revised: Deﬁning f

Original: ( k ) to be the right hand side of the equation, we can verify that f is a monotonically decreasing function in k when σ i < σ j .
Revised: ( k ) to be the right hand side of the equation, we can verify that f is a monotonically decreasing function in k when σ i < σ j .

Original: This shows that this ratio of singular values in bounded above and in general, must decrease towards some limit lim k →∞ f ( k ) .
Revised: This shows that this ratio of singular values in bounded above and in general, must decrease towards some limit lim k →∞ f ( k ) .

Original: Construction of Subsequence: We now show that there exists a subsequence ( k l ) ∞ l =1 for which S k l is approximately positive semi-deﬁnite.
Revised: Construction of Subsequence: We now show that there exists a subsequence ( k l ) ∞ l =1 for which S k l is approximately positive semi-deﬁnite.

Original: For ease of notation, let’s represent the i-th eigenvalue as λ i ( S )
Revised: For ease of notation, let’s represent the i-th eigenvalue as λ i ( S )

Original: =
Revised: =

Original: | λ i ( S ) | · e iθ i , where θ i > 0 is the polar angle of the complex value
Revised: | λ i ( S ) | · e iθ i , where θ i > 0 is the polar angle of the complex value

Original: λ
Revised: λ

Original: i ( s ) and | λ i ( S ) | is its magnitude (norm).
Revised: i ( s ) and | λ i ( S ) | is its magnitude (norm).

Original: Now, using Lemma C.1.2, we can approximate any polar angle, θ i using a rational approximation, i.e. , , we apply lemma C.1.2 on θ i 2 π
Revised: Now, using Lemma C.1.2, we can approximate any polar angle, θ i using a rational approximation, i.e. , , we apply lemma C.1.2 on θ i 2 π

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the choice of q i is within our control we can estimate θ i for all eigenvalues λ
Revised: Since the choice of q i is within our control we can estimate θ i for all eigenvalues λ

Original: i ( S ) to inﬁnitesimal accuracy.
Revised: i ( S ) to inﬁnitesimal accuracy.

Original: Hence, we can approximate θ i ≈
Revised: Hence, we can approximate θ i ≈

Original: π p i q i .
Revised: π p i q i .

Original: We will now use this approximation to construct an inﬁnite sequence ( k l ) ∞ l =1 , shown below:
Revised: We will now use this approximation to construct an inﬁnite sequence ( k l ) ∞ l =1 , shown below:

Original: [Equation]
Revised: [Equation]

Original: where LCM is the least-common-multiple of natural numbers q 1 , · ·
Revised: where LCM is the least-common-multiple of natural numbers q 1 , · ·

Original: [Equation]
Revised: [Equation]

Original: q n .
Revised: q n .

Original: In the absence of any approximation error in θ i , we note that for any i and for any l ∈ N as deﬁned above, λ i ( S ) k l = |
Revised: In the absence of any approximation error in θ i , we note that for any i and for any l ∈ N as deﬁned above, λ i ( S ) k l = |

Original: λ i ( S ) | k l · exp (cid:16) 2 iπ · p i q i · k l (cid:17) =
Revised: λ i ( S ) | k l · exp (cid:16) 2 iπ · p i q i · k l (cid:17) =

Original: | λ i ( S ) | k l , since the polar angle for any k l is going to be a multiple of 2 π , and hence it would fall on the real line.
Revised: | λ i ( S ) | k l , since the polar angle for any k l is going to be a multiple of 2 π , and hence it would fall on the real line.

Original: As a result, S k l will be positive semi-deﬁnite, since all eigenvalues are positive and real.
Revised: As a result, S k l will be positive semi-deﬁnite, since all eigenvalues are positive and real.

Original: Now by using the proof for Lemma C.1.1, we obtain the ratio of i and j singular values are increasing over the sequence of iterations ( k j ) ∞
Revised: Now by using the proof for Lemma C.1.1, we obtain the ratio of i and j singular values are increasing over the sequence of iterations ( k j ) ∞

Original: j =1 .
Revised: j =1 .

Original: Since the approximation error in θ i can be controlled to be inﬁnitesimally small to prevent any increase in the value of srank δ due to it (this can be done given the discrete form of srank δ ), we note that the above argument applies even with the approximation, proving the required result on the subsequence.
Revised: Since the approximation error in θ i can be controlled to be inﬁnitesimally small to prevent any increase in the value of srank δ due to it (this can be done given the discrete form of srank δ ), we note that the above argument applies even with the approximation, proving the required result on the subsequence.

Original: Controlling All Fitting Iterations using Subsequence:
Revised: Controlling All Fitting Iterations using Subsequence:

Original: We now relate the ratio of singular values within this chosen subsequence to the ratio of singular values elsewhere.
Revised: We now relate the ratio of singular values within this chosen subsequence to the ratio of singular values elsewhere.

Original: Choose t, l ∈
Revised: Choose t, l ∈

Original: N
Revised: N

Original: such that t > k l .
Revised: such that t > k l .

Original: Earlier in this proof, we showed that the ratio between singular values is bounded above by a monotonically decreasing function f ( t ) , so
Revised: Earlier in this proof, we showed that the ratio between singular values is bounded above by a monotonically decreasing function f ( t ) , so

Original: [Equation]
Revised: [Equation]

Original: (C.26)
Revised: (C.26)

Original: Now, we show that that f ( k l ) is in fact very close to the ratio of singular values:
Revised: Now, we show that that f ( k l ) is in fact very close to the ratio of singular values:

Original: [Equation]
Revised: [Equation]

Original: The second term goes to zero as k l increases; algebraic manipulation shows that this gap be bounded by
Revised: The second term goes to zero as k l increases; algebraic manipulation shows that this gap be bounded by

Original: [Equation]
Revised: [Equation]

Original: Putting these inequalities together proves the ﬁnal statement,
Revised: Putting these inequalities together proves the ﬁnal statement,

Original: [Equation]
Revised: [Equation]

Original: Extension to approximately-normal S .
Revised: Extension to approximately-normal S .

Original: We can extend the result in Theorem C.1 (and hence also
Revised: We can extend the result in Theorem C.1 (and hence also

Original: Theorem 4.1) to approximately-normal S .
Revised: Theorem 4.1) to approximately-normal S .

Original: Note that the main requirement for normality of S (i.e., σ i ( S ) =
Revised: Note that the main requirement for normality of S (i.e., σ i ( S ) =

Original: | λ i ( s ) | ) is because it is straightforward to relate the eigenvalue of S to M as shown below.
Revised: | λ i ( s ) | ) is because it is straightforward to relate the eigenvalue of S to M as shown below.

Original: [Equation]
Revised: [Equation]

Original: Now, since the matrix S is approximately normal, we can express it using its Schur’s triangular form as, S
Revised: Now, since the matrix S is approximately normal, we can express it using its Schur’s triangular form as, S

Original: = U · (Λ + N ) · U H , where Λ is a diagonal matrix and N is an “offset” matrix.
Revised: = U · (Λ + N ) · U H , where Λ is a diagonal matrix and N is an “offset” matrix.

Original: The departure from normality of S is deﬁned as: ∆( S ) :=
Revised: The departure from normality of S is deﬁned as: ∆( S ) :=

Original: inf N || N || 2 , where the inﬁmum is computed over all matrices N that can appear in the Schur triangular form for S .
Revised: inf N || N || 2 , where the inﬁmum is computed over all matrices N that can appear in the Schur triangular form for S .

Original: For a normal S only a single value of N = 0 satisﬁes the Schur’s triangular form.
Revised: For a normal S only a single value of N = 0 satisﬁes the Schur’s triangular form.

Original: For an approximately normal matrix S , || N || 2 ≤ ∆( S ) ≤ ε , for a small ε .
Revised: For an approximately normal matrix S , || N || 2 ≤ ∆( S ) ≤ ε , for a small ε .

Original: Furthermore note that from Equation 6 in Ruhe (1975), we obtain that
Revised: Furthermore note that from Equation 6 in Ruhe (1975), we obtain that

Original: [Equation]
Revised: [Equation]

Original: implying that singular values and norm-eigenvalues are close to each other for S .
Revised: implying that singular values and norm-eigenvalues are close to each other for S .

Original: Next, let us evaluate the departure from normality of M k .
Revised: Next, let us evaluate the departure from normality of M k .

Original: First note that,
Revised: First note that,

Original: S j = U · (Λ + N ) j · U H , and so, M k
Revised: S j = U · (Λ + N ) j · U H , and so, M k

Original: = U · (cid:16)(cid:80) kj =1 (Λ + N ) j (cid:17) · U H and if || N || 2 ≤ ε , for a small epsilon ( i.e. , considering only terms that are linear in N for (Λ + N ) j ), we note that:
Revised: = U · (cid:16)(cid:80) kj =1 (Λ + N ) j (cid:17) · U H and if || N || 2 ≤ ε , for a small epsilon ( i.e. , considering only terms that are linear in N for (Λ + N ) j ), we note that:

Original: [Equation]
Revised: [Equation]

Original: Thus, the matrix M k is also approximately normal provided that the max eigenvalue norm of S is less than 1.
Revised: Thus, the matrix M k is also approximately normal provided that the max eigenvalue norm of S is less than 1.

Original: This is true, since S = γP π A (see Theorem 4.1, where both P π and A have eigenvalues less than 1, and γ < 1 .
Revised: This is true, since S = γP π A (see Theorem 4.1, where both P π and A have eigenvalues less than 1, and γ < 1 .

Original: Given that we have shown that M k is approximately normal, we can show that srank δ ( M k ) only differs from srank δ,λ ( M k ) , i.e. , , the effective rank of eigenvalues, in a bounded amount.
Revised: Given that we have shown that M k is approximately normal, we can show that srank δ ( M k ) only differs from srank δ,λ ( M k ) , i.e. , , the effective rank of eigenvalues, in a bounded amount.

Original: If the value of ε is then small enough, we still retain the conclusion that srank δ ( M k ) generally decreases with more training by following the proof of Theorem C.1.
Revised: If the value of ε is then small enough, we still retain the conclusion that srank δ ( M k ) generally decreases with more training by following the proof of Theorem C.1.

Original: D P ROOFS FOR S ECTION 4. 
Revised: D P ROOFS FOR S ECTION 4.

Original: In this section, we provide technical proofs from Section 4.2.
Revised: In this section, we provide technical proofs from Section 4.2.

Original: We start by deriving properties of optimization trajectories of the weight matrices of the deep linear network similar to Arora et al.
Revised: We start by deriving properties of optimization trajectories of the weight matrices of the deep linear network similar to Arora et al.

Original: but customized to our set of assumptions, then prove Proposition 4.1, and ﬁnally discuss how to extend these results to the ﬁtted Q-iteration setting and some extensions not discussed in the main paper.
Revised: but customized to our set of assumptions, then prove Proposition 4.1, and ﬁnally discuss how to extend these results to the ﬁtted Q-iteration setting and some extensions not discussed in the main paper.

Original: Similar to Section 4.1, we assume access to a dataset of transitions, D
Revised: Similar to Section 4.1, we assume access to a dataset of transitions, D

Original: = { ( s i , a i , r ( s i , a i ) , s
Revised: = { ( s i , a i , r ( s i , a i ) , s

Original: [Equation]
Revised: [Equation]

Original: i } in this section, and assume that the same data is used to re-train the function.
Revised: i } in this section, and assume that the same data is used to re-train the function.

Original: Notation and Deﬁnitions.
Revised: Notation and Deﬁnitions.

Original: The Q-function is represented using a deep linear network with at least 3 layers, such that
Revised: The Q-function is represented using a deep linear network with at least 3 layers, such that

Original: [Equation]
Revised: [Equation]

Original: and W i ∈
Revised: and W i ∈

Original: R
Revised: R

Original: d i × d i − 1 for i = 1 , . . .
Revised: d i × d i − 1 for i = 1 , . . .

Original: , N − 1 .
Revised: , N − 1 .

Original: We index the weight matrices by a tuple ( k, t ) :
Revised: We index the weight matrices by a tuple ( k, t ) :

Original: W j ( k, t ) denotes the weight matrix W j at the t -th step of gradient descent during the k -th ﬁtting iteration (Algorithm 1).
Revised: W j ( k, t ) denotes the weight matrix W j at the t -th step of gradient descent during the k -th ﬁtting iteration (Algorithm 1).

Original: Let the end-to-end weight matrix W N W N − 1 · · · W 1 be denoted shorthand as
Revised: Let the end-to-end weight matrix W N W N − 1 · · · W 1 be denoted shorthand as

Original: W N :1 , and let the features of the penultimate layer of the network, be denoted as W φ ( k, t ) :
Revised: W N :1 , and let the features of the penultimate layer of the network, be denoted as W φ ( k, t ) :

Original: =
Revised: =

Original: W N − 1 ( k, t ) · · · W 1 ( k, t ) .
Revised: W N − 1 ( k, t ) · · · W 1 ( k, t ) .

Original: W φ ( k, t ) is the matrix that maps an input [ s ; a ] to corresponding features Φ( s , a ) .
Revised: W φ ( k, t ) is the matrix that maps an input [ s ; a ] to corresponding features Φ( s , a ) .

Original: In our analysis, it is sufﬁcient to consider the effective rank of W φ ( k, t ) since the features Φ are given by: Φ( k, t ) =
Revised: In our analysis, it is sufﬁcient to consider the effective rank of W φ ( k, t ) since the features Φ are given by: Φ( k, t ) =

Original: W φ ( k, t )[ S ; A ] , which indicates that:
Revised: W φ ( k, t )[ S ; A ] , which indicates that:

Original: [Equation]
Revised: [Equation]

Original: Assuming the state-action space has full rank, we are only concerned about rank ( W φ ( k, t )) which justiﬁes our choice for analyzing srank δ ( W φ ( k, t )) .
Revised: Assuming the state-action space has full rank, we are only concerned about rank ( W φ ( k, t )) which justiﬁes our choice for analyzing srank δ ( W φ ( k, t )) .

Original: Let L k +

Revised: Let L k +

Original: ( W N :1 ( k, t )) denote the mean squared Bellman error optimization objective in the k -th ﬁtting iteration.
Revised: ( W N :1 ( k, t )) denote the mean squared Bellman error optimization objective in the k -th ﬁtting iteration.

Original: [Equation]
Revised: [Equation]

Original: a i )) 2 , where y k = R + γP π Q k .
Revised: a i )) 2 , where y k = R + γP π Q k .

Original: When gradient descent is used to update the weight matrix, the updates to W i ( k, t ) are given by:
Revised: When gradient descent is used to update the weight matrix, the updates to W i ( k, t ) are given by:

Original: [Equation]
Revised: [Equation]

Original: If the learning rate η is small, we can approximate this discrete time process with a continuous-time differential equation, which we will use for our analysis.
Revised: If the learning rate η is small, we can approximate this discrete time process with a continuous-time differential equation, which we will use for our analysis.

Original: We use ˙ W ( k, t ) to denote the derivative of
Revised: We use ˙ W ( k, t ) to denote the derivative of

Original: W ( k, t ) with respect to t , for a given k .
Revised: W ( k, t ) with respect to t , for a given k .

Original: [Equation]
Revised: [Equation]

Original: In order to quantify the evolution of singular values of the weight matrix, W φ ( k, t ) , we start by quantifying the evolution of the weight matrix W φ ( k, t ) using a more interpretable differential equation.
Revised: In order to quantify the evolution of singular values of the weight matrix, W φ ( k, t ) , we start by quantifying the evolution of the weight matrix W φ ( k, t ) using a more interpretable differential equation.

Original: In order to do so, we make an assumption similar to but not identical as Arora et al.
Revised: In order to do so, we make an assumption similar to but not identical as Arora et al.

Original: (2018), that assumes that all except the last weight matrix are “balanced” at initialization t = 0 , k = 0 .
Revised: (2018), that assumes that all except the last weight matrix are “balanced” at initialization t = 0 , k = 0 .

Original: i.e.
Revised: i.e.

Original: [Equation]
Revised: [Equation]

Original: Note the distinction from Arora et al.
Revised: Note the distinction from Arora et al.

Original: (2018), the last layer is not assumed to be balanced.
Revised: (2018), the last layer is not assumed to be balanced.

Original: As a result, we may not be able to comment about the learning dynamics of the end-to-end weight matrix, but we prevent the vacuous case where all the weight matrices are rank 1.
Revised: As a result, we may not be able to comment about the learning dynamics of the end-to-end weight matrix, but we prevent the vacuous case where all the weight matrices are rank 1.

Original: Now we are ready to derive the evolution of the feature matrix, W φ ( k, t ) .
Revised: Now we are ready to derive the evolution of the feature matrix, W φ ( k, t ) .

Original: Lemma D.0.1 (Adaptation of balanced weights (Arora et al., 2018) across FQI iterations) .
Revised: Lemma D.0.1 (Adaptation of balanced weights (Arora et al., 2018) across FQI iterations) .

Original: Assume the weight matrices evolve according to Equation D.2, with respect to L k for all ﬁtting iterations k .
Revised: Assume the weight matrices evolve according to Equation D.2, with respect to L k for all ﬁtting iterations k .

Original: Assume balanced initialization only for the ﬁrst N − 1 layers, i.e., W j +1 (0 , 0) T W j +

Revised: Assume balanced initialization only for the ﬁrst N − 1 layers, i.e., W j +1 (0 , 0) T W j +

Original: (0 , 0) =
Revised: (0 , 0) =

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: , ∀
Revised: , ∀

Original: [Equation]
Revised: [Equation]

Original: 1 , · · · , N − 2 .
Revised: 1 , · · · , N − 2 .

Original: Then the weights remain balanced throughout, i.e.
Revised: Then the weights remain balanced throughout, i.e.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First consider the special case of k = 0 .
Revised: First consider the special case of k = 0 .

Original: To beign with, in order to show that weights remain balanced throughout training in k = 0 iteration, we will follow the proof technique in Arora et al.
Revised: To beign with, in order to show that weights remain balanced throughout training in k = 0 iteration, we will follow the proof technique in Arora et al.

Original: (2018), with some modiﬁcations.
Revised: (2018), with some modiﬁcations.

Original: First note that the expression for ∂L k +1 ( W N :1 ( k,t ))
Revised: First note that the expression for ∂L k +1 ( W N :1 ( k,t ))

Revised: [Equation]

Original: ∂ W j ( k,t ) can be expressed as:
Revised: W j ( k,t ) can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: Now, since the weight matrices evolve as per Equation D.2, by multiplying the similar differential equation for W j with W Tj ( k, t ) on the right and multiplying evolution of W j +1 with W Tj +1 ( k, t ) from the left, and adding the two equations, we obtain:
Revised: Now, since the weight matrices evolve as per Equation D.2, by multiplying the similar differential equation for W j with W Tj ( k, t ) on the right and multiplying evolution of W j +1 with W Tj +1 ( k, t ) from the left, and adding the two equations, we obtain:

Original: [Equation]
Revised: [Equation]

Original: We can then take transpose of the equation above, and add it to itself, to obtain an easily integrable expression:
Revised: We can then take transpose of the equation above, and add it to itself, to obtain an easily integrable expression:

Original: [Equation]
Revised: [Equation]

Original: Since we have assumed the balancedness condition at the initial timestep 0 , and the derivatives of the two quantities are equal, their integral will also be the same, hence we obtain:
Revised: Since we have assumed the balancedness condition at the initial timestep 0 , and the derivatives of the two quantities are equal, their integral will also be the same, hence we obtain:

Original: [Equation]
Revised: [Equation]

Original: Now, since the weights after T iterations in ﬁtting iteration k = 0 are still balanced, the initialization for k = 1 is balanced.
Revised: Now, since the weights after T iterations in ﬁtting iteration k = 0 are still balanced, the initialization for k = 1 is balanced.

Original: Note that since the balancedness property does not depend on which objective gradient is used to optimize the weights, as long as W j and W j +1 utilize the same gradient of the loss function.
Revised: Note that since the balancedness property does not depend on which objective gradient is used to optimize the weights, as long as W j and W j +1 utilize the same gradient of the loss function.

Original: Formalizing this, we can show inductively that the weights will remain balanced across all ﬁtting iterations k and at all steps t within each ﬁtting iteration.
Revised: Formalizing this, we can show inductively that the weights will remain balanced across all ﬁtting iterations k and at all steps t within each ﬁtting iteration.

Original: Thus, we have shown the result in Equation D.4.
Revised: Thus, we have shown the result in Equation D.4.

Original: Our next result aims at deriving the evolution of the feature-matrix that under the balancedness condition.
Revised: Our next result aims at deriving the evolution of the feature-matrix that under the balancedness condition.

Original: We will show that the feature matrix, W φ ( k, t ) evolves according to a similar, but distinct differential equation as the end-to-end weight matrix, W N :

Revised: We will show that the feature matrix, W φ ( k, t ) evolves according to a similar, but distinct differential equation as the end-to-end weight matrix, W N :

Original: ( k, t ) , which still allows us to appeal to techniques and results from Arora et al.
Revised: ( k, t ) , which still allows us to appeal to techniques and results from Arora et al.

Original: (2019) to study properties of the singular value evolution and hence, discuss properties related to the effective rank of the matrix, W φ ( k, t ) .
Revised: (2019) to study properties of the singular value evolution and hence, discuss properties related to the effective rank of the matrix, W φ ( k, t ) .

Original: Lemma D.0.2 ((Extension of Theorem 1 from Arora et al.
Revised: Lemma D.0.2 ((Extension of Theorem 1 from Arora et al.

Original: Under conditions speciﬁed in
Revised: Under conditions speciﬁed in

Original: Lemma D.0.1, the feature matrix, W φ ( k, t ) evolves as per the following continuous-time differential equation, for all ﬁtting iterations k :
Revised: Lemma D.0.1, the feature matrix, W φ ( k, t ) evolves as per the following continuous-time differential equation, for all ﬁtting iterations k :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In order to prove this statement, we utilize the fact that the weights upto layer N − 2 are balanced throughout training.
Revised: In order to prove this statement, we utilize the fact that the weights upto layer N − 2 are balanced throughout training.

Original: Now consider the singular value decomposition of any weight w j (unless otherwise states, we use W j to refer to W j ( k, t ) in this section, for ease of notation.
Revised: Now consider the singular value decomposition of any weight w j (unless otherwise states, we use W j to refer to W j ( k, t ) in this section, for ease of notation.

Original: W j =
Revised: W j =

Original: U j Σ j V Tj .
Revised: U j Σ j V Tj .

Original: The belancedness condition re-written using SVD of the weight matrices is equivalent to
Revised: The belancedness condition re-written using SVD of the weight matrices is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Thus for all j on which the balancedness condition is valid, it must hold that Σ Tj
Revised: Thus for all j on which the balancedness condition is valid, it must hold that Σ Tj

Original: [Equation]
Revised: [Equation]

Original: Σ
Revised: Σ

Original: [Equation]
Revised: [Equation]

Original: since these are both the eigendecompositions of the same matrix (as they are equal).
Revised: since these are both the eigendecompositions of the same matrix (as they are equal).

Original: As a result, the weight matrices W j and W j +1 share the same singular value space which can be written as ρ 1 ≥ ρ 2 ≥ · · · ≥ ρ m .
Revised: As a result, the weight matrices W j and W j +1 share the same singular value space which can be written as ρ 1 ≥ ρ 2 ≥ · · · ≥ ρ m .

Original: The ordering of eigenvalues can be different, and the matrices U and V can also be different (and be rotations of one other) but the unique values that the singular values would take are the same.
Revised: The ordering of eigenvalues can be different, and the matrices U and V can also be different (and be rotations of one other) but the unique values that the singular values would take are the same.

Original: Note the distinction from Arora et al.
Revised: Note the distinction from Arora et al.

Original: (2018), where they apply balancedness on all matrices, and that in our case would trivially give a rank-1 matrix.
Revised: (2018), where they apply balancedness on all matrices, and that in our case would trivially give a rank-1 matrix.

Original: Now this implies, that we can express the feature matrix, also in terms of the common singular values, ( ρ 1 , ρ 2 , · · · , ρ m ) , for example, as W j ( k, t ) =
Revised: Now this implies, that we can express the feature matrix, also in terms of the common singular values, ( ρ 1 , ρ 2 , · · · , ρ m ) , for example, as W j ( k, t ) =

Original: U j +1 Diag (cid:0) √ ρ 1 , · · · , √
Revised: U j +1 Diag (cid:0) √ ρ 1 , · · · , √

Original: ρ m (cid:1) V Tj , where
Revised: ρ m (cid:1) V Tj , where

Original: U )
Revised: U )

Original: j = V
Revised: j = V

Original: j +1 O j , where O j is an orthonormal matrix.
Revised: j +1 O j , where O j is an orthonormal matrix.

Original: Using this relationship, we can say the following:
Revised: Using this relationship, we can say the following:

Original: [Equation]
Revised: [Equation]

Original: Now, we can use these expressions to obtain the desired result, by taking the differential equations governing the evolution of W i ( k, t ) , for i ∈ [1 , · · · , N − 1] , multiplying the i -th equation by (cid:81)
Revised: Now, we can use these expressions to obtain the desired result, by taking the differential equations governing the evolution of W i ( k, t ) , for i ∈ [1 , · · · , N − 1] , multiplying the i -th equation by (cid:81)

Original: N −i +1 W j ( k, t ) from the left, and (cid:81) i − 

Revised: N −i +1 W j ( k, t ) from the left, and (cid:81) i −

Original: W j ( k, t ) to the right, and then summing over i .
Revised: W j ( k, t ) to the right, and then summing over i .

Original: ˙
Revised: ˙

Original: [Equation]
Revised: [Equation]

Original: N (cid:89) i +
Revised: N (cid:89) i +

Original: W j ( k, t ) T (cid:33) dL [-
-]
Revised: W j ( k, t ) T (cid:33) dL [-
-]k

Original: ( W N :1 )
Revised: ( W N :1 )

Original: d W N :   i − 1 (cid:89) j =
Revised: d W N :   i − 1 (cid:89) j =

Original: W j ( k, t ) T i − 1 (cid:89) j =
Revised: W j ( k, t ) T i − 1 (cid:89) j =

Original: W j ( k, t ) 
Revised: W j ( k, t ) 

Original: 
Revised: 

Original: The above equation simpliﬁes to the desired result by taking out W N ( k, t ) from the ﬁrst summation, and using the identities above for each of the terms.
Revised: The above equation simpliﬁes to the desired result by taking out W N ( k, t ) from the ﬁrst summation, and using the identities above for each of the terms.

Original: Comparing the previous result with Theorem 1 in Arora et al.
Revised: Comparing the previous result with Theorem 1 in Arora et al.

Original: , we note that the resulting differential equation for weights holds true for arbitrary representations or features in the network provided that the layers from the input to the feature layer are balanced.
Revised: , we note that the resulting differential equation for weights holds true for arbitrary representations or features in the network provided that the layers from the input to the feature layer are balanced.

Original: A direct application of Arora et al.
Revised: A direct application of Arora et al.

Original: (2018) restricts the model class to only fully balanced networks for convergence analysis and the resulting solutions to the feature matrix will then only have one active singular value, leading to less-expressive neural network conﬁgurations.
Revised: (2018) restricts the model class to only fully balanced networks for convergence analysis and the resulting solutions to the feature matrix will then only have one active singular value, leading to less-expressive neural network conﬁgurations.

Original: Proof of Proposition 4.1.
Revised: Proof of Proposition 4.1.

Original: Finally, we are ready to use Lemma D.0.2 to prove the relationship with evolution on singular values.
Revised: Finally, we are ready to use Lemma D.0.2 to prove the relationship with evolution on singular values.

Original: This proof can be shown via a direct application of Theorem 3 in Arora et al.
Revised: This proof can be shown via a direct application of Theorem 3 in Arora et al.

Original: Given that the feature matrix, W φ ( k, t ) satisﬁes a very similar differential equation as the end-to-end matrix, with the exception that the gradient of the loss with respect to the end-to-end matrix is pre-multiplied by W N ( k, t ) T .
Revised: Given that the feature matrix, W φ ( k, t ) satisﬁes a very similar differential equation as the end-to-end matrix, with the exception that the gradient of the loss with respect to the end-to-end matrix is pre-multiplied by W N ( k, t ) T .

Original: As a result, we can directly invoke Arora et al.
Revised: As a result, we can directly invoke Arora et al.

Original: (2019)’s result and hence, we have ∀ r ∈
Revised: (2019)’s result and hence, we have ∀ r ∈

Original: [1 , · · · , dim( W )] that:
Revised: [1 , · · · , dim( W )] that:

Original: [Equation]
Revised: [Equation]

Original: Further, as another consequence of the result describing the evolution of weight matrices, we can also obtain a result similar to Arora et al.
Revised: Further, as another consequence of the result describing the evolution of weight matrices, we can also obtain a result similar to Arora et al.

Original: (2019) that suggests that the goal of the gradient update on the singular vectors U ( k, t ) and V ( k, t ) of the features W φ ( k, t )
Revised: (2019) that suggests that the goal of the gradient update on the singular vectors U ( k, t ) and V ( k, t ) of the features W φ ( k, t )

Original: =
Revised: =

Original: U ( k, t ) S ( k, t ) V ( K, t ) T , is to align these spaces with W N ( k, t ) T dL N,
Revised: U ( k, t ) S ( k, t ) V ( K, t ) T , is to align these spaces with W N ( k, t ) T dL N,

Original: k
Revised: k

Original: ( W K,t ) d W .
Revised: ( W K,t ) d W .

Original: D.
Revised: D.

Original: E XPLAINING R ANK D ECREASE B ASED ON S INGULAR V ALUE E VOLUTION
Revised: E XPLAINING R ANK D ECREASE B ASED ON S INGULAR V ALUE E VOLUTION

Original: [Figure]
Revised: [Figure]

Original: v a l u e
Revised: v a l u e

Original: [Figure]
Revised: [Figure]

Original: ( l o g s c a l e )
Revised: ( l o g s c a l e )

Original: [Figure]
Revised: [Figure]

Original: In this section, we discuss why the evolution of singular values discussed in Equation 4 indicates a decrease in the rank of the feature matrix within a ﬁtting iteration k .
Revised: In this section, we discuss why the evolution of singular values discussed in Equation 4 indicates a decrease in the rank of the feature matrix within a ﬁtting iteration k .

Original: To see this, let’s consider the case when gradient descent has been run long enough ( i.e. , the data-efﬁcient RL case) for the singular vectors to stabilize, and consider the evolution of singular values post timestep t ≥ t 0 in training.
Revised: To see this, let’s consider the case when gradient descent has been run long enough ( i.e. , the data-efﬁcient RL case) for the singular vectors to stabilize, and consider the evolution of singular values post timestep t ≥ t 0 in training.

Original: First of all, when the singular vectors stabilize, we obtain that u
Revised: First of all, when the singular vectors stabilize, we obtain that u

Original: [Equation]
Revised: [Equation]

Original: v r ( k, t ) is diagonal (extending result of Corollary 1 from Arora et al.
Revised: v r ( k, t ) is diagonal (extending result of Corollary 1 from Arora et al.

Original: Thus, we can assume that
Revised: Thus, we can assume that

Original: [Equation]
Revised: [Equation]

Original: where e r is given by the unit basis vector for the standard orthonormal basis, f is a shorthand for the gradient norm of the loss function pre-multiplied by the transpose of the last layer weight matrix, and d r denotes the singular values of the state-action input matrix, [ S , A ] .
Revised: where e r is given by the unit basis vector for the standard orthonormal basis, f is a shorthand for the gradient norm of the loss function pre-multiplied by the transpose of the last layer weight matrix, and d r denotes the singular values of the state-action input matrix, [ S , A ] .

Original: In this case, we can re-write Equation 4 as:
Revised: In this case, we can re-write Equation 4 as:

Original: [Equation]
Revised: [Equation]

Original: Note again that unlike Arora et al.
Revised: Note again that unlike Arora et al.

Original: (2019), the expression for f ( k, t ) is different from the gradient of the loss, since the weight matrix W N ( k, t ) is multiplied to the expression of the gradient in our case.
Revised: (2019), the expression for f ( k, t ) is different from the gradient of the loss, since the weight matrix W N ( k, t ) is multiplied to the expression of the gradient in our case.

Original: By using the fact that the expression for f ( k, t ) is shared across all singular values, we can obtain differential equations that are equal across different σ r ( k, t ) and σ r (cid:48)
Revised: By using the fact that the expression for f ( k, t ) is shared across all singular values, we can obtain differential equations that are equal across different σ r ( k, t ) and σ r (cid:48)

Original: ( k, t ) .
Revised: ( k, t ) .

Original: Integrating, we can show that depending upon the ratio e r · d r e r (cid:48) · d r (cid:48) , and the value of N , the singular values σ r ( k, t ) and σ r (cid:48) ( k, t ) will take on different magnitude values, in particular, they will grow at disproportionate rates.
Revised: Integrating, we can show that depending upon the ratio e r · d r e r (cid:48) · d r (cid:48) , and the value of N , the singular values σ r ( k, t ) and σ r (cid:48) ( k, t ) will take on different magnitude values, in particular, they will grow at disproportionate rates.

Original: By then appealing to the result from Proposition 4.1 in Section 4.1 (kernel regression argument), we can say that disproportionately growing singular values would imply that the value of srank δ ( W φ ( k, t )) decreases within each iteration k .
Revised: By then appealing to the result from Proposition 4.1 in Section 4.1 (kernel regression argument), we can say that disproportionately growing singular values would imply that the value of srank δ ( W φ ( k, t )) decreases within each iteration k .

Original: D. P ROOF FOR T HEOREM 4.2: R ANK C OLLAPSE N EAR A F IXED P OINT
Revised: D. P ROOF FOR T HEOREM 4.2: R ANK C OLLAPSE N EAR A F IXED P OINT

Original: Now, we will prove Theorem 4.2 by showing that when the current Q-function iterate Q k is close to a ﬁxed point but have not converged yet, i.e. , , when || Q k − ( R + γP π Q k ) || ≤ ε , then rank-decrease happens.
Revised: Now, we will prove Theorem 4.2 by showing that when the current Q-function iterate Q k is close to a ﬁxed point but have not converged yet, i.e. , , when || Q k − ( R + γP π Q k ) || ≤ ε , then rank-decrease happens.

Original: To prove this, we evaluate the (inﬁnitesimal) change in the singular values of the features of W φ ( k, t ) as a result of an addition in the value of ε .
Revised: To prove this, we evaluate the (inﬁnitesimal) change in the singular values of the features of W φ ( k, t ) as a result of an addition in the value of ε .

Original: In this case, the change (or differential) in the singular value matrix, S ( k, T ) , ( W φ ( k, T ) =
Revised: In this case, the change (or differential) in the singular value matrix, S ( k, T ) , ( W φ ( k, T ) =

Original: U ( k, T ) S ( k, T ) V ( k, T ) T ) is given by:
Revised: U ( k, T ) S ( k, T ) V ( k, T ) T ) is given by:

Original: [Equation]
Revised: [Equation]

Original: D.
Revised: D.

Original: [Equation]
Revised: [Equation]

Original: using results on computing the derivative of the singular value decomposition (Townsend, 2016).
Revised: using results on computing the derivative of the singular value decomposition (Townsend, 2016).

Original: Proof strategy: Our high-level strategy in this proof is to design a form for ε such that the value of effective rank for feature matrix obtained when the resulting target value
Revised: Proof strategy: Our high-level strategy in this proof is to design a form for ε such that the value of effective rank for feature matrix obtained when the resulting target value

Original: y k
Revised: y k

Original: = Q k − 1 + ε is expressed in the deep linear network function class, let’s say with feature matrix, W (cid:48)
Revised: = Q k − 1 + ε is expressed in the deep linear network function class, let’s say with feature matrix, W (cid:48)

Original: φ ( k, T ) and the last layer weights, W N ( k, T ) =
Revised: φ ( k, T ) and the last layer weights, W N ( k, T ) =

Original: W N ( k − 1 , T ) , then the value of srank δ ( W (cid:48) φ ( k, T )) can be larger than srank δ ( W φ ( k − 1 , T )) by only a bounded limited amount α that depends on ε .
Revised: W N ( k − 1 , T ) , then the value of srank δ ( W (cid:48) φ ( k, T )) can be larger than srank δ ( W φ ( k − 1 , T )) by only a bounded limited amount α that depends on ε .

Original: More formally, we desire a result of the form
Revised: More formally, we desire a result of the form

Original: [Equation]
Revised: [Equation]

Original: where || (cid:15) || (cid:28) ||
Revised: where || (cid:15) || (cid:28) ||

Original: Q k − 1 || , and y k ( s , a
Revised: Q k − 1 || , and y k ( s , a

Original: [Equation]
Revised: [Equation]

Original: = W N ( k − 1 , T ) W (cid:48) φ ( k, T )[ s ; a ] .
Revised: = W N ( k − 1 , T ) W (cid:48) φ ( k, T )[ s ; a ] .

Original: (D.12)
Revised: (D.12)

Original: Once we obtain such a parameterization of y k in the linear network function class, using the argument in Section 4.2 about self-training, we can say that just copying over the weights W (cid:48) φ ( k, T ) is sufﬁcient to obtain a bounded increase in srank δ ( W φ ( k, T )) at the cost of incurring 

Revised: Once we obtain such a parameterization of y k in the linear network function class, using the argument in Section 4.2 about self-training, we can say that just copying over the weights W (cid:48) φ ( k, T ) is sufﬁcient to obtain a bounded increase in srank δ ( W φ ( k, T )) at the cost of incurring

Original: TD error (since the targets can be exactly ﬁt).
Revised: TD error (since the targets can be exactly ﬁt).

Original: As a result, the optimal solution found by Equation 5 will attain lower srank δ ( W φ ( k, T )) value if the TD error is non-zero.
Revised: As a result, the optimal solution found by Equation 5 will attain lower srank δ ( W φ ( k, T )) value if the TD error is non-zero.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: As a result we need to examine the factors that affect srank δ ( W φ ( k, T )) – (1) an increase in srank δ ( W φ ( k, T )) due to an addition of α to the rank, and (2) a decrease in srank δ ( W φ ( k, T )) due to the implicit behavior of gradient descent.
Revised: As a result we need to examine the factors that affect srank δ ( W φ ( k, T )) – (1) an increase in srank δ ( W φ ( k, T )) due to an addition of α to the rank, and (2) a decrease in srank δ ( W φ ( k, T )) due to the implicit behavior of gradient descent.

Original: Proof: In order to obtain the desired result (Equation D.12), we can express ε ( s , a ) as a function of the last layer weight matrix of the current Q-function, W N ( k − 1 , T ) and the state-action inputs, [ s ; a ] .
Revised: Proof: In order to obtain the desired result (Equation D.12), we can express ε ( s , a ) as a function of the last layer weight matrix of the current Q-function, W N ( k − 1 , T ) and the state-action inputs, [ s ; a ] .

Original: Formally, let ε ( s , a ) =
Revised: Formally, let ε ( s , a ) =

Original: W N ( k − 1 , T ) ζ [ s ; a ] , where ζ is a matrix of the same size as W φ such that || ζ || ∞ (cid:28) τ
Revised: W N ( k − 1 , T ) ζ [ s ; a ] , where ζ is a matrix of the same size as W φ such that || ζ || ∞ (cid:28) τ

Original: = O ( || W φ ( k, T ) || ∞ ) , i.e. , ζ has entries with ignorable magnitude compared the actual feature matrix, W φ ( k, T ) .
Revised: = O ( || W φ ( k, T ) || ∞ ) , i.e. , ζ has entries with ignorable magnitude compared the actual feature matrix, W φ ( k, T ) .

Original: Using this form of ε ( s , a ) , we note that the targets y k used for the backup can be written as
Revised: Using this form of ε ( s , a ) , we note that the targets y k used for the backup can be written as

Original: [Equation]
Revised: [Equation]

Original: Using the equation for sensitivity of singular values due to a change in matrix entries, we obtain the maximum change in the singular values of the resulting “effective” feature matrix of the targets y k in the linear function class, denoted as W (cid:48) φ ( k, T ) , is bounded:
Revised: Using the equation for sensitivity of singular values due to a change in matrix entries, we obtain the maximum change in the singular values of the resulting “effective” feature matrix of the targets y k in the linear function class, denoted as W (cid:48) φ ( k, T ) , is bounded:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: let’s use this inequality to ﬁnd out the maximum change in the function, h k ( i ) used to compute
Revised: let’s use this inequality to ﬁnd out the maximum change in the function, h k ( i ) used to compute

Original: [Equation]
Revised: [Equation]

Original: The above equation implies that the maximal change in the effective rank of the feature matrix generated by the targets, y k (denoted as W (cid:48) φ ( k, T ) ) and the effective rank of the features of the current Q-function Q k (denoted as W φ ( k, T ) ) are given by:
Revised: The above equation implies that the maximal change in the effective rank of the feature matrix generated by the targets, y k (denoted as W (cid:48) φ ( k, T ) ) and the effective rank of the features of the current Q-function Q k (denoted as W φ ( k, T ) ) are given by:

Original: [Equation]
Revised: [Equation]

Original: where α can be formally written by the cardinality of the set:
Revised: where α can be formally written by the cardinality of the set:

Original: [Equation]
Revised: [Equation]

Original: Note that by choosing ε ( s , a ) and thus ζ to be small enough, we can obtain W (cid:48) φ ( k, T ) such that α = 0 .
Revised: Note that by choosing ε ( s , a ) and thus ζ to be small enough, we can obtain W (cid:48) φ ( k, T ) such that α = 0 .

Original: Now, the self-training argument discussed above applies and gradient descent in the next iteration will give us solutions that reduce rank.
Revised: Now, the self-training argument discussed above applies and gradient descent in the next iteration will give us solutions that reduce rank.

Original: Assuming r > 1 and srank δ ( W φ ( k, T ))
Revised: Assuming r > 1 and srank δ ( W φ ( k, T ))

Original: =
Revised: =

Original: r , we know that h k ( r − 1) <
Revised: r , we know that h k ( r − 1) <

Original: 1 − δ , while h k ( r ) ≥ 1 − δ .
Revised: 1 − δ , while h k ( r ) ≥ 1 − δ .

Original: Thus, srank δ ( W (cid:48) φ ( k, T )) to be equal to r , it is sufﬁcient to show that h k ( r ) − | dh
Revised: Thus, srank δ ( W (cid:48) φ ( k, T )) to be equal to r , it is sufﬁcient to show that h k ( r ) − | dh

Original: k
Revised: k

Original: ( r ) | ≥ 1 − δ since both h k ( i ) and dh k ( i ) are increasing for i = 0 , 1 , · · · , r .
Revised: ( r ) | ≥ 1 − δ since both h k ( i ) and dh k ( i ) are increasing for i = 0 , 1 , · · · , r .

Original: Thus, srank δ ( W (cid:48) φ ( k, T ))
Revised: Thus, srank δ ( W (cid:48) φ ( k, T ))

Original: =
Revised: =

Original: r ∀ ζ , whenever
Revised: r ∀ ζ , whenever

Original: [Equation]
Revised: [Equation]

Original: This implies that ε ≤ || W N ( k, T ) || ∞
Revised: This implies that ε ≤ || W N ( k, T ) || ∞

Original: [Equation]
Revised: [Equation]

Original: rj
Revised: rj

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof summary: We have thus shown that there exists a neighborhood around the optimal ﬁxed point of the Bellman equation, parameterized by ε ( s , a ) where bootstrapping behaves like selftraining.
Revised: Proof summary: We have thus shown that there exists a neighborhood around the optimal ﬁxed point of the Bellman equation, parameterized by ε ( s , a ) where bootstrapping behaves like selftraining.

Original: In this case, it is possible to reduce srank while the TD error is non-zero.
Revised: In this case, it is possible to reduce srank while the TD error is non-zero.

Original: And of course, this would give rise to a rank reduction close to the optimal solution.
Revised: And of course, this would give rise to a rank reduction close to the optimal solution.