Original: DR3: V ALUE -B ASED D EEP R EINFORCEMENT L EARNING R EQUIRES E XPLICIT R EGULARIZATION
Revised: DR3: V ALUE -B ASED D EEP R EINFORCEMENT L EARNING R EQUIRES E XPLICIT R EGULARIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite overparameterization, deep networks trained via supervised learning are surprisingly easy to optimize and exhibit excellent generalization.
Revised: Despite overparameterization, deep networks trained via supervised learning are surprisingly easy to optimize and exhibit excellent generalization.

Original: One hypothesis to explain this is that overparameterized deep networks enjoy the beneﬁts of implicit regularization induced by stochastic gradient descent, which favors parsimonious solutions that generalize well on test inputs.
Revised: One hypothesis to explain this is that overparameterized deep networks enjoy the beneﬁts of implicit regularization induced by stochastic gradient descent, which favors parsimonious solutions that generalize well on test inputs.

Original: It is reasonable to surmise that deep reinforcement learning (RL) methods could also beneﬁt from this effect.
Revised: It is reasonable to surmise that deep reinforcement learning (RL) methods could also beneﬁt from this effect.

Original: In this paper, we discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the ofﬂine deep RL setting, leading to poor generalization and degenerate feature representations.
Revised: In this paper, we discuss how the implicit regularization effect of SGD seen in supervised learning could in fact be harmful in the ofﬂine deep RL setting, leading to poor generalization and degenerate feature representations.

Original: Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning, the resulting derived regularizer favors degenerate solutions with excessive “aliasing”, in stark contrast to the supervised learning case.
Revised: Our theoretical analysis shows that when existing models of implicit regularization are applied to temporal difference learning, the resulting derived regularizer favors degenerate solutions with excessive “aliasing”, in stark contrast to the supervised learning case.

Original: We back up these ﬁndings empirically, showing that feature representations learned by a deep network value function trained via bootstrapping can indeed become degenerate, aliasing the representations for state-action pairs that appear on either side of the Bellman backup.
Revised: We back up these ﬁndings empirically, showing that feature representations learned by a deep network value function trained via bootstrapping can indeed become degenerate, aliasing the representations for state-action pairs that appear on either side of the Bellman backup.

Original: To address this issue, we derive the form of this implicit regularizer and, inspired by this derivation, propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer.
Revised: To address this issue, we derive the form of this implicit regularizer and, inspired by this derivation, propose a simple and effective explicit regularizer, called DR3, that counteracts the undesirable effects of this implicit regularizer.

Original: When combined with existing ofﬂine RL methods, DR substantially improves performance and stability, alleviating unlearning in Atari 2600 games, D4RL domains and robotic manipulation from images.
Revised: When combined with existing ofﬂine RL methods, DR substantially improves performance and stability, alleviating unlearning in Atari 2600 games, D4RL domains and robotic manipulation from images.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks are overparameterized, which in principle should leave them vulnerable to overﬁtting.
Revised: Deep neural networks are overparameterized, which in principle should leave them vulnerable to overﬁtting.

Original: Despite this, supervised learning with deep networks still learn representations that generalize well.
Revised: Despite this, supervised learning with deep networks still learn representations that generalize well.

Original: A widely held consensus is that deep nets ﬁnd simple solutions that generalize due to various implicit regularization effects (Damian et al., 2021; Blanc et al., 2020;
Revised: A widely held consensus is that deep nets ﬁnd simple solutions that generalize due to various implicit regularization effects (Damian et al., 2021; Blanc et al., 2020;

Original: Woodworth et al., 2020; Arora et al., 2018; Gunasekar et al., 2017;
Revised: Woodworth et al., 2020; Arora et al., 2018; Gunasekar et al., 2017;

Original: Li et al., 2019; Wei et al., 2020).
Revised: Li et al., 2019; Wei et al., 2020).

Original: We may surmise that using deep nets in reinforcement learning (RL) will work well for the same reason, learning effective representations that generalize due to such implicit regularization effects.
Revised: We may surmise that using deep nets in reinforcement learning (RL) will work well for the same reason, learning effective representations that generalize due to such implicit regularization effects.

Original: But is this actually the case for value functions trained via bootstrapping?
Revised: But is this actually the case for value functions trained via bootstrapping?

Original: In this paper, we argue that, while implicit regularization leads to effective representations in supervised deep learning, it may lead to poor learned representations when training overparameterized deep network value functions.
Revised: In this paper, we argue that, while implicit regularization leads to effective representations in supervised deep learning, it may lead to poor learned representations when training overparameterized deep network value functions.

Original: There is already evidence that value functions trained via bootstrapping learn poor representations: value functions trained with ofﬂine deep RL eventually degrade in performance (Agarwal et al., 2020; Kumar et al., 2021) and this degradation is correlated with the emergence of low-rank features in the value network (Kumar et al., 2021).We focus speciﬁcally on the ofﬂine RL setting, in order to rule out any confounding effects from exploration and nonstationary data.
Revised: There is already evidence that value functions trained via bootstrapping learn poor representations: value functions trained with ofﬂine deep RL eventually degrade in performance (Agarwal et al., 2020; Kumar et al., 2021) and this degradation is correlated with the emergence of low-rank features in the value network (Kumar et al., 2021).We focus speciﬁcally on the ofﬂine RL setting, in order to rule out any confounding effects from exploration and nonstationary data.

Original: In this paper, we aim to understand the underlying cause of the emergence of poor representations and develop a potential solution.
Revised: In this paper, we aim to understand the underlying cause of the emergence of poor representations and develop a potential solution.

Original: Building on the theoretical framework developed by Blanc et al.
Revised: Building on the theoretical framework developed by Blanc et al.

Original: ; Damian et al.
Revised: ; Damian et al.

Original: (2021), we characterize the implicit regularizer that arises when training deep value functions with TD learning.
Revised: (2021), we characterize the implicit regularizer that arises when training deep value functions with TD learning.

Original: This implicit regularizer implies that TD-learning would co-adapt representations of state-action tuples that appear on either side of a Bellman backup.
Revised: This implicit regularizer implies that TD-learning would co-adapt representations of state-action tuples that appear on either side of a Bellman backup.

Original: We show that this theoretically predicted implicit regularization phenomenon manifests in practice as feature co-adaptation , where the features of consecutive state-action tuples learned by the Q-value network become extremely similar in terms of their dot product (Section 3).
Revised: We show that this theoretically predicted implicit regularization phenomenon manifests in practice as feature co-adaptation , where the features of consecutive state-action tuples learned by the Q-value network become extremely similar in terms of their dot product (Section 3).

Original: This co-adaptation can lead to oscillatory learning dynamics, and potentially to poor performance.
Revised: This co-adaptation can lead to oscillatory learning dynamics, and potentially to poor performance.

Original: We ﬁnd that even when Q-values are not overestimated, prolonged training in ofﬂine RL can result in performance degradation as feature co-adaptation increases.
Revised: We ﬁnd that even when Q-values are not overestimated, prolonged training in ofﬂine RL can result in performance degradation as feature co-adaptation increases.

Original: To mitigate the co-adaptation issue that arises as a result of implicit regularization, we propose an explicit regularizer that we call DR3 (Section 4).
Revised: To mitigate the co-adaptation issue that arises as a result of implicit regularization, we propose an explicit regularizer that we call DR3 (Section 4).

Original: While exactly estimating and cancelling the effects of the theoretically derived implicit regularizer is computationally difﬁcult, DR3 provides a simple and tractable theoretically-inspired approximation, that mitigates the issues discussed above.
Revised: While exactly estimating and cancelling the effects of the theoretically derived implicit regularizer is computationally difﬁcult, DR3 provides a simple and tractable theoretically-inspired approximation, that mitigates the issues discussed above.

Original: In practice, DR3 amounts to regularizing the features at consecutive state-action pairs to be dissimilar in terms of their dot-product similarity.
Revised: In practice, DR3 amounts to regularizing the features at consecutive state-action pairs to be dissimilar in terms of their dot-product similarity.

Original: Empirically, we ﬁnd that DR3 prevents previously noted pathologies such as feature rank collapse (Kumar et al., 2021), gives methods that train for longer and improves performance relative to the base ofﬂine RL method employed in practice.
Revised: Empirically, we ﬁnd that DR3 prevents previously noted pathologies such as feature rank collapse (Kumar et al., 2021), gives methods that train for longer and improves performance relative to the base ofﬂine RL method employed in practice.

Original: Our ﬁrst contribution is the derivation of the implicit regularizer that arises when training deep net value functions via TD learning, and empirically demonstrate it manifests as feature co-adaptation in the ofﬂine deep RL setting.
Revised: Our ﬁrst contribution is the derivation of the implicit regularizer that arises when training deep net value functions via TD learning, and empirically demonstrate it manifests as feature co-adaptation in the ofﬂine deep RL setting.

Original: Feature co-adaptation accounts at least in part for some of the challenges of ofﬂine deep RL, including degradation of performance with prolonged training.
Revised: Feature co-adaptation accounts at least in part for some of the challenges of ofﬂine deep RL, including degradation of performance with prolonged training.

Original: Second, we propose a simple and effective explicit regularizer for ofﬂine value-based RL, DR3, which minimizes the feature similarity between state-action pairs appearing in a bootstrapping update.
Revised: Second, we propose a simple and effective explicit regularizer for ofﬂine value-based RL, DR3, which minimizes the feature similarity between state-action pairs appearing in a bootstrapping update.

Original: DR3 is inspired by the theoretical derivation of the implicit regularizer, it alleviates co-adaptation and can be easily combined with modern ofﬂine RL methods, such as REM (Agarwal et al., 2020), CQL (Kumar et al., 2020b), and BRAC (Wu et al., 2019).
Revised: DR3 is inspired by the theoretical derivation of the implicit regularizer, it alleviates co-adaptation and can be easily combined with modern ofﬂine RL methods, such as REM (Agarwal et al., 2020), CQL (Kumar et al., 2020b), and BRAC (Wu et al., 2019).

Original: Empirically, using DR3 in conjunction with existing ofﬂine RL methods provides about 60% performance improvement on the harder D4RL (Fu et al., 2020) tasks, and 160% and 25% stability gains for REM and CQL respectively, on ofﬂine RL tasks in 17 Atari 2600 games.
Revised: Empirically, using DR3 in conjunction with existing ofﬂine RL methods provides about 60% performance improvement on the harder D4RL (Fu et al., 2020) tasks, and 160% and 25% stability gains for REM and CQL respectively, on ofﬂine RL tasks in 17 Atari 2600 games.

Original: Additionally, we observe large improvements on image-based robotic manipulation tasks (Singh et al., 2020).
Revised: Additionally, we observe large improvements on image-based robotic manipulation tasks (Singh et al., 2020).

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: The goal in RL is to maximize the long-term discounted reward in an MDP, deﬁned as ( S , A , R, P, γ ) (Puterman, 1994), with state space S , action space A , a reward function R ( s , a ) , dynamics P ( s ′ | s , a ) and a discount factor γ ∈ [0 , 1) .
Revised: The goal in RL is to maximize the long-term discounted reward in an MDP, deﬁned as ( S , A , R, P, γ ) (Puterman, 1994), with state space S , action space A , a reward function R ( s , a ) , dynamics P ( s ′ | s , a ) and a discount factor γ ∈ [0 , 1) .

Original: The Q-function Q π ( s , a ) for a policy π ( a | s ) is the expected sum of discounted rewards obtained by executing action a at state s and following π ( a | s ) thereafter.
Revised: The Q-function Q π ( s , a ) for a policy π ( a | s ) is the expected sum of discounted rewards obtained by executing action a at state s and following π ( a | s ) thereafter.

Original: Q π ( s , a ) is the ﬁxed point of Q ( s , a ) :
Revised: Q π ( s , a ) is the ﬁxed point of Q ( s , a ) :

Original: = R
Revised: = R

Original: [Equation]
Revised: [Equation]

Original: We study the ofﬂine RL setting, where the algorithm must learn a policy only using a given dataset D = { ( s i , a i , s ′ i , r i ) } , generated from some behavior policy, π β ( a | s ) , without active data collection.
Revised: We study the ofﬂine RL setting, where the algorithm must learn a policy only using a given dataset D = { ( s i , a i , s ′ i , r i ) } , generated from some behavior policy, π β ( a | s ) , without active data collection.

Original: The Q-function is parameterized with a neural net with parameters θ .
Revised: The Q-function is parameterized with a neural net with parameters θ .

Original: We will denote the penultimate layer of the deep network (the learned features ) ϕ
Revised: We will denote the penultimate layer of the deep network (the learned features ) ϕ

Original: θ ( s , a ) , such that Q θ ( s , a ) =
Revised: θ ( s , a ) , such that Q θ ( s , a ) =

Original: w T ϕ ( s , a ) , where w ∈ R d .
Revised: w T ϕ ( s , a ) , where w ∈ R d .

Original: Standard deep RL methods (Mnih et al., 2015; Haarnoja et al., 2018) convert the Bellman equation into a squared temporal difference (TD) error loss (pseudo-objective) for Q θ :
Revised: Standard deep RL methods (Mnih et al., 2015; Haarnoja et al., 2018) convert the Bellman equation into a squared temporal difference (TD) error loss (pseudo-objective) for Q θ :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ¯ Q θ is a delayed copy of same Q-network, referred to as the target network and a ′ is computed by maximizing the target Q-function at state s ′ for Q-learning (i.e., when computing Q ∗ ) and by sampling a ′ ∼ π ( ·| s ) when computing the Q-value Q π of a policy π .
Revised: where ¯ Q θ is a delayed copy of same Q-network, referred to as the target network and a ′ is computed by maximizing the target Q-function at state s ′ for Q-learning (i.e., when computing Q ∗ ) and by sampling a ′ ∼ π ( ·| s ) when computing the Q-value Q π of a policy π .

Original: A major problem in ofﬂine RL is the issue of distributional shift between the learned policy and the behavior policy (Levine et al., 2020).
Revised: A major problem in ofﬂine RL is the issue of distributional shift between the learned policy and the behavior policy (Levine et al., 2020).

Original: Since our goal is to study the effect of implicit regularization in TD-learning and not distributional shift, we build on top of existing ofﬂine RL methods in our experiments: CQL (Kumar et al., 2020b), which penalizes erroneous Q-values during training, REM (Agarwal et al., 2020), which utilizes an ensemble of Q-functions, and BRAC (Wu et al., 2019), which applies a policy constraint.
Revised: Since our goal is to study the effect of implicit regularization in TD-learning and not distributional shift, we build on top of existing ofﬂine RL methods in our experiments: CQL (Kumar et al., 2020b), which penalizes erroneous Q-values during training, REM (Agarwal et al., 2020), which utilizes an ensemble of Q-functions, and BRAC (Wu et al., 2019), which applies a policy constraint.

Original: An overview of these methods is provided in Appendix E.
Revised: An overview of these methods is provided in Appendix E.

Original:  I MPLICIT R EGULARIZATION IN D EEP RL VIA TD-L EARNING
Revised:  I MPLICIT R EGULARIZATION IN D EEP RL VIA TD-L EARNING

Original: While the “deadly-triad” (Sutton & Barto, 2018) suggests that training value function approximators with bootstrapping can lead to divergence, modern deep RL algorithms have been able to successfully combine these properties (van Hasselt et al., 2018).
Revised: While the “deadly-triad” (Sutton & Barto, 2018) suggests that training value function approximators with bootstrapping can lead to divergence, modern deep RL algorithms have been able to successfully combine these properties (van Hasselt et al., 2018).

Original: However, making too many TD updates to the Q-function in ofﬂine deep RL is known to sometimes lead to performance degradation and unlearning, even for otherwise effective modern algorithms (Agarwal et al., 2020; Kumar et al., 2021).
Revised: However, making too many TD updates to the Q-function in ofﬂine deep RL is known to sometimes lead to performance degradation and unlearning, even for otherwise effective modern algorithms (Agarwal et al., 2020; Kumar et al., 2021).

Original: Such unlearning is not typically observed when training overparameterized models via supervised learning, so what about TD learning is responsible for it?
Revised: Such unlearning is not typically observed when training overparameterized models via supervised learning, so what about TD learning is responsible for it?

Original: We show that one possible explanation behind this pathology is the implicit regularization induced by minimizing TD error on a deep Qnetwork.
Revised: We show that one possible explanation behind this pathology is the implicit regularization induced by minimizing TD error on a deep Qnetwork.

Original: Our theoretical results suggest that this implicit regularization “co-adapts” (co-adapts is used qualitatively here, we will discuss our measure shortly) the representations of state-action pairs that appear in a Bellman backup, and is exacerbated when utilizing unseen state-action pairs for the Bellman backup.
Revised: Our theoretical results suggest that this implicit regularization “co-adapts” (co-adapts is used qualitatively here, we will discuss our measure shortly) the representations of state-action pairs that appear in a Bellman backup, and is exacerbated when utilizing unseen state-action pairs for the Bellman backup.

Original: Empirically, this typically manifests as “co-adapted” features for consecutive state-action tuples, even with specialized TD-learning algorithms that account for distributional shift.
Revised: Empirically, this typically manifests as “co-adapted” features for consecutive state-action tuples, even with specialized TD-learning algorithms that account for distributional shift.

Original: Highly co-adapted features in turn lead to poor solutions as we will theoretically and empirically show.
Revised: Highly co-adapted features in turn lead to poor solutions as we will theoretically and empirically show.

Original: We ﬁrst provide empirical evidence of the existence of this co-adaptation phenomenon in Section 3.1 (additional evidence in Appendix A.1) and then theoretically characterize the implicit regularization in TD learning, and discuss how it can explain the phenomenon in Section 3.2.
Revised: We ﬁrst provide empirical evidence of the existence of this co-adaptation phenomenon in Section 3.1 (additional evidence in Appendix A.1) and then theoretically characterize the implicit regularization in TD learning, and discuss how it can explain the phenomenon in Section 3.2.

Original: 3. F EATURE C O -A DAPTATION A ND H OW I T R ELATES T O I MPLICIT R EGULARIZATION
Revised: 3. F EATURE C O -A DAPTATION A ND H OW I T R ELATES T O I MPLICIT R EGULARIZATION

Original: In this section, we empirically identify a feature co-adaptation phenomenon that appears when training value functions via bootstrapping, where the feature representations of consecutive stateaction pairs become excessively similar as measured by their dot product ϕ
Revised: In this section, we empirically identify a feature co-adaptation phenomenon that appears when training value functions via bootstrapping, where the feature representations of consecutive stateaction pairs become excessively similar as measured by their dot product ϕ

Original: ( s , a ) ⊤ ϕ
Revised: ( s , a ) ⊤ ϕ

Original: ( s ′ , a ′ ) .
Revised: ( s ′ , a ′ ) .

Original: Feature co-adaptation appears even when there is no explicit afﬁnity to increase feature similarity.
Revised: Feature co-adaptation appears even when there is no explicit afﬁnity to increase feature similarity.

Original: Experimental setup.
Revised: Experimental setup.

Original: To empirically observe feature co-adaptation, we ran supervised regression and three variants of approximate dynamic programming (ADP) on an ofﬂine dataset consisting of 1% of uniformly-sampled data from the replay buffer of DQN on two Atari games, previously used in Agarwal et al.
Revised: To empirically observe feature co-adaptation, we ran supervised regression and three variants of approximate dynamic programming (ADP) on an ofﬂine dataset consisting of 1% of uniformly-sampled data from the replay buffer of DQN on two Atari games, previously used in Agarwal et al.

Original: First, for comparison, we trained a Q-function via supervised regression to Monte-Carlo (MC) return estimates on the ofﬂine dataset to estimate the value of the behavior policy.
Revised: First, for comparison, we trained a Q-function via supervised regression to Monte-Carlo (MC) return estimates on the ofﬂine dataset to estimate the value of the behavior policy.

Original: Then, we trained variants of ADP which differ in the selection procedure for the action a ′ that appears in the target value in L TD ( θ ) (Equation 1).
Revised: Then, we trained variants of ADP which differ in the selection procedure for the action a ′ that appears in the target value in L TD ( θ ) (Equation 1).

Original: The ofﬂine SARSA variant aims to estimate the value of the behavior policy, Q π β , and sets a ′ to the actual action observed at the next time step in the dataset, such that ( s ′ , a ′ ) ∈ D .
Revised: The ofﬂine SARSA variant aims to estimate the value of the behavior policy, Q π β , and sets a ′ to the actual action observed at the next time step in the dataset, such that ( s ′ , a ′ ) ∈ D .

Original: The TD-learning variant also aims to estimate the value of the behavior policy, but utilizes the expectation of action a ′ sampled from the behavior policy π β , a ′ ∼ π β ( ·| s ′ ) .
Revised: The TD-learning variant also aims to estimate the value of the behavior policy, but utilizes the expectation of action a ′ sampled from the behavior policy π β , a ′ ∼ π β ( ·| s ′ ) .

Original: We do not have access to the functional form of π β since it is induced by the replay buffer of an online DQN, so we train a model for this policy using supervised learning.
Revised: We do not have access to the functional form of π β since it is induced by the replay buffer of an online DQN, so we train a model for this policy using supervised learning.

Original: In
Revised: In

Original: Appendix A.8.2 we show that our ﬁndings hold when the actual behavior policy is available for
Revised: Appendix A.8.2 we show that our ﬁndings hold when the actual behavior policy is available for

Original: TD-learning.
Revised: TD-learning.

Original: We also train Q-learning , which chooses the action a ′ to maximize the learned Qfunction.
Revised: We also train Q-learning , which chooses the action a ′ to maximize the learned Qfunction.

Original: While Q-learning learns a different Q-function, we can still compare the relative stability of these methods to gain intuition about the learning dynamics.
Revised: While Q-learning learns a different Q-function, we can still compare the relative stability of these methods to gain intuition about the learning dynamics.

Original: In addition to feature dot products ϕ ( s , a ) ⊤ ϕ
Revised: In addition to feature dot products ϕ ( s , a ) ⊤ ϕ

Original: ( s ′ , a ′ ) , we also track the average prediction of the Q-network over the dataset to measure whether the predictions diverge or are stable in expectation.
Revised: ( s ′ , a ′ ) , we also track the average prediction of the Q-network over the dataset to measure whether the predictions diverge or are stable in expectation.

Original: Observing feature co-
Revised: Observing feature co-

Original: adaptation empirically.
Revised: adaptation empirically.

Original: As shown in Figure 1 (right), the average dot product (top row) between features at consecutive state-action tuples continuously increases for both
Revised: As shown in Figure 1 (right), the average dot product (top row) between features at consecutive state-action tuples continuously increases for both

Original: Q-learning and TD-learning (after enough gradient steps), whereas it ﬂatlines and converges to a small value for supervised regression.
Revised: Q-learning and TD-learning (after enough gradient steps), whereas it ﬂatlines and converges to a small value for supervised regression.

Original: We might at ﬁrst think that this is simply a case of Q-learning failing to converge.
Revised: We might at ﬁrst think that this is simply a case of Q-learning failing to converge.

Original: However, the bottom row shows that the average Q-values do in fact converge to a stable value.
Revised: However, the bottom row shows that the average Q-values do in fact converge to a stable value.

Original: Despite this, the optimizer drives the network towards higher feature dot products.
Revised: Despite this, the optimizer drives the network towards higher feature dot products.

Original: There is no explicit term in the TD error objective that encourages this behavior, indicating the presence of some implicit regularization phenomenon.
Revised: There is no explicit term in the TD error objective that encourages this behavior, indicating the presence of some implicit regularization phenomenon.

Original: This implicit preference towards maximizing the dot products of features at consecutive state-action tuples is what we call “feature co-adaptation.”
Revised: This implicit preference towards maximizing the dot products of features at consecutive state-action tuples is what we call “feature co-adaptation.”

Original: When does feature co-adaptation emerge?
Revised: When does feature co-adaptation emerge?

Original: Observe in Figure 1 (right) that the feature dot products for ofﬂine SARSA converge quickly and are relatively ﬂat, similarly to supervised regression.
Revised: Observe in Figure 1 (right) that the feature dot products for ofﬂine SARSA converge quickly and are relatively ﬂat, similarly to supervised regression.

Original: This indicates that utilizing a bootstrapped update alone is not responsible for the increasing dot-products and instability, because while ofﬂine SARSA uses backups, it behaves similarly to supervised MC regression.
Revised: This indicates that utilizing a bootstrapped update alone is not responsible for the increasing dot-products and instability, because while ofﬂine SARSA uses backups, it behaves similarly to supervised MC regression.

Original: Unlike ofﬂine SARSA, feature co-adaptation emerges for TD-learning, which is surprising as TD-learning also aims to estimate the value of the behavior policy, and hence should match ofﬂine SARSA in expectation.
Revised: Unlike ofﬂine SARSA, feature co-adaptation emerges for TD-learning, which is surprising as TD-learning also aims to estimate the value of the behavior policy, and hence should match ofﬂine SARSA in expectation.

Original: The key difference is that while ofﬂine SARSA always utilizes actions a ′ observed in the training dataset for the backup, TD-learning may utilize potentially unseen actions a ′ in the backup, even though these actions a ′ ∼ π β ( ·| s ′ ) are within the distribution of the data-generating policy.
Revised: The key difference is that while ofﬂine SARSA always utilizes actions a ′ observed in the training dataset for the backup, TD-learning may utilize potentially unseen actions a ′ in the backup, even though these actions a ′ ∼ π β ( ·| s ′ ) are within the distribution of the data-generating policy.

Original: This suggests that utilizing out-of-sample actions, even when they are not out-of-distribution, in the Bellman backup critically alters the learning dynamics.
Revised: This suggests that utilizing out-of-sample actions, even when they are not out-of-distribution, in the Bellman backup critically alters the learning dynamics.

Original: This ﬁnding is absent from literature in ofﬂine RL which primarily focuses on the impact of out-of-distribution actions (Levine et al., 2020), but not out-of-sample actions.
Revised: This ﬁnding is absent from literature in ofﬂine RL which primarily focuses on the impact of out-of-distribution actions (Levine et al., 2020), but not out-of-sample actions.

Original: The theoretical model developed in
Revised: The theoretical model developed in

Original: Section 3.2 will provide an explanation for this observation with a discussion about how feature co-adaption caused due to out-of-sample actions can be detrimental in ofﬂine RL.
Revised: Section 3.2 will provide an explanation for this observation with a discussion about how feature co-adaption caused due to out-of-sample actions can be detrimental in ofﬂine RL.

Original: 3. T
Revised: 3. T

Original: [Equation]
Revised: [Equation]

Original: C HARACTERIZING I MPLICIT R EGULARIZATION IN TD-L EARNING
Revised: C HARACTERIZING I MPLICIT R EGULARIZATION IN TD-L EARNING

Original: Why does feature co-adaptation emerge in TD-learning and what do out-of-sample actions have to do with it?
Revised: Why does feature co-adaptation emerge in TD-learning and what do out-of-sample actions have to do with it?

Original: To answer this question, we theoretically characterize the implicit regularization effects in TD-learning.
Revised: To answer this question, we theoretically characterize the implicit regularization effects in TD-learning.

Original: We analyze the learning dynamics of TD learning in the overparameterized regime, where there are many different parameter vectors θ that fully minimize the training set temporal difference error.
Revised: We analyze the learning dynamics of TD learning in the overparameterized regime, where there are many different parameter vectors θ that fully minimize the training set temporal difference error.

Original: When training an overparameterized f θ ( x ) via supervised regression using the squared loss, many different values of θ will satisfy L ( θ ) = 0 on the training set due to overparameterization, but Blanc et al.
Revised: When training an overparameterized f θ ( x ) via supervised regression using the squared loss, many different values of θ will satisfy L ( θ ) = 0 on the training set due to overparameterization, but Blanc et al.

Original: ; Damian et al.
Revised: ; Damian et al.

Original: (2021) argue that the dynamics of noisy
Revised: (2021) argue that the dynamics of noisy

Original: SGD will result in only those ﬁxed points to be stable that additionally satisfy ∇ θ R ( θ ∗ )
Revised: SGD will result in only those ﬁxed points to be stable that additionally satisfy ∇ θ R ( θ ∗ )

Original: = 0 , for a particular implicit regularizer R ( θ ) .
Revised: = 0 , for a particular implicit regularizer R ( θ ) .

Original: The noisy gradient updates analyzed in this model have the form:
Revised: The noisy gradient updates analyzed in this model have the form:

Original: [Equation]
Revised: [Equation]

Original: and Blanc et al.
Revised: and Blanc et al.

Original: ; Damian et al.
Revised: ; Damian et al.

Original: (2021) show that when the regression targets are corrupted with N (0 , 1) label noise, then M = P |D| i =1 ∇ θ f θ ( x i ) ∇ θ f θ ( x i ) ⊤ and the induced implicit regularizer is given by R ( θ ) =
Revised: (2021) show that when the regression targets are corrupted with N (0 , 1) label noise, then M = P |D| i =1 ∇ θ f θ ( x i ) ∇ θ f θ ( x i ) ⊤ and the induced implicit regularizer is given by R ( θ ) =

Original: η P |D| i ||∇ θ f θ ( x i )
Revised: η P |D| i ||∇ θ f θ ( x i )

Original: ||
Revised: ||

Original: Any solution θ ∗ found by Equation 2 must satisfy ∇ θ R ( θ ∗ ) = 0 .
Revised: Any solution θ ∗ found by Equation 2 must satisfy ∇ θ R ( θ ∗ ) = 0 .

Original: This model corroborates empirical ﬁndings (Mulayoff & Michaeli, 2020) about the solutions found by SGD with deep nets, which motivates our use of this framework.
Revised: This model corroborates empirical ﬁndings (Mulayoff & Michaeli, 2020) about the solutions found by SGD with deep nets, which motivates our use of this framework.

Original: Following this framework, we analyze the ﬁxed points of noisy TD-learning.
Revised: Following this framework, we analyze the ﬁxed points of noisy TD-learning.

Original: We consider noisy pseudo-gradient
Revised: We consider noisy pseudo-gradient

Original: TD updates with a general noise covariance M :
Revised: TD updates with a general noise covariance M :

Original: [Equation]
Revised: [Equation]

Original: We use a deterministic policy a ′ i
Revised: We use a deterministic policy a ′ i

Original: =
Revised: =

Original: π ( s ′ i ) to simplify exposition.
Revised: π ( s ′ i ) to simplify exposition.

Original: Following Damian et al.
Revised: Following Damian et al.

Original: (2021), we can set the noise model M as M = P i ∇ θ Q ( s i , a i ) ∇ θ Q ( s i , a i ) ⊤ , or utilize a different choice of M , but we will derive the general form ﬁrst.
Revised: (2021), we can set the noise model M as M = P i ∇ θ Q ( s i , a i ) ∇ θ Q ( s i , a i ) ⊤ , or utilize a different choice of M , but we will derive the general form ﬁrst.

Original: Let θ ∗ denote a stationary point of the training TD error, such that the pseudo-gradient g ( θ ∗ ) = 0 .
Revised: Let θ ∗ denote a stationary point of the training TD error, such that the pseudo-gradient g ( θ ∗ ) = 0 .

Original: Further, we denote to the derivative of g ( θ ) w.r.t.
Revised: Further, we denote to the derivative of g ( θ ) w.r.t.

Original: θ as matrix G ( θ ) and refer to as the pseudo-Hessian : although G ( θ ) is not actually the second derivative of any well-deﬁned objective, since TD updates are not proper gradient updates, as we will see it will play a similar role to the Hessian in gradient descent.
Revised: θ as matrix G ( θ ) and refer to as the pseudo-Hessian : although G ( θ ) is not actually the second derivative of any well-deﬁned objective, since TD updates are not proper gradient updates, as we will see it will play a similar role to the Hessian in gradient descent.

Original: For brevity, deﬁne G =
Revised: For brevity, deﬁne G =

Original: G ( θ ∗ ) , g = g ( θ ∗ ) and ∇ G = ∇ θ G ( θ ∗ ) .
Revised: G ( θ ∗ ) , g = g ( θ ∗ ) and ∇ G = ∇ θ G ( θ ∗ ) .

Original: Also, λ i ( P ) denotes the i -th eigenvalue of matrix P .
Revised: Also, λ i ( P ) denotes the i -th eigenvalue of matrix P .

Original: Assumptions.
Revised: Assumptions.

Original: To simplify analysis, we assume that matrices G and M (i.e., the noise covariance matrix) span the same n -dimensional basis in d -dimensional space, where d is the number of param- eters and n is the number of datapoints, and n ≪
Revised: To simplify analysis, we assume that matrices G and M (i.e., the noise covariance matrix) span the same n -dimensional basis in d -dimensional space, where d is the number of param- eters and n is the number of datapoints, and n ≪

Original: d due to overparameterization.
Revised: d due to overparameterization.

Original: We also require θ ∗ to satisfy a technical criterion that requires approximate alignment between the eigenspaces of G and the gradient of the Q-function, without which noisy TD may not be stable at θ ∗ .
Revised: We also require θ ∗ to satisfy a technical criterion that requires approximate alignment between the eigenspaces of G and the gradient of the Q-function, without which noisy TD may not be stable at θ ∗ .

Original: We summarize all the assumptions in Appendix C, and ﬁrst present our regularizer.
Revised: We summarize all the assumptions in Appendix C, and ﬁrst present our regularizer.

Original: Theorem 3.1 (Implicit regularizer at TD ﬁxed points) .
Revised: Theorem 3.1 (Implicit regularizer at TD ﬁxed points) .

Original: Under the assumptions so far, a ﬁxed point of
Revised: Under the assumptions so far, a ﬁxed point of

Original: TD-learning, θ ∗ , where Q θ ∗ ( s i , a i ) =
Revised: TD-learning, θ ∗ , where Q θ ∗ ( s i , a i ) =

Original: r i + γQ θ ∗ ( s ′
Revised: r i + γQ θ ∗ ( s ′

Original: i , a ′ i ) for every ( s i , a i , s ′ i )
Revised: i , a ′ i ) for every ( s i , a i , s ′ i )

Original: ∈ D is stable if: (1) it satisﬁes Re( λ i ( G )) ≥ 0 , ∀ i and Re( λ i ( G )) > 

Revised: ∈ D is stable if: (1) it satisﬁes Re( λ i ( G )) ≥ 0 , ∀ i and Re( λ i ( G )) >

Original: if | Imag( λ i ( G )) | > 0 , and (2) θ ∗ is the stationary point of the induced implicit regularizer,
Revised: if | Imag( λ i ( G )) | > 0 , and (2) θ ∗ is the stationary point of the induced implicit regularizer,

Original: [Equation]
Revised: [Equation]

Original: trace
Revised: trace

Original: [Equation]
Revised: [Equation]

Original: where ( s i , a i ) and ( s ′ i , a ′ i ) denote state-action pairs that appear together in a Bellman update, [[ □ ]] denotes the stop-gradient function, which does not pass partial derivatives w.r.t.
Revised: where ( s i , a i ) and ( s ′ i , a ′ i ) denote state-action pairs that appear together in a Bellman update, [[ □ ]] denotes the stop-gradient function, which does not pass partial derivatives w.r.t.

Original: θ into □ .
Revised: θ into □ .

Original: Σ ∗ M is the ﬁxed point of the discrete Lyapunov equation: Σ ∗ M :=
Revised: Σ ∗ M is the ﬁxed point of the discrete Lyapunov equation: Σ ∗ M :=

Original: ( I − ηG )Σ ∗ M ( I − ηG ) ⊤ + η 2 M .
Revised: ( I − ηG )Σ ∗ M ( I − ηG ) ⊤ + η 2 M .

Original: A proof of Theorem 3.1 is provided in Appendix C. Next, we explain the intuition behind this result and provide a proof sketch.
Revised: A proof of Theorem 3.1 is provided in Appendix C. Next, we explain the intuition behind this result and provide a proof sketch.

Original: To derive the induced implicit regularizer for a stable ﬁxed point θ ∗ of TD error, we study the learning dynamics of noisy TD learning (Equation 3) initialized at θ ∗ , and derive conditions under which this noisy update would stay close to θ ∗ with multiple updates.
Revised: To derive the induced implicit regularizer for a stable ﬁxed point θ ∗ of TD error, we study the learning dynamics of noisy TD learning (Equation 3) initialized at θ ∗ , and derive conditions under which this noisy update would stay close to θ ∗ with multiple updates.

Original: This gives rise to the two conditions shown in Theorem 3.1 which can be understood as controlling stability in mutually exclusive directions in the parameter space.
Revised: This gives rise to the two conditions shown in Theorem 3.1 which can be understood as controlling stability in mutually exclusive directions in the parameter space.

Original: If condition (1) is not satisﬁed, then even under-parameterized TD will diverge away from θ ∗ , since I − ηG would be a non-contraction as the spectral radius, ρ ( I − ηG ) ≥ 1 in that case.
Revised: If condition (1) is not satisﬁed, then even under-parameterized TD will diverge away from θ ∗ , since I − ηG would be a non-contraction as the spectral radius, ρ ( I − ηG ) ≥ 1 in that case.

Original: Thus, θ k − θ ∗ will grow or not decrease in some direction.
Revised: Thus, θ k − θ ∗ will grow or not decrease in some direction.

Original: When (1) is satisﬁed for all directions in the parameter space, there are still directions where both the real and imaginary parts of the eigenvalue are 0 due to overparameterization.
Revised: When (1) is satisﬁed for all directions in the parameter space, there are still directions where both the real and imaginary parts of the eigenvalue are 0 due to overparameterization.

Original: In such directions, learning is governed by the projection of the noise under the third-order derivative ∇ G , which appears in the Taylor expansion of θ k − θ ∗ around the point θ ∗ :
Revised: In such directions, learning is governed by the projection of the noise under the third-order derivative ∇ G , which appears in the Taylor expansion of θ k − θ ∗ around the point θ ∗ :

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne ν
Revised: where we deﬁne ν

Original: k := θ k − θ ∗ .
Revised: k := θ k − θ ∗ .

Original: θ ∗ is stable if it is a stationary point of the implicit regularizer R TD (condition (2) ), which ensures that total noise (i.e., accumulated ε k over iterations k ) does not lead to a large deviation in ν k .
Revised: θ ∗ is stable if it is a stationary point of the implicit regularizer R TD (condition (2) ), which ensures that total noise (i.e., accumulated ε k over iterations k ) does not lead to a large deviation in ν k .

Original: Interpretation of Theorem 3.1.
Revised: Interpretation of Theorem 3.1.

Original: While the choice of the noise model M will change the form of the implicit regularizer, of course, in practice, the form of M is not known.
Revised: While the choice of the noise model M will change the form of the implicit regularizer, of course, in practice, the form of M is not known.

Original: We can consider other choices of M for interpretation, but Theorem 3.1 can be easy to qualitatively interpret for M such that Σ ∗ M
Revised: We can consider other choices of M for interpretation, but Theorem 3.1 can be easy to qualitatively interpret for M such that Σ ∗ M

Original: = I .
Revised: = I .

Original: In this case, we ﬁnd that the implicit preference towards local minima of R TD ( θ ) can explain feature co-adaptation.
Revised: In this case, we ﬁnd that the implicit preference towards local minima of R TD ( θ ) can explain feature co-adaptation.

Original: In this case, the regularizer takes a simpler form: R TD ( θ ) :=
Revised: In this case, the regularizer takes a simpler form: R TD ( θ ) :=

Original: P i ||∇ Q
Revised: P i ||∇ Q

Original: θ ( s i , a i ) ||
Revised: θ ( s i , a i ) ||

Original: 2 2 − γ ∇ Q θ ( s i , a i ) ∇ [[ Q θ ( s ′
Revised: 2 2 − γ ∇ Q θ ( s i , a i ) ∇ [[ Q θ ( s ′

Original: i , a ′ i )]] .
Revised: i , a ′ i )]] .

Original: The ﬁrst term is equal to the squared perdatapoint gradient norm, which is same as the implicit regularizer in supervised learning obtained by
Revised: The ﬁrst term is equal to the squared perdatapoint gradient norm, which is same as the implicit regularizer in supervised learning obtained by

Original: Blanc et al.
Revised: Blanc et al.

Original: ; Damian et al.
Revised: ; Damian et al.

Original: (2021) with label noise.
Revised: (2021) with label noise.

Original: However, R TD ( θ ) additionally includes a second term that is equal to the dot product of the gradient of the Q-function at the current and next states, ∇ θ Q θ ( s i , a i )
Revised: However, R TD ( θ ) additionally includes a second term that is equal to the dot product of the gradient of the Q-function at the current and next states, ∇ θ Q θ ( s i , a i )

Original: ⊤ ∇ θ Q θ ( s ′
Revised: ⊤ ∇ θ Q θ ( s ′

Original: i , a ′ i ) , and thus this term is effectively maximized .
Revised: i , a ′ i ) , and thus this term is effectively maximized .

Original: When restricted to the last-layer parameters of a neural network, this term is equal to the dot product of the features at consecutive state-action tuples: P i ∇ θ Q θ ( s i , a i )
Revised: When restricted to the last-layer parameters of a neural network, this term is equal to the dot product of the features at consecutive state-action tuples: P i ∇ θ Q θ ( s i , a i )

Original: ⊤ ∇ θ Q θ ( s ′
Revised: ⊤ ∇ θ Q θ ( s ′

Original: i , a ′ i )
Revised: i , a ′ i )

Original: =
Revised: =

Original: P i ϕ
Revised: P i ϕ

Original: ( s i , a i )
Revised: ( s i , a i )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ( s ′ i , a ′ i ) .
Revised: ( s ′ i , a ′ i ) .

Original: The tendency to maximize this quantity to attain a local minimizer of the implicit regularizer corroborates the empirical ﬁndings of increased dot product in Section 3.1.
Revised: The tendency to maximize this quantity to attain a local minimizer of the implicit regularizer corroborates the empirical ﬁndings of increased dot product in Section 3.1.

Original: Explaining the difference between utilizing seen and unseen actions in the backup.
Revised: Explaining the difference between utilizing seen and unseen actions in the backup.

Original: If all stateaction pairs ( s ′ i , a ′ i ) appearing on the right-hand-side of the Bellman update also appear in the dataset D , as in the case of ofﬂine SARSA (Figure 1), the preference to increase dot products will be balanced by the afﬁnity to reduce gradient norm (ﬁrst term of R TD ( θ ) when Σ ∗ M
Revised: If all stateaction pairs ( s ′ i , a ′ i ) appearing on the right-hand-side of the Bellman update also appear in the dataset D , as in the case of ofﬂine SARSA (Figure 1), the preference to increase dot products will be balanced by the afﬁnity to reduce gradient norm (ﬁrst term of R TD ( θ ) when Σ ∗ M

Original: = I ): for example, when ( s ′ i , a ′ i ) are permutations of ( s i , a i ) , R TD is lower bounded by (1 − γ ) P i ||∇ θ Q θ ( x i )
Revised: = I ): for example, when ( s ′ i , a ′ i ) are permutations of ( s i , a i ) , R TD is lower bounded by (1 − γ ) P i ||∇ θ Q θ ( x i )

Original: ||
Revised: ||

Original: 22 and hence minimizing R TD ( θ ) would minimize the feature norm instead of maximizing dot products.
Revised: 22 and hence minimizing R TD ( θ ) would minimize the feature norm instead of maximizing dot products.

Original: Note that this also corresponds to the implicit regularizer we would obtain when training Q-functions via supervised learning and hence, our analysis would predict that ofﬂine SARSA with in-sample actions (i.e., when ( s ′ , a ′ ) ∈ D ) would behave similarly to supervised regression.
Revised: Note that this also corresponds to the implicit regularizer we would obtain when training Q-functions via supervised learning and hence, our analysis would predict that ofﬂine SARSA with in-sample actions (i.e., when ( s ′ , a ′ ) ∈ D ) would behave similarly to supervised regression.

Original: However, the regularizer behaves very differently when unseen state-action pairs ( s ′ i , a ′ i ) appear only on the right-hand-side of the backup.
Revised: However, the regularizer behaves very differently when unseen state-action pairs ( s ′ i , a ′ i ) appear only on the right-hand-side of the backup.

Original: This happens with any algorithm where a ′ is not the dataset action, which is the case for all deep RL algorithms that compute target values by selecting a ′ according to the current policy.
Revised: This happens with any algorithm where a ′ is not the dataset action, which is the case for all deep RL algorithms that compute target values by selecting a ′ according to the current policy.

Original: In this case, we expect the dot product of gradients at ( s , a ) and ( s ′ , a ′ ) to be large at any attractive ﬁxed point, since this minimizes R TD ( θ ) .
Revised: In this case, we expect the dot product of gradients at ( s , a ) and ( s ′ , a ′ ) to be large at any attractive ﬁxed point, since this minimizes R TD ( θ ) .

Original: This is precisely a form of co-adaptation: gradients at out-of-sample state-action tuples are highly similar to gradients at observed state-action pairs measured by the dot product .
Revised: This is precisely a form of co-adaptation: gradients at out-of-sample state-action tuples are highly similar to gradients at observed state-action pairs measured by the dot product .

Original: This observation is also supported by the analysis in Section 3.1.
Revised: This observation is also supported by the analysis in Section 3.1.

Original: Finally, note that the choice of M is a modelling assumption, and to derive our explicit regularizer, later in the paper, we will make a simplifying choice of M , though we empirically verify that a different M also performs well (Appendix A.5).
Revised: Finally, note that the choice of M is a modelling assumption, and to derive our explicit regularizer, later in the paper, we will make a simplifying choice of M , though we empirically verify that a different M also performs well (Appendix A.5).

Original: Why implicit regularization and hence feature co-adaptation can be detrimental to policy performance?
Revised: Why implicit regularization and hence feature co-adaptation can be detrimental to policy performance?

Original: To answer this question, we present theoretical and empirical evidence that depicts the adverse effects of this implicit regularizer in TD-learning.
Revised: To answer this question, we present theoretical and empirical evidence that depicts the adverse effects of this implicit regularizer in TD-learning.

Original: Empirically, we ran two algorithms,
Revised: Empirically, we ran two algorithms,

Original: DQN and CQL initialized from a high-performing Q-function checkpoint obtained using DR3 (Section 4), which attains relatively small feature dot products (i.e., the second term of R TD ( θ ) is small).
Revised: DQN and CQL initialized from a high-performing Q-function checkpoint obtained using DR3 (Section 4), which attains relatively small feature dot products (i.e., the second term of R TD ( θ ) is small) (details in Appendix A.9).

Original: Our goal is to see if TD updates starting from such a “good”initialization still stay around it or diverge to poorer solutions.
Revised: Our goal is to see if TD updates starting from such a “good”initialization still stay around it or diverge to poorer solutions.

Original: Our theoretical analysis in Section 3.2 would predict that TD learning would destabilize from such a solution since it would not be a stable ﬁxed point.
Revised: Our theoretical analysis in Section 3.2 would predict that TD learning would destabilize from such a solution since it would not be a stable ﬁxed point.

Original: Indeed, as shown in Figure 2, the policy starts to degrade right from the beginning, and the the dot-product similarities start to increase.
Revised: Indeed, as shown in Figure 2, the policy starts to degrade right from the beginning, and the the dot-product similarities start to increase.

Original: This even happens with CQL, which explicitly corrects for any distributional shift confounds, implying that the performance drop cannot be directly explained by the typical out-ofdistribution action explanations.
Revised: This even happens with CQL, which explicitly corrects for any distributional shift confounds, implying that the performance drop cannot be directly explained by the typical out-of-distribution action explanations.

Original: To investigate the reasons behind this drop, we also measured the overall training error for these algorithms (i.e., TD error for DQN and TD error + CQL regularizer for CQL) and ﬁnd in Figure 2 that the loss values are generally small for both CQL and DQN.
Revised: To investigate the reasons behind this drop, we also measured the overall training error for these algorithms (i.e., TD error for DQN and TD error + CQL regularizer for CQL) and ﬁnd in Figure 2 that the loss values are generally small for both CQL and DQN.

Original: In particular, the values attained by TD error are small and do not increase signiﬁcantly, indicating that the preference to increase dot products is a consequence of an implicit phenomenon and is not explained by smaller or larger values of the training loss.
Revised: In particular, the values attained by TD error are small and do not increase signiﬁcantly, indicating that the preference to increase dot products is a consequence of an implicit phenomenon and is not explained by smaller or larger values of the training loss.

Original: To motivate why co-adapted features can lead to poor performance in TD-learning, we study the convergence of linear TD-learning on co-adapted features.
Revised: To motivate why co-adapted features can lead to poor performance in TD-learning, we study the convergence of linear TD-learning on co-adapted features.

Original: Our theoretical result characterizes a lower bound on the feature dot products in terms of the feature norms for state-action pairs in the dataset D , which if satisﬁed, will inhibit convergence:
Revised: Our theoretical result characterizes a lower bound on the feature dot products in terms of the feature norms for state-action pairs in the dataset D , which if satisﬁed, will inhibit convergence:

Original: Proposition 3.2 (TD-learning on co-adapted features) .
Revised: Proposition 3.2 (TD-learning on co-adapted features) .

Original: Assume that the features Φ =
Revised: Assume that the features Φ =

Original: [ ϕ ( s , a )] s , a are used for linear TD-learning.
Revised: [ ϕ ( s , a )] s , a are used for linear TD-learning.

Original: Then, if P s , a , s ′ ∈D ϕ ( s , a ) ⊤ ϕ ( s ′ , a ′ ) ≥ 1 γ P s , a ∈D ϕ ( s , a ) ⊤ ϕ ( s , a ) , linear TD-learning using features Φ will not converge.
Revised: Then, if P s , a , s ′ ∈D ϕ ( s , a ) ⊤ ϕ ( s ′ , a ′ ) ≥ 1 γ P s , a ∈D ϕ ( s , a ) ⊤ ϕ ( s , a ) , linear TD-learning using features Φ will not converge.

Original: A proof of Proposition 3.2 is provided in Appendix D and it relies on a stability analysis of linear TD.
Revised: A proof of Proposition 3.2 is provided in Appendix D and it relies on a stability analysis of linear TD.

Original: While features change during training for TD-learning with neural networks, and arguably linear TD is a simple model to study consequences of co-adapted features, even in this simple linear setting,
Revised: While features change during training for TD-learning with neural networks, and arguably linear TD is a simple model to study consequences of co-adapted features, even in this simple linear setting,

Original: Proposition 3.2 indicates that TD-learning may be non-convergent as a result of co-adaptation.
Revised: Proposition 3.2 indicates that TD-learning may be non-convergent as a result of co-adaptation.

Original:  DR3: E XPLICIT R EGULARIZATION FOR D EEP TD-L EARNING
Revised:  DR3: E XPLICIT R EGULARIZATION FOR D EEP TD-L EARNING

Original: Since the implicit regularization effects in TD-learning can lead to feature co-adaptation, which in turn is correlated with poor performance, can we instead derive an explicit regularizer to alleviate this issue?
Revised: Since the implicit regularization effects in TD-learning can lead to feature co-adaptation, which in turn is correlated with poor performance, can we instead derive an explicit regularizer to alleviate this issue?

Original: Inspired by the analysis in the previous section, we will propose an explicit regularizer that attempts to counteract the second term in Equation 4, which would otherwise lead to coadaptation and poor representations.
Revised: Inspired by the analysis in the previous section, we will propose an explicit regularizer that attempts to counteract the second term in Equation 4, which would otherwise lead to coadaptation and poor representations.

Original: The explicit regularizer that offsets the difference between the two implicit regularizers is given by: ∆( θ ) =
Revised: The explicit regularizer that offsets the difference between the two implicit regularizers is given by: ∆( θ ) =

Original: P i trace (cid:2)
Revised: P i trace (cid:2)

Original: Σ ∗⊤ M ∇ θ Q θ ( s i , a i ) ∇ θ Q θ ( s ′
Revised: Σ ∗⊤ M ∇ θ Q θ ( s i , a i ) ∇ θ Q θ ( s ′

Original: i , a ′ i ) ⊤ (cid:3) , which represents the second term of R TD ( θ ) .
Revised: i , a ′ i ) ⊤ (cid:3) , which represents the second term of R TD ( θ ) .

Original: Note that we drop the stop gradient on Q θ ( s ′ i , a ′ i ) in ∆( θ ) as it performs slightly better in practice (Table A.1), although as shown in that Table, the version with the stop gradient also signiﬁcantly improves over the base method.
Revised: Note that we drop the stop gradient on Q θ ( s ′ i , a ′ i ) in ∆( θ ) as it performs slightly better in practice (Table A.1), although as shown in that Table, the version with the stop gradient also signiﬁcantly improves over the base method.

Original: The ﬁrst term of R TD ( θ ) corresponds to the regularizer from supervised learning.
Revised: The ﬁrst term of R TD ( θ ) corresponds to the regularizer from supervised learning.

Original: Our proposed method, DR3, simply combines approximations to ∆( θ ) with various ofﬂine RL algorithms.
Revised: Our proposed method, DR3, simply combines approximations to ∆( θ ) with various ofﬂine RL algorithms.

Original: For any ofﬂine RL algorithm, A LG , with objective L A LG ( θ ) , the training objective with DR3 is given by: L ( θ ) :
Revised: For any ofﬂine RL algorithm, A LG , with objective L A LG ( θ ) , the training objective with DR3 is given by: L ( θ ) :

Original: =
Revised: =

Original: L A LG ( θ ) + c 0 ∆( θ ) , where c 0 is the DR3 coefﬁcient.
Revised: L A LG ( θ ) + c 0 ∆( θ ) , where c 0 is the DR3 coefﬁcient.

Original: See Appendix E.3 for details on how we tune c 0 in this paper.
Revised: See Appendix E.3 for details on how we tune c 0 in this paper.

Original: Practical version of DR3.
Revised: Practical version of DR3.

Original: In order to practically instantiate DR3, we need to choose a particular noise model M .
Revised: In order to practically instantiate DR3, we need to choose a particular noise model M .

Original: In general, it is not possible to know beforehand the “correct” choice of M (Equation 3), even in supervised learning, as this is a complicated function of the data distribution, neural network architecture and initialization.
Revised: In general, it is not possible to know beforehand the “correct” choice of M (Equation 3), even in supervised learning, as this is a complicated function of the data distribution, neural network architecture and initialization.

Original: Therefore, we instantiate DR3 with two heuristic choices of M :
Revised: Therefore, we instantiate DR3 with two heuristic choices of M :

Original: (i) M induced by label noise studied in prior work for supervised learning and for which we need to run computationally heavy ﬁxed-point computation for M , and (ii) a simpler alternative that sets Σ ∗ M
Revised: (i) M induced by label noise studied in prior work for supervised learning and for which we need to run computationally heavy ﬁxed-point computation for M , and (ii) a simpler alternative that sets Σ ∗ M

Original: = I .
Revised: = I .

Original: We ﬁnd that both of these variants generally perform equally well empirically (Figure 6), and so we utilize (ii) in practice due to low computational costs.
Revised: We ﬁnd that both of these variants generally perform equally well empirically (Figure 6), and so we utilize (ii) in practice due to low computational costs.

Original: Additionally, because computing and backpropagating through per-example gradient dot products is slow, we instead approximate ∆( θ ) with the contribution only from the last layer parameters ( i.e. , P i ∇ w Q θ ( s i , a i )
Revised: Additionally, because computing and backpropagating through per-example gradient dot products is slow, we instead approximate ∆( θ ) with the contribution only from the last layer parameters ( i.e. , P i ∇ w Q θ ( s i , a i )

Original: ⊤ ∇ w Q θ ( s ′
Revised: ⊤ ∇ w Q θ ( s ′

Original: i , a ′ i ) ), similarly to tractable Bayesian neural nets.
Revised: i , a ′ i ) ), similarly to tractable Bayesian neural nets.

Original: As shown in Appendix A.5, the practical version of DR3 performs similarly to the theoretically derived version.
Revised: As shown in Appendix A.5, the practical version of DR3 performs similarly to the theoretically derived version.

Original: Explicit DR3 regularizer :
Revised: Explicit DR3 regularizer :

Original: [Equation]
Revised: [Equation]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior analyses of learning dynamics in RL has focused primarily on analyzing error propagation in tabular or linear settings ( e.g. , Chen & Jiang, 2019; Duan et al., 2020; Xie & Jiang, 2020;
Revised: Prior analyses of learning dynamics in RL has focused primarily on analyzing error propagation in tabular or linear settings ( e.g. , Chen & Jiang, 2019; Duan et al., 2020; Xie & Jiang, 2020;

Original: Wang et al., 2021a;b; Farahmand et al., 2010; De Farias, 2002), understanding instabilities in deep
Revised: Wang et al., 2021a;b; Farahmand et al., 2010; De Farias, 2002), understanding instabilities in deep

Original: RL (Achiam et al., 2019; Bengio et al., 2020; Kumar et al., 2020a; Van Hasselt et al., 2018) and deriving weighted TD updates that enjoy convergence guarantees (Maei et al., 2009; Mahmood et al., 2016;
Revised: RL (Achiam et al., 2019; Bengio et al., 2020; Kumar et al., 2020a; Van Hasselt et al., 2018) and deriving weighted TD updates that enjoy convergence guarantees (Maei et al., 2009; Mahmood et al., 2016;

Original: Sutton et al., 2016), but these methods do not reason about implicit regularization or any form of representation learning.
Revised: Sutton et al., 2016), but these methods do not reason about implicit regularization or any form of representation learning.

Original: Ghosh & Bellemare (2020) focuses on understanding the stability of
Revised: Ghosh & Bellemare (2020) focuses on understanding the stability of

Original: TD-learning in underparameterized linear settings, whereas our focus is on the overparameterized setting, when optimizing TD error and learning representations via SGD.
Revised: TD-learning in underparameterized linear settings, whereas our focus is on the overparameterized setting, when optimizing TD error and learning representations via SGD.

Original: Kumar et al.
Revised: Kumar et al.

Original: (2021) studies the learning dynamics of Q-learning and observes that the rank of the feature matrix, Φ , drops during training.
Revised: (2021) studies the learning dynamics of Q-learning and observes that the rank of the feature matrix, Φ , drops during training.

Original: While this observation is related, our analysis characterizes the implicit preference of learning towards feature co-adaptation (Theorem 3.1) on out-of-sample actions as the primary culprit for aliasing.
Revised: While this observation is related, our analysis characterizes the implicit preference of learning towards feature co-adaptation (Theorem 3.1) on out-of-sample actions as the primary culprit for aliasing.

Original: Additionally, while the goal of our work is not to increase srank (Φ) , utilizing
Revised: Additionally, while the goal of our work is not to increase srank (Φ) , utilizing

Original: DR3 not only outperforms the srank (Φ) penalty in Kumar et al.
Revised: DR3 not only outperforms the srank (Φ) penalty in Kumar et al.

Original: (2021) by more than 100% ,
Revised: (2021) by more than 100% ,

Original: but it also alleviates rank collapse.
Revised: but it also alleviates rank collapse.

Original: Somewhat related to DR3, Durugkar & Stone (2018)
Revised: Somewhat related to DR3, Durugkar & Stone (2018)

Original: ; Pohlen et al.
Revised: ; Pohlen et al.

Original: (2018) heuristically constrain gradients of TD to prevent changes in target Q-values to prevent divergence.
Revised: (2018) heuristically constrain gradients of TD to prevent changes in target Q-values to prevent divergence.

Original: Contrary to such heuristic approaches, DR3 is inspired from a theoretical model of implicit regularization, and does not prevent changes in target values, but rather reduces feature dot products.
Revised: Contrary to such heuristic approaches, DR3 is inspired from a theoretical model of implicit regularization, and does not prevent changes in target values, but rather reduces feature dot products.

Original:  E XPERIMENTAL E VALUATION OF DR 
Revised:  E XPERIMENTAL E VALUATION OF DR

Original: Our experiments aim to evaluate the extent to which DR3 improves performance in ofﬂine RL in practice, and to study its effect on prior observations of rank collapse.
Revised: Our experiments aim to evaluate the extent to which DR3 improves performance in ofﬂine RL in practice, and to study its effect on prior observations of rank collapse.

Original: To this end, we investigate if DR3 improves ofﬂine RL performance and stability on three ofﬂine RL benchmarks: Atarigames with discrete actions (Agarwal et al., 2020), continuous control tasks from D4RL (Fu et al., 2020), and image-based robotic manipulation tasks (Singh et al., 2020).
Revised: To this end, we investigate if DR3 improves ofﬂine RL performance and stability on three ofﬂine RL benchmarks: Atarigames with discrete actions (Agarwal et al., 2020), continuous control tasks from D4RL (Fu et al., 2020), and image-based robotic manipulation tasks (Singh et al., 2020).

Original: Following prior work (Fu et al., 2020; Gulcehre et al., 2020), we evaluate DR3 in terms of ﬁnal ofﬂine RL performance after a given number of iterations.
Revised: Following prior work (Fu et al., 2020; Gulcehre et al., 2020), we evaluate DR3 in terms of ﬁnal ofﬂine RL performance after a given number of iterations.

Original: Additionally, we report training stability , which is important in practice as ofﬂine RL does not admit cheap validation of trained policies for model selection.
Revised: Additionally, we report training stability , which is important in practice as ofﬂine RL does not admit cheap validation of trained policies for model selection.

Original: To evaluate stability, we train for a large number of gradient steps (2-3x longer than prior work) and either report the average performance over the course of training or the ﬁnal performance at the end of training.
Revised: To evaluate stability, we train for a large number of gradient steps (2-3x longer than prior work) and either report the average performance over the course of training or the ﬁnal performance at the end of training.

Original: We expect that a stable method that does not unlearn with more gradient steps, should have better average performance, as compared to a method that attains good peak performance but degrades with more training.
Revised: We expect that a stable method that does not unlearn with more gradient steps, should have better average performance, as compared to a method that attains good peak performance but degrades with more training.

Original: See Appendix E for further details.
Revised: See Appendix E for further details.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: to make these more challenging.
Revised: to make these more challenging.

Original: COG + DR3 outperforms COG in training and attains higher average and ﬁnal performance.
Revised: COG + DR3 outperforms COG in training and attains higher average and ﬁnal performance.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: al
Revised: al

Original: [Table]
Revised: [Table]

Original: Ofﬂine RL on Atari 2600 games.
Revised: Ofﬂine RL on Atari 2600 games.

Original: We compare DR3 to prior ofﬂine RL methods on a set of ofﬂine
Revised: We compare DR3 to prior ofﬂine RL methods on a set of ofﬂine

Original: Atari datasets of varying sizes and quality, akin to Agarwal et al.
Revised: Atari datasets of varying sizes and quality, akin to Agarwal et al.

Original: ; Kumar et al.
Revised: ; Kumar et al.

Original: We evaluated on three datasets: (1) 1% and 5% samples drawn uniformly at random from DQN replay; (2) a dataset with more suboptimal data consisting of the ﬁrst 10% samples observed by an online
Revised: We evaluated on three datasets: (1) 1% and 5% samples drawn uniformly at random from DQN replay; (2) a dataset with more suboptimal data consisting of the ﬁrst 10% samples observed by an online

Original: DQN.
Revised: DQN.

Original: Following Agarwal et al.
Revised: Following Agarwal et al.

Original: (2021), we report the interquartile mean (IQM) normalized scores across 17 games over the course of training in Figure 4 and report the IQM average performance in Table 1.
Revised: (2021), we report the interquartile mean (IQM) normalized scores across 17 games over the course of training in Figure 4 and report the IQM average performance in Table 1.

Original: Observe that combining DR3 with modern ofﬂine RL methods (CQL, REM) attains the best ﬁnal and average performance across the 17 Atari games tested on, directly improving upon prior methods across all the datasets.
Revised: Observe that combining DR3 with modern ofﬂine RL methods (CQL, REM) attains the best ﬁnal and average performance across the 17 Atari games tested on, directly improving upon prior methods across all the datasets.

Original: When DR3 is used in conjunction with REM, it prevents severe unlearning and performance degradation with more training.
Revised: When DR3 is used in conjunction with REM, it prevents severe unlearning and performance degradation with more training.

Original: CQL + DR3 improves by 20% over CQL on ﬁnal performance and attains 25% better average performance.
Revised: CQL + DR3 improves by 20% over CQL on ﬁnal performance and attains 25% better average performance.

Original: While DR3 is not unequivocally “stable”, as its performance also degrades relative to the peak it achieves (Figure 4), it is more stable relative to base ofﬂine RL algorithms.
Revised: While DR3 is not unequivocally “stable”, as its performance also degrades relative to the peak it achieves (Figure 4), it is more stable relative to base ofﬂine RL algorithms.

Original: We also compare DR3 to the srank(Φ) penalty proposed to counter rank collapse (Kumar et al., 2021).
Revised: We also compare DR3 to the srank(Φ) penalty proposed to counter rank collapse (Kumar et al., 2021).

Original: Directly taking median normalized score improvements reported by Kumar et al.
Revised: Directly taking median normalized score improvements reported by Kumar et al.

Original: (2021), CQL + DR3 improves by over 2x (31.5%) over naïve CQL relative to the srank penalty (14.1%), indicating DR3’s efﬁcacy.
Revised: (2021), CQL + DR3 improves by over 2x (31.5%) over naïve CQL relative to the srank penalty (14.1%), indicating DR3’s efﬁcacy.

Original: Ofﬂine RL on robotic manipulation from images.
Revised: Ofﬂine RL on robotic manipulation from images.

Original: Next, we aim to evaluate the efﬁcacy of DR3 on two image-based robotic manipulation tasks (Singh et al., 2020) (visualized on the right) that require composition of skills (e.g., opening a drawer, closing a drawer, picking an obstructive object, placing an object, etc.)
Revised: Next, we aim to evaluate the efﬁcacy of DR3 on two image-based robotic manipulation tasks (Singh et al., 2020) (visualized on the right) that require composition of skills (e.g., opening a drawer, closing a drawer, picking an obstructive object, placing an object, etc.)

Original: over extended horizons using only a sparse 0-1 reward.
Revised: over extended horizons using only a sparse 0-1 reward.

Original: As shown in Figure 3, combining DR3 with COG not only improves over COG, but also learns faster and attains a better average performance.
Revised: As shown in Figure 3, combining DR3 with COG not only improves over COG, but also learns faster and attains a better average performance.

Original: Ofﬂine RL on D4RL tasks.
Revised: Ofﬂine RL on D4RL tasks.

Original: Finally, we evaluate DR3 in conjunction with CQL on the antmaze and kitchen domains in D4RL (Fu et al., 2020).
Revised: Finally, we evaluate DR3 in conjunction with CQL on the antmaze and kitchen domains in D4RL (Fu et al., 2020).

Original: To assess if DR3 is stable and able to prevent unlearning that eventually appears in CQL, we trained CQL+DR3 for 6x longer: 2M steps with 3x higher learning rate.
Revised: To assess if DR3 is stable and able to prevent unlearning that eventually appears in CQL, we trained CQL+DR3 for 6x longer: 2M steps with 3x higher learning rate.

Original: Observe in Table 2, that CQL + DR3 outperforms CQL (statistical signiﬁcance shown in Appendix A.6).
Revised: Observe in Table 2, that CQL + DR3 outperforms CQL (statistical signiﬁcance shown in Appendix A.6).

Original: We show that CQL + DR3 also outperforms base CQL in terms of performance
Revised: We show that CQL + DR3 also outperforms base CQL in terms of performance

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: and stability on MuJoCo tasks previously studied in Kumar et al.
Revised: and stability on MuJoCo tasks previously studied in Kumar et al.

Original: (2021) in Appendix A.3.
Revised: (2021) in Appendix A.3.

Original: Further, we also compare the effect of adding DR3 to BRAC (Wu et al., 2019).
Revised: Further, we also compare the effect of adding DR3 to BRAC (Wu et al., 2019).

Original: DR3 applied on BRAC improves ﬁnal median normalized score performance by 13.8 and stability by 8.1 across 15 MuJoCo tasks.
Revised: DR3 applied on BRAC improves ﬁnal median normalized score performance by 13.8 and stability by 8.1 across 15 MuJoCo tasks.

Original: Numbers for BRAC can be found in Table F.2.
Revised: Numbers for BRAC can be found in Table F.2.

Original: To summarize , these results indicate that DR3 is a versatile explicit regularizer that improves performance and stability of a wide range of ofﬂine RL methods, including conservative methods (e.g,
Revised: To summarize , these results indicate that DR3 is a versatile explicit regularizer that improves performance and stability of a wide range of ofﬂine RL methods, including conservative methods (e.g,

Original: CQL, COG), policy constraint methods (e.g., BRAC) and ensemble-based methods (e.g., REM).
Revised: CQL, COG), policy constraint methods (e.g., BRAC) and ensemble-based methods (e.g., REM).

Original: DR3 does not suffer from rank collapse.
Revised: DR3 does not suffer from rank collapse.

Original: Prior work (Kumar et al., 2021) has shown that implicit regularization can lead to a rank collapse issue in TD-learning, preventing Q-networks from using full capacity.
Revised: Prior work (Kumar et al., 2021) has shown that implicit regularization can lead to a rank collapse issue in TD-learning, preventing Q-networks from using full capacity.

Original: To see if DR3 addresses the rank collapse issue, we follow Kumar et al.
Revised: To see if DR3 addresses the rank collapse issue, we follow Kumar et al.

Original: and plot the effective rank of learned features with DR3 in Figure 5 (DQN and REM in Appendix A.4).
Revised: and plot the effective rank of learned features with DR3 in Figure 5 (DQN and REM in Appendix A.4).

Original: While the value of the effective rank decreases during training with naïve bootstrapping, we ﬁnd that rank of DR3 features typically does not collapse, despite no explicit term encouraging this.
Revised: While the value of the effective rank decreases during training with naïve bootstrapping, we ﬁnd that rank of DR3 features typically does not collapse, despite no explicit term encouraging this.

Original: Additionally, we test the robustness/sensitivity of each layer in the learned Q-network to re-initialization (Zhang et al., 2019) during training and ﬁnd that DR3 alters the representations trained with TD to behave similarly to supervised learning (Figure A.2).
Revised: Additionally, we test the robustness/sensitivity of each layer in the learned Q-network to re-initialization (Zhang et al., 2019) during training and ﬁnd that DR3 alters the representations trained with TD to behave similarly to supervised learning (Figure A.2).

Original: Comparing explicit regularizers for different choices of noise covariance M .
Revised: Comparing explicit regularizers for different choices of noise covariance M .

Original: Finally, we investigate the behavior of different implicit regularizers derived via two choices of M in Equation 4 and the corresponding explicit regularizers.
Revised: Finally, we investigate the behavior of different implicit regularizers derived via two choices of M in Equation 4 and the corresponding explicit regularizers.

Original: While the explicit regularizer we use in practice is a simplifying choice that works well, another choice of M is the covariance matrix induced by label noise, which requires explicit computation of Σ ∗ M .
Revised: While the explicit regularizer we use in practice is a simplifying choice that works well, another choice of M is the covariance matrix induced by label noise, which requires explicit computation of Σ ∗ M .

Original: Observe in Figure 6 that the explicit regularizer for our simplifying choice is not worse than the different choice of
Revised: Observe in Figure 6 that the explicit regularizer for our simplifying choice is not worse than the different choice of

Original: M .
Revised: M .

Original: This justiﬁes utilizing our simpliﬁed, heuristic choice of setting Σ ∗ M
Revised: This justiﬁes utilizing our simpliﬁed, heuristic choice of setting Σ ∗ M

Original: = I in practice.
Revised: = I in practice.

Original: Results on ﬁve Atari games are shown in Appendix A.5.
Revised: Results on ﬁve Atari games are shown in Appendix A.5.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We characterized the implicit preference of TD-learning towards solutions that maximally co-adapt gradients (or features) at consecutive state-action tuples that appear in Bellman backup.
Revised: We characterized the implicit preference of TD-learning towards solutions that maximally co-adapt gradients (or features) at consecutive state-action tuples that appear in Bellman backup.

Original: This regularization effect is exacerbated when out-of-sample state-action samples are used for the Bellman backup and it can lead to poor policy performance.
Revised: This regularization effect is exacerbated when out-of-sample state-action samples are used for the Bellman backup and it can lead to poor policy performance.

Original: Inspired by the theory, we propose a practical explicit regularizer, DR3 that aims to counteracts this implicit regularizer.
Revised: Inspired by the theory, we propose a practical explicit regularizer, DR3 that aims to counteracts this implicit regularizer.

Original: DR3 yields substantial improvements in stability and performance on a wide range of ofﬂine RL problems.
Revised: DR3 yields substantial improvements in stability and performance on a wide range of ofﬂine RL problems.

Original: We believe that an understanding of implicit regularization in value-based RL methods can inspire cross-validation and model selection strategies, an important, open challenge in ofﬂine RL, for which existing offpolicy evaluation techniques are not practically sufﬁcient (Fu et al., 2021).
Revised: We believe that an understanding of implicit regularization in value-based RL methods can inspire cross-validation and model selection strategies, an important, open challenge in ofﬂine RL, for which existing offpolicy evaluation techniques are not practically sufﬁcient (Fu et al., 2021).

Original: Devising value-based RL methods that are stable, in an absolute sense, and never degrade once they reach good solutions is an avenue for future work.
Revised: Devising value-based RL methods that are stable, in an absolute sense, and never degrade once they reach good solutions is an avenue for future work.

Original: Extending our approach to the online RL setting is also part of future work.
Revised: Extending our approach to the online RL setting is also part of future work.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure reproducibility in our empirical results, we provide complete experimental details regarding methods and hyper-parameters used in the Appendix E. Additionally, we follow the recommendations of Agarwal et al.
Revised: To ensure reproducibility in our empirical results, we provide complete experimental details regarding methods and hyper-parameters used in the Appendix E. Additionally, we follow the recommendations of Agarwal et al.

Original: (2021) for reliable evaluation in deep RL and report the statistical uncertainty in reported results including aggregate performance metrics.
Revised: (2021) for reliable evaluation in deep RL and report the statistical uncertainty in reported results including aggregate performance metrics.

Original: We provide individual scores in
Revised: We provide individual scores in

Original: Appendix F and we will release scores for individual runs as well as open-source our code.
Revised: Appendix F and we will release scores for individual runs as well as open-source our code.

Original: For theoretical results, we provide explanations of all the assumptions and a complete proof of the claims in Appendix C
Revised: For theoretical results, we provide explanations of all the assumptions and a complete proof of the claims in Appendix C

Original: Appendices A A DDITIONAL V ISUALIZATIONS AND E XPERIMENTS FOR DR 
Revised: Appendices A A DDITIONAL V ISUALIZATIONS AND E XPERIMENTS FOR DR

Original: In this section, we provide visualizations and diagnostic experiments evaluating various aspects of feature co-adaptation and the DR3 regularizer.
Revised: In this section, we provide visualizations and diagnostic experiments evaluating various aspects of feature co-adaptation and the DR3 regularizer.

Original: We ﬁrst provide more empirical evidence showing the presence of feature co-adaptation in modern deep ofﬂine RL algorithms.
Revised: We ﬁrst provide more empirical evidence showing the presence of feature co-adaptation in modern deep ofﬂine RL algorithms.

Original: We will also visualize
Revised: We will also visualize

Original: DR3 inspired from the implicit regularizer term in TD-learning alleviates rank collapse discussed in
Revised: DR3 inspired from the implicit regularizer term in TD-learning alleviates rank collapse discussed in

Original: Kumar et al.
Revised: Kumar et al.

Original: We will compare the efﬁcacies of the explicit regularizer induced for different choices of the noise covariance matrix M (Equation 4), understand the effect of dropping the stop gradient term in our practical regularizer and ﬁnally, perform diagnostic experiments visualizing if the Q-networks learned with DR3 resemble more like neural networks trained via supervised learning, measured in terms of sensitivity and robustness to layer reinitialization (Zhang et al., 2019).
Revised: We will compare the efﬁcacies of the explicit regularizer induced for different choices of the noise covariance matrix M (Equation 4), understand the effect of dropping the stop gradient term in our practical regularizer and ﬁnally, perform diagnostic experiments visualizing if the Q-networks learned with DR3 resemble more like neural networks trained via supervised learning, measured in terms of sensitivity and robustness to layer reinitialization (Zhang et al., 2019).

Original: A. M ORE E MPIRICAL E VIDENCE OF F EATURE C O -A DAPTATION
Revised: A. M ORE E MPIRICAL E VIDENCE OF F EATURE C O -A DAPTATION

Original: In this section, we provide more empirical evidence demonstrating the existence of the feature coadaptation issue in modern ofﬂine RL algorithms such as DQN and CQL.
Revised: In this section, we provide more empirical evidence demonstrating the existence of the feature coadaptation issue in modern ofﬂine RL algorithms such as DQN and CQL.

Original: As shown below in Figure A.1, while the average dataset Q-value for both CQL and DQN exhibit a ﬂatline trend, the dot product similarity for consecutive state-action tuples generally continues to increase throughout training and does not ﬂatline.
Revised: As shown below in Figure A.1, while the average dataset Q-value for both CQL and DQN exhibit a ﬂatline trend, the dot product similarity for consecutive state-action tuples generally continues to increase throughout training and does not ﬂatline.

Original: While DQN eventually diverges in Seaquest, the dot products increase with more gradient steps even before divergence starts to appear.
Revised: While DQN eventually diverges in Seaquest, the dot products increase with more gradient steps even before divergence starts to appear.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: i m
Revised: i m

Original: [Figure]
Revised: [Figure]

Original: r i t y
Revised: r i t y

Original: [Figure]
Revised: [Figure]

Original: A. L AYER - WISE STRUCTURE OF A Q- NETWORK TRAINED WITH DR 
Revised: A. L AYER - WISE STRUCTURE OF A Q- NETWORK TRAINED WITH DR

Original: To understand if DR3 indeed makes Q-networks behave as if they were trained via supervised learning, utilizing the empirical analysis tools from Zhang et al.
Revised: To understand if DR3 indeed makes Q-networks behave as if they were trained via supervised learning, utilizing the empirical analysis tools from Zhang et al.

Original: we test the robustness/sensitivity of each layer in the learned network to re-initialization, while keeping the other layers ﬁxed.
Revised: we test the robustness/sensitivity of each layer in the learned network to re-initialization, while keeping the other layers ﬁxed.

Original: This tests if a particular layer is critical to the predictions of the learned neural network and enables us to reason about generalization properties (Zhang et al., 2019; Chatterji et al., 2020).
Revised: This tests if a particular layer is critical to the predictions of the learned neural network and enables us to reason about generalization properties (Zhang et al., 2019; Chatterji et al., 2020).

Original: We ran CQL and
Revised: We ran CQL and

Original: REM and saved all the intermediate checkpoints.
Revised: REM and saved all the intermediate checkpoints.

Original: Then, as shown in Figure A.2, we ﬁrst loaded a checkpoint ( x -axis), and computed policy performance (shaded color; colorbar) by re-initializing a given layer ( y -axis) of the network to its initialization value before training for the same run.
Revised: Then, as shown in Figure A.2, we ﬁrst loaded a checkpoint ( x -axis), and computed policy performance (shaded color; colorbar) by re-initializing a given layer ( y -axis) of the network to its initialization value before training for the same run.

Original: Note in Figure A.2, that while almost all layers are absolutely critical for the base CQL algorithm, utilizing DR3 substantially reduces sensitivity to the latter layers in the Q-network over the course of training.
Revised: Note in Figure A.2, that while almost all layers are absolutely critical for the base CQL algorithm, utilizing DR3 substantially reduces sensitivity to the latter layers in the Q-network over the course of training.

Original: This is similar to what Zhang et al.
Revised: This is similar to what Zhang et al.

Original: (2019) observed for supervised learning, where the initial layers of a network were the most critical, and the latter layers primarily performed near-random transformations without affecting the performance of the network.
Revised: (2019) observed for supervised learning, where the initial layers of a network were the most critical, and the latter layers primarily performed near-random transformations without affecting the performance of the network.

Original: This indicates that utilizing DR alters the internal layers of a Q-network trained with TD to behave closer to supervised learning.
Revised: This indicates that utilizing DR alters the internal layers of a Q-network trained with TD to behave closer to supervised learning.

Original: A. R ESULTS ON M U J O C O D OMAINS
Revised: A. R ESULTS ON M U J O C O D OMAINS

Original: In this section, we provide the results of applying DR3 on the MuJoCo tasks shown in Figure A.5.
Revised: In this section, we provide the results of applying DR3 on the MuJoCo tasks shown in Figure A.5.

Original: (Appendix A) of Kumar et al. (2021).
Revised: (Appendix A) of Kumar et al. (2021).

Original: To brieﬂy describe the setup, in these tasks we train on the three gym tasks (Hopper-v2, Ant-v2, Walker2d-v2) using 20% of the ofﬂine data, uniformly subsampled from the run of an online SAC agent, mimicking the setup from Kumar et al.
Revised: To brieﬂy describe the setup, in these tasks we train on the three gym tasks (Hopper-v2, Ant-v2, Walker2d-v2) using 20% of the ofﬂine data, uniformly subsampled from the run of an online SAC agent, mimicking the setup from Kumar et al.

Original: Rather than retraining an SAC agent to collect data, we subsampled the Gym-MuJoCo *-full-replay-v replay buffers from the latest D4RL (Fu et al., 2020).
Revised: Rather than retraining an SAC agent to collect data, we subsampled the Gym-MuJoCo *-full-replay-v replay buffers from the latest D4RL (Fu et al., 2020).

Original: In these cases we plot the srank δ values, the feature dot products and the corresponding performance values with and without the DR3 regularizer for 4M steps (Kumar et al.
Revised: In these cases we plot the srank δ values, the feature dot products and the corresponding performance values with and without the DR3 regularizer for 4M steps (Kumar et al.

Original: (2021) showed their plots for just under 4M steps) in Figure A.3.
Revised: (2021) showed their plots for just under 4M steps) in Figure A.3.

Original: Observe in Figure A.3, that while the standard CQL algorithm performs poorly and suffers from performance degradation within about 1M-1.5M steps for Walker2d and Ant, CQL + DR3 is able to prevent the performance degradation and trains stably.
Revised: Observe in Figure A.3, that while the standard CQL algorithm performs poorly and suffers from performance degradation within about 1M-1.5M steps for Walker2d and Ant, CQL + DR3 is able to prevent the performance degradation and trains stably.

Original: Base CQL demonstrates oscillatory performance on Hopper, but CQL + DR3 stabilizes the performace of CQL.
Revised: Base CQL demonstrates oscillatory performance on Hopper, but CQL + DR3 stabilizes the performace of CQL.

Original: This indicates that DR3 is effective on MuJoCo domains, and prevents the instabilities with CQL.
Revised: This indicates that DR3 is effective on MuJoCo domains, and prevents the instabilities with CQL.

Original: For details, the weight on the CQL regularizer in this case is equal to 5.0 across all the tasks, and weight on the DR3 regularizer is 0.01.
Revised: For details, the weight on the CQL regularizer in this case is equal to 5.0 across all the tasks, and weight on the DR3 regularizer is 0.01.

Original: We also attempted to tune the CQL coefﬁcient for the baseline
Revised: We also attempted to tune the CQL coefﬁcient for the baseline

Original: [Figure]
Revised: [Figure]

Original: CQL algorithm within { 1 . 0 , 2 . 0 , 5 . 0 , 10 . 0 , 20 . 0 } to see if it address the performance degradation issues but did not ﬁnd any difference in the collapsing behavior of base CQL.
Revised: CQL algorithm within { 1 . 0 , 2 . 0 , 5 . 0 , 10 . 0 , 20 . 0 } to see if it address the performance degradation issues but did not ﬁnd any difference in the collapsing behavior of base CQL.

Original: Our CQL baseline is therefore well-tuned and DR3 improves the performance over this baseline.
Revised: Our CQL baseline is therefore well-tuned and DR3 improves the performance over this baseline.

Original: A. R ANK C OLLAPSE IS A LLEVIATED W ITH DR 
Revised: A. R ANK C OLLAPSE IS A LLEVIATED W ITH DR

Original: [Figure]
Revised: [Figure]

Original: Prior work (Kumar et al., 2021) has shown that implicit regularization in TD-learning can lead to a feature rank collapse phenomenon in the Q-function
Revised: Prior work (Kumar et al., 2021) has shown that implicit regularization in TD-learning can lead to a feature rank collapse phenomenon in the Q-function

Original: [Figure]
Revised: [Figure]

Original: which hinders the Q-function from using its full representational capacity.
Revised: which hinders the Q-function from using its full representational capacity.

Original: Such a phenomenon is absent in supervised learning
Revised: Such a phenomenon is absent in supervised learning

Original: [Figure]
Revised: [Figure]

Original: where the feature rank does not collapse.
Revised: where the feature rank does not collapse.

Original: Since DR3 is inspired by mitigating the effects of the term in the im- plicit regularizer (Equation 4) that only appears in the case of TD-learning, we wish to understand if utilizing DR3 also alleviates rank collapse.
Revised: Since DR3 is inspired by mitigating the effects of the term in the im- plicit regularizer (Equation 4) that only appears in the case of TD-learning, we wish to understand if utilizing DR3 also alleviates rank collapse.

Original: To do so, we compute the effective rank srank δ ( ϕ ) metric of the features learned by Q-functions trained via CQL and CQL with DR3 explicit regularizer.
Revised: To do so, we compute the effective rank srank δ ( ϕ ) metric of the features learned by Q-functions trained via CQL and CQL with DR3 explicit regularizer.

Original: As shown in Figure A.4, for the case of ﬁve Atari games, utilizing DR3 alleviates the rank collapse issue completely (i.e., the ranks do not collapse to very small values when CQL + DR3 is trained for long).
Revised: As shown in Figure A.4, for the case of ﬁve Atari games, utilizing DR3 alleviates the rank collapse issue completely (i.e., the ranks do not collapse to very small values when CQL + DR3 is trained for long).

Original: We do not claim that the ranks with DR3 are necessarily higher, and infact as we show below, a higher srank of features may not always imply a better solution.
Revised: We do not claim that the ranks with DR3 are necessarily higher, and infact as we show below, a higher srank of features may not always imply a better solution.

Original: The fact that DR3 can prevent rank collapse is potentially surprising, because no term in the practical DR3 regularizer explicitly aims to increase rank: feature dot products can be made smaller while retaining low ranks by simply rescaling the feature vectors.
Revised: The fact that DR3 can prevent rank collapse is potentially surprising, because no term in the practical DR3 regularizer explicitly aims to increase rank: feature dot products can be made smaller while retaining low ranks by simply rescaling the feature vectors.

Original: But, as we observe, utilizing DR3 enables learning features that do not exhibit collapsed ranks, thus we hypothesize that correcting for appropriate terms in R TD ( θ ) can address some of the previously observed pathologies in TD-learning.
Revised: But, as we observe, utilizing DR3 enables learning features that do not exhibit collapsed ranks, thus we hypothesize that correcting for appropriate terms in R TD ( θ ) can address some of the previously observed pathologies in TD-learning.

Original: [Figure]
Revised: [Figure]

Original: We now investigate the feature ranks of a Q-network trained when DR3 is applied in conjunction with a standard DQN and REM (Agarwal et al., 2020) on the Atari domains.
Revised: We now investigate the feature ranks of a Q-network trained when DR3 is applied in conjunction with a standard DQN and REM (Agarwal et al., 2020) on the Atari domains.

Original: We plot the values of srank δ ( ϕ ) , the feature dot products and the performance of the algorithm for DQN in Figure A.5 and for REM in Figure A.6.
Revised: We plot the values of srank δ ( ϕ ) , the feature dot products and the performance of the algorithm for DQN in Figure A.5 and for REM in Figure A.6.

Original: In the case of DQN, we ﬁnd that unlike the base DQN algorithm for which feature rank does begin to collapse with more training, the srank for DQN + DR3 is increasing.
Revised: In the case of DQN, we ﬁnd that unlike the base DQN algorithm for which feature rank does begin to collapse with more training, the srank for DQN + DR3 is increasing.

Original: We also note that DQN + DR3 attains a better performance compared to DQN, throughout training.
Revised: We also note that DQN + DR3 attains a better performance compared to DQN, throughout training.

Original: However, we note that the opposite trend is true for the case of REM: while REM + DR3 attains a better performance than REM, adding DR3 leads to a reduction in the srank value compared to base REM.
Revised: However, we note that the opposite trend is true for the case of REM: while REM + DR3 attains a better performance than REM, adding DR3 leads to a reduction in the srank value compared to base REM.

Original: At a ﬁrst glance, this might seem contradicting Kumar et al.
Revised: At a ﬁrst glance, this might seem contradicting Kumar et al.

Original: , but this is not the case: to our understanding, Kumar et al.
Revised: , but this is not the case: to our understanding, Kumar et al.

Original: (2021) establish a correlation between extremely low rank values (i.e., rank collapse) and poor performance, but this does not mean that all high rank features will lead to good performance.
Revised: (2021) establish a correlation between extremely low rank values (i.e., rank collapse) and poor performance, but this does not mean that all high rank features will lead to good performance.

Original: We suspect that since REM trains a multi-headed Q-function with shared features and randomized target values, it is able to preserve high-rank features, but this need not mean that these features are useful.
Revised: We suspect that since REM trains a multi-headed Q-function with shared features and randomized target values, it is able to preserve high-rank features, but this need not mean that these features are useful.

Original: In fact, as shown in Figure A.7, we ﬁnd that the base REM algorithm does exhibit feature co-adaptation.
Revised: In fact, as shown in Figure A.7, we ﬁnd that the base REM algorithm does exhibit feature co-adaptation.

Original: This case is an example where the srank metric from
Revised: This case is an example where the srank metric from

Original: Kumar et al.
Revised: Kumar et al.

Original: (2021) may not indicate poor performance.
Revised: (2021) may not indicate poor performance.

Original: A. I NDUCED I MPLICIT R EGULARIZER : T HEORY A ND P RACTICE
Revised: A. I NDUCED I MPLICIT R EGULARIZER : T HEORY A ND P RACTICE

Original: In this section, we compare the performance of our practical DR3 regularizer to the regularizers (Equation 4) obtained for different choices of M , such as M induced by noise, studied in previous work, and also evaluate the effect of dropping the stop gradient function from the practical version of our regularizer.
Revised: In this section, we compare the performance of our practical DR3 regularizer to the regularizers (Equation 4) obtained for different choices of M , such as M induced by noise, studied in previous work, and also evaluate the effect of dropping the stop gradient function from the practical version of our regularizer.

Original: [Figure]
Revised: [Figure]

Original: (2021) as we discuss in text.
Revised: (2021) as we discuss in text.

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: Empirically comparing the explicit regularizers for different noise covariance matrices, M .
Revised: Empirically comparing the explicit regularizers for different noise covariance matrices, M .

Original: The theoretically derived regularizer (Equation 4) suggests that for a given choice of M , the following equivalent of feature dot products should increase over the course of training:
Revised: The theoretically derived regularizer (Equation 4) suggests that for a given choice of M , the following equivalent of feature dot products should increase over the course of training:

Original: [Equation]
Revised: [Equation]

Original: We evaluate the efﬁcacy of the explicit regularizer that penalizes the generalized dot products, ∆ M ( θ ) , in improving the performance of the policy, with the goal of identifying if our practical method performs similar to this regularizer on generalized dot products.. While Σ ∗ M must be explicitly computed by running ﬁxed point iteration for every parameter iterate θ found during TD-learning – which makes this method signiﬁcantly computationally expensive 1 , we evaluated it on ﬁve Atari games for 50 × 62.5k gradient steps as a proof of concept.
Revised: We evaluate the efﬁcacy of the explicit regularizer that penalizes the generalized dot products, ∆ M ( θ ) , in improving the performance of the policy, with the goal of identifying if our practical method performs similar to this regularizer on generalized dot products.. While Σ ∗ M must be explicitly computed by running ﬁxed point iteration for every parameter iterate θ found during TD-learning – which makes this method signiﬁcantly computationally expensive 1 , we evaluated it on ﬁve Atari games for 50 × 62.5k gradient steps as a proof of concept.

Original: As shown in Figure A.8, the DR3 penalty with the choice of M which corresponds to label noise, and the dot product DR3 penalty, which is our main practical approach in this paper generally perform similarly on these domains, attaining almost identical learning curves on 4/5 games , and clearly improving over the base algorithm.
Revised: As shown in Figure A.8, the DR3 penalty with the choice of M which corresponds to label noise, and the dot product DR3 penalty, which is our main practical approach in this paper generally perform similarly on these domains, attaining almost identical learning curves on 4/5 games , and clearly improving over the base algorithm.

Original: This hints at the possibility of utilizing other noise covariance matrices to derive an explicit regularizer.
Revised: This hints at the possibility of utilizing other noise covariance matrices to derive an explicit regularizer.

Original: Deriving more computationally efﬁcient versions of the regularizer for a general M and identifying the best choice of M are subject to future work.
Revised: Deriving more computationally efﬁcient versions of the regularizer for a general M and identifying the best choice of M are subject to future work.

Original: Effect of stop gradient.
Revised: Effect of stop gradient.

Original: Finally, we investigate the effect of utilizing a stop gradient in the DR regularizer.
Revised: Finally, we investigate the effect of utilizing a stop gradient in the DR regularizer.

Original: We run a variant of DR3: ∆ ′ ( ϕ ) =
Revised: We run a variant of DR3: ∆ ′ ( ϕ ) =

Original: P s , a , s ′ ϕ
Revised: P s , a , s ′ ϕ

Original: ( s , a ) ⊤ [[ ϕ ( s ′ , a ′ )]] , with the stop gradient on the second term ( s ′ , a ′ ) and present a comparison to the one without the stop gradient in Table A.
Revised: ( s , a ) ⊤ [[ ϕ ( s ′ , a ′ )]] , with the stop gradient on the second term ( s ′ , a ′ ) and present a comparison to the one without the stop gradient in Table A.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Observe that in all the ﬁve games the DR3 regularizer derived for the choice of M from Blanc et al
Revised: Observe that in all the ﬁve games the DR3 regularizer derived for the choice of M from Blanc et al

Original: 2020) also leads to a substantial increase in performance over the base algorithm, and in four of ﬁve games,
Revised: 2020) also leads to a substantial increase in performance over the base algorithm, and in four of ﬁve games,

Original: DR3 (label-noise) works just as well as DR for REM as the base ofﬂine method, averaged over 17 games.
Revised: DR3 (label-noise) works just as well as DR for REM as the base ofﬂine method, averaged over 17 games.

Original: Note that this version of DR3, with the stop gradient, also improves upon the baseline ofﬂine RL method (i e., REM) by 130% .
Revised: Note that this version of DR3, with the stop gradient, also improves upon the baseline ofﬂine RL method (i e., REM) by 130% .

Original: While this performs largely similar, but somewhat worse than the complete version without the stop gradient, these results do indicate that utilizing ∆ ′
Revised: While this performs largely similar, but somewhat worse than the complete version without the stop gradient, these results do indicate that utilizing ∆ ′

Original: ( ϕ ) can also lead to signiﬁcant gains in performance.
Revised: ( ϕ ) can also lead to signiﬁcant gains in performance.

Original: A. S TATISTICAL S IGNIFICANCE OF O UR D4RL A NTMAZE AND K ITCHEN R ESULTS
Revised: A. S TATISTICAL S IGNIFICANCE OF O UR D4RL A NTMAZE AND K ITCHEN R ESULTS

Original: In order to assess the statistical signiﬁcance of our D4RL Antmaze and Kitchen results, we follow the recommendedations by Agarwal et al.
Revised: In order to assess the statistical signiﬁcance of our D4RL Antmaze and Kitchen results, we follow the recommendedations by Agarwal et al.

Original: (2021) for comparing deep RL algorithms considering their statistical uncertainties.
Revised: (2021) for comparing deep RL algorithms considering their statistical uncertainties.

Original: Speciﬁcally, we computed the average probability of improvement (Agarwal et al., 2021) of CQL + DR3 over CQL on the antmaze and kitchen domains, and we ﬁnd that
Revised: Speciﬁcally, we computed the average probability of improvement (Agarwal et al., 2021) of CQL + DR3 over CQL on the antmaze and kitchen domains, and we ﬁnd that

Original: DR3 does signiﬁcantly and meaningfully improve over CQL on both the Kitchen and AntMaze domains.
Revised: DR3 does signiﬁcantly and meaningfully improve over CQL on both the Kitchen and AntMaze domains.

Original: Before presenting the results, let us ﬁrst describe the metric we compute.
Revised: Before presenting the results, let us ﬁrst describe the metric we compute.

Original: Probability of improvement and statistical signiﬁcance .
Revised: Probability of improvement and statistical signiﬁcance .

Original: For two given algorithms Alg 1 and Alg 2 , and runs X k, 1 , X k, 2 , · · · , X k,m from Alg 1 and runs Y k, 1 , Y k, 2 , · · · , Y k,n from Alg 2 on task k , the probability of improvement of Alg 1 over Alg 2 is given by P ( Alg 1 > Alg 2 )
Revised: For two given algorithms Alg 1 and Alg 2 , and runs X k, 1 , X k, 2 , · · · , X k,m from Alg 1 and runs Y k, 1 , Y k, 2 , · · · , Y k,n from Alg 2 on task k , the probability of improvement of Alg 1 over Alg 2 is given by P ( Alg 1 > Alg 2 )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: Alg k 2 ) .
Revised: Alg k 2 ) .

Original: The probability of improvement on a given task k , P ( Alg k 1 > Alg k 2 ) is computed using the
Revised: The probability of improvement on a given task k , P ( Alg k 1 > Alg k 2 ) is computed using the

Original: Mann-Whitney U-statistic and is given by:
Revised: Mann-Whitney U-statistic and is given by:

Original: P ( Alg k 1 > Alg k 2 ) =
Revised: P ( Alg k 1 > Alg k 2 ) =

Original: MN
Revised: MN

Original: M X i =
Revised: M X i =

Original: N X j =
Revised: N X j =

Original: S ( X k,i , Y k,j ) where S ( x, y ) =
Revised: S ( X k,i , Y k,j ) where S ( x, y ) =

Original: 8< : 1 , if y < x, 12 , if y = x, 0 , if y > x.
Revised: 8< : 1 , if y < x, 12 , if y = x, 0 , if y > x.

Original: Alg 1 leads to statistically signiﬁcant improve over Alg 2 if the lower CI for P ( Alg 1 > Alg 2 ) is larger than 0.5.
Revised: Alg 1 leads to statistically signiﬁcant improve over Alg 2 if the lower CI for P ( Alg 1 > Alg 2 ) is larger than 0.5.

Original: Per the Neyman-Pearson statistical testing criterion in Bouthillier et al.
Revised: Per the Neyman-Pearson statistical testing criterion in Bouthillier et al.

Original: (2021), Algleads to statistically meaningful improvement over Alg 2 if the upper conﬁdence interval (CI) of
Revised: (2021), Algleads to statistically meaningful improvement over Alg 2 if the upper conﬁdence interval (CI) of

Original: P ( Alg 1 > Alg 2 ) is larger than 0.75.
Revised: P ( Alg 1 > Alg 2 ) is larger than 0.75.

Original: Figure A.9 presents the value of P ( CQL + DR3 > CQL ) on the AntMaze and Kitchen domains at 2M gradient steps along with the 95% CI for this statistic.
Revised: Figure A.9 presents the value of P ( CQL + DR3 > CQL ) on the AntMaze and Kitchen domains at 2M gradient steps along with the 95% CI for this statistic.

Original: DR3 improves over CQL on the AntMaze domains with probability 0.83 with 95% CI (0.7, 0.96) and on the Kitchen domains with probability 0.8 with 95% CI (0.6, 1.0).
Revised: DR3 improves over CQL on the AntMaze domains with probability 0.83 with 95% CI (0.7, 0.96) and on the Kitchen domains with probability 0.8 with 95% CI (0.6, 1.0).

Original: These values pass the criterion of being both statistically signiﬁcant and meaningful per the above deﬁnitions, implying that DR3 does signiﬁcantly and meaningfully improve upon CQL on these domains.
Revised: These values pass the criterion of being both statistically signiﬁcant and meaningful per the above deﬁnitions, implying that DR3 does signiﬁcantly and meaningfully improve upon CQL on these domains.

Original: A. L EARNING C URVE FOR 1% U NIFORM R EPLAY
Revised: A. L EARNING C URVE FOR 1% U NIFORM R EPLAY

Original: In this section, we present the learning curve for REM + DR3 when training for 200 × 62.5k gradient steps, extending the learning curve in Figure 4 to twice as long for the 1% replay dataset.
Revised: In this section, we present the learning curve for REM + DR3 when training for 200 × 62.5k gradient steps, extending the learning curve in Figure 4 to twice as long for the 1% replay dataset.

Original: This is 10x longer training iterations as the prior work Agarwal et al.
Revised: This is 10x longer training iterations as the prior work Agarwal et al.

Original: (2020) which evaluated algorithms on this dataset.
Revised: (2020) which evaluated algorithms on this dataset.

Original: We present the aggregate learning curve in Figure A.10, and ﬁnd that while REM +
Revised: We present the aggregate learning curve in Figure A.10, and ﬁnd that while REM +

Original: DR3 does destabilize compared the peak solution it attains, the performance of REM + DR3 atiterations (= thrice as long as when the peak performance in REM appears) is still comparable to the peak in the REM performance.
Revised: DR3 does destabilize compared the peak solution it attains, the performance of REM + DR3 atiterations (= thrice as long as when the peak performance in REM appears) is still comparable to the peak in the REM performance.

Original: [Figure]
Revised: [Figure]

Original: This degradation is perhaps expected since REM does not correct for out-of-distribution actions, and DR3 does not explicitly correct for overestimation due to such actions either.
Revised: This degradation is perhaps expected since REM does not correct for out-of-distribution actions, and DR3 does not explicitly correct for overestimation due to such actions either.

Original: Overestimation at out-of-distribution actions are known to lead to poor-performing policies (Levine et al., 2020).
Revised: Overestimation at out-of-distribution actions are known to lead to poor-performing policies (Levine et al., 2020).

Original: However, note that applying DR3 in conjunction with REM does allow a practitioner to choose policies from a 10x larger interval, compared to base REM.
Revised: However, note that applying DR3 in conjunction with REM does allow a practitioner to choose policies from a 10x larger interval, compared to base REM.

Original: A. U NDERSTANDING F EATURE C O -A DAPTATION S OME M ORE
Revised: A. U NDERSTANDING F EATURE C O -A DAPTATION S OME M ORE

Original: In this section, we present some more empirical evidence to understand feature co-adaptation.
Revised: In this section, we present some more empirical evidence to understand feature co-adaptation.

Original: The three factors we wish to study are: (1) the effect of target update frequency on feature co-adaptation; (2) understand the trend in normalized similarities and compare these to the trend in dot products; and (3) understand the effect of out-on-sample actions in TD-learning and compare it to ofﬂine
Revised: The three factors we wish to study are: (1) the effect of target update frequency on feature co-adaptation; (2) understand the trend in normalized similarities and compare these to the trend in dot products; and (3) understand the effect of out-on-sample actions in TD-learning and compare it to ofﬂine

Original: SARSA on a simpler gridworld domain.
Revised: SARSA on a simpler gridworld domain.

Original: We answer these questions one by one via experiments aiming to verify each hypothesis.
Revised: We answer these questions one by one via experiments aiming to verify each hypothesis.

Original: A.8. E FFECT OF T ARGET U PDATE F REQUENCY ON F EATURE C O -A DAPTATION
Revised: A.8. E FFECT OF T ARGET U PDATE F REQUENCY ON F EATURE C O -A DAPTATION

Original: We studied the effect of target update frequency on feature co-adaptation, on some gridworld domains from Fu et al.
Revised: We studied the effect of target update frequency on feature co-adaptation, on some gridworld domains from Fu et al.

Original: We utilized the grid16smoothobs environment, where the goal of the agent is to navigate from the center of a 16 × 16 gridworld maze to one of its corners while avoiding obstacles and “lava” cells.
Revised: We utilized the grid16smoothobs environment, where the goal of the agent is to navigate from the center of a 16 × 16 gridworld maze to one of its corners while avoiding obstacles and “lava” cells.

Original: The observations provided to the RL algorithm are given by a high-dimensional random transformation of the ( x, y ) coordinates, smoothed over neighboring cells in the gridworld.
Revised: The observations provided to the RL algorithm are given by a high-dimensional random transformation of the ( x, y ) coordinates, smoothed over neighboring cells in the gridworld.

Original: We sampled an ofﬂine dataset of 256 transitions and trained a Q-network with two hidden layers of size (1024 , 1024) via ﬁtted Q-iteration (FQI)
Revised: We sampled an ofﬂine dataset of 256 transitions and trained a Q-network with two hidden layers of size (1024 , 1024) via ﬁtted Q-iteration (FQI)

Original: (Riedmiller, 2005).
Revised: (Riedmiller, 2005).

Original: We evaluated the feature dot products for Q-functions trained with a varying target update frequencies, given generically as: updating the target network using a hard target update once per N gradient steps, where N takes on values N = 5 , 10 , 50 , 100 , 200 , 500 , and present the results in Figure A.
Revised: We evaluated the feature dot products for Q-functions trained with a varying target update frequencies, given generically as: updating the target network using a hard target update once per N gradient steps, where N takes on values N = 5 , 10 , 50 , 100 , 200 , 500 , and present the results in Figure A.

Original: (left), averaged over 3 random seeds.
Revised: (left), averaged over 3 random seeds.

Original: Thee feature dot products initially decrease from N = 5 to
Revised: Thee feature dot products initially decrease from N = 5 to

Original: N = 10 , because the target network is updated slower, but then starts to rapidly increase when when the target network is slowed down further to N = 50 and N = 200 in one case and N = 500 in the other case.
Revised: N = 10 , because the target network is updated slower, but then starts to rapidly increase when when the target network is slowed down further to N = 50 and N = 200 in one case and N = 500 in the other case.

Original: [Figure]
Revised: [Figure]

Original: We also evaluated the feature dot products when using CQL as the base ofﬂine RL algorithm.
Revised: We also evaluated the feature dot products when using CQL as the base ofﬂine RL algorithm.

Original: As shown in
Revised: As shown in

Original: Figure A.11 (right), while CQL does reduce the absolute range of the feature dot products, slow target updates with N = 500 still lead to the highest feature dot products as training progresses.
Revised: Figure A.11 (right), while CQL does reduce the absolute range of the feature dot products, slow target updates with N = 500 still lead to the highest feature dot products as training progresses.

Original: Takeaway: While it is intuitive to think that a slower target network might alleviate co-adaptation, we see that this is not the case empirically with both FQI and CQL, suggesting a deeper question that is an interesting avenue for future study.
Revised: Takeaway: While it is intuitive to think that a slower target network might alleviate co-adaptation, we see that this is not the case empirically with both FQI and CQL, suggesting a deeper question that is an interesting avenue for future study.

Original: A.8. G RIDWORLD E XPERIMENTS C OMPARING TD- LEARNING VS O FFLINE SARSA
Revised: A.8. G RIDWORLD E XPERIMENTS C OMPARING TD- LEARNING VS O FFLINE SARSA

Original: To supplement the analysis in Section 3.1, we ran some experiments in the gridworld domains from
Revised: To supplement the analysis in Section 3.1, we ran some experiments in the gridworld domains from

Original: Fu et al.
Revised: Fu et al.

Original: In this case, we used the grid16smoothsparse and grid16randomsparse domains, which present challenging navigation tasks in a maze under a 0-1 sparse reward signal, provided at the end of the trajectory.
Revised: In this case, we used the grid16smoothsparse and grid16randomsparse domains, which present challenging navigation tasks in a maze under a 0-1 sparse reward signal, provided at the end of the trajectory.

Original: Additionally, the observations available to the ofﬂine RL agent do not consist of the raw ( x, y ) locations of the agent in the maze, but rather high-dimensional randomly chosen transformations of ( x, y ) in the case of grid16randomsparse , which are additionally smoothed locally around a particular state to obtain grid16smoothsparse .
Revised: Additionally, the observations available to the ofﬂine RL agent do not consist of the raw ( x, y ) locations of the agent in the maze, but rather high-dimensional randomly chosen transformations of ( x, y ) in the case of grid16randomsparse , which are additionally smoothed locally around a particular state to obtain grid16smoothsparse .

Original: Since our goal is to compare feature co-adaptation in TD-learning and ofﬂine SARSA, we consider a case where we evaluate a “mixed” behavior policy that chooses the optimal action with a probability of 0.7 at a given state, and chooses a random, suboptimal action with 0.3.
Revised: Since our goal is to compare feature co-adaptation in TD-learning and ofﬂine SARSA, we consider a case where we evaluate a “mixed” behavior policy that chooses the optimal action with a probability of 0.7 at a given state, and chooses a random, suboptimal action with 0.3.

Original: We then generate a dataset of size 256 transitions and train ofﬂine SARSA and TD-learning on this data.
Revised: We then generate a dataset of size 256 transitions and train ofﬂine SARSA and TD-learning on this data.

Original: While SARSA backups the next action observed in the ofﬂine dataset, TD-learning computes a full expectation of the Q-function E a ′ ∼ π β ( ·| s ′ )
Revised: While SARSA backups the next action observed in the ofﬂine dataset, TD-learning computes a full expectation of the Q-function E a ′ ∼ π β ( ·| s ′ )

Original: [ Q ( s ′ , a ′ )] under the behavior policy for computing Bellman backup targets.
Revised: [ Q ( s ′ , a ′ )] under the behavior policy for computing Bellman backup targets.

Original: The behavior policy is fully known to the TD-learning agent.
Revised: The behavior policy is fully known to the TD-learning agent.

Original: Our Q-network consists of two hidden layers of size (1024 , 1024) as before.
Revised: Our Q-network consists of two hidden layers of size (1024 , 1024) as before.

Original: [Figure]
Revised: [Figure]

Original: to ofﬂine SARSA.
Revised: to ofﬂine SARSA.

Original: We present the trends in the feature dot products for TD-learning and ofﬂine SARSA in Figure A.12, averaged over three seeds.
Revised: We present the trends in the feature dot products for TD-learning and ofﬂine SARSA in Figure A.12, averaged over three seeds.

Original: Observe that the trends in the dot product values for TD-learning and ofﬂine SARSA closely follow each other for the initial few gradient steps, soon, the dot products in TD-learning start growing faster.
Revised: Observe that the trends in the dot product values for TD-learning and ofﬂine SARSA closely follow each other for the initial few gradient steps, soon, the dot products in TD-learning start growing faster.

Original: In contrast, the dot products for SARSA either saturate or start decreasing.
Revised: In contrast, the dot products for SARSA either saturate or start decreasing.

Original: The only difference between TD-learning and SARSA is the set of actions used to compute
Revised: The only difference between TD-learning and SARSA is the set of actions used to compute

Original: Bellman targets – while the actions used for computing Bellman backup targets in SARSA are in-sample actions and are observed in the dataset, the actions used by TDlearning may be out-of-sample, but are still within the distribution of the data-generating behavior policy.
Revised: Bellman targets – while the actions used for computing Bellman backup targets in SARSA are in-sample actions and are observed in the dataset, the actions used by TDlearning may be out-of-sample, but are still within the distribution of the data-generating behavior policy.

Original: This supports our empirical evidence in the main paper showing that out-of-sample actions can lead to feature co-adaptation.
Revised: This supports our empirical evidence in the main paper showing that out-of-sample actions can lead to feature co-adaptation.

Original: [Figure]
Revised: [Figure]

Original: A.8. F EATURE C O -A DAPTATION AND N ORMALIZED F EATURE S IMILARITIES
Revised: A.8. F EATURE C O -A DAPTATION AND N ORMALIZED F EATURE S IMILARITIES

Original: Note that we characterized feature co-adaptation via the dot products of features.
Revised: Note that we characterized feature co-adaptation via the dot products of features.

Original: In this section, we explore the trends in other notions of similarity, such as cosine similarity between ϕ ( s , a ) and ϕ ( s ′ , a ′ ) which measures the dot product of feature vectors at consecutive state-action tuples after normalization.
Revised: In this section, we explore the trends in other notions of similarity, such as cosine similarity between ϕ ( s , a ) and ϕ ( s ′ , a ′ ) which measures the dot product of feature vectors at consecutive state-action tuples after normalization.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: We plot the trend in the cosine similarity with and without DR3 for ﬁve Atari games in Figure A. with CQL, DQN and REM, and for the three MuJoCo tasks studied in Appendix A.3 in Figure A.14.
Revised: We plot the trend in the cosine similarity with and without DR3 for ﬁve Atari games in Figure A. with CQL, DQN and REM, and for the three MuJoCo tasks studied in Appendix A.3 in Figure A.14.

Original: We ﬁnd that the cosine similarity is generally very high on the Atari domains, close to 1, and not indicative of performance degradation.
Revised: We ﬁnd that the cosine similarity is generally very high on the Atari domains, close to 1, and not indicative of performance degradation.

Original: On the Ant and Walker2d MuJoCo domains, we ﬁnd that the cosine similarity ﬁrst rises up close to 1 and roughly saturates there.
Revised: On the Ant and Walker2d MuJoCo domains, we ﬁnd that the cosine similarity ﬁrst rises up close to 1 and roughly saturates there.

Original: On the Hopper domain, the cosine similarity even decreases over training.
Revised: On the Hopper domain, the cosine similarity even decreases over training.

Original: However we observe that the feature dot products are increasing for all the domains.
Revised: However we observe that the feature dot products are increasing for all the domains.

Original: Applying DR3 in both cases improves performance (as shown in earlier Appendix), and generally gives rise to reduced cosine similarity values, though it can also increase the cosine similarity values occasionally.
Revised: Applying DR3 in both cases improves performance (as shown in earlier Appendix), and generally gives rise to reduced cosine similarity values, though it can also increase the cosine similarity values occasionally.

Original: Furthermore, even when the cosine similarities were decreasing for the base
Revised: Furthermore, even when the cosine similarities were decreasing for the base

Original: algorithm
Revised: algorithm

Original: (e.g., in the case of Hopper), addition of DR3 reduced the feature dot products and helped improve performance.
Revised: (e.g., in the case of Hopper), addition of DR3 reduced the feature dot products and helped improve performance.

Original: This indicates that both the norm and directional alignment are contributors to the co-adaptation issue, which is what DR3 aims to ﬁx and independently directional alignment does not indicate poor performance.
Revised: This indicates that both the norm and directional alignment are contributors to the co-adaptation issue, which is what DR3 aims to ﬁx and independently directional alignment does not indicate poor performance.

Original: A. S TABILITY OF DR3 F ROM A G OOD S OLUTION
Revised: A. S TABILITY OF DR3 F ROM A G OOD S OLUTION

Original: In this appendix, we study the trend of CQL + DR3 when starting learning from a good initialization, which was studied in Figure 2.
Revised: In this appendix, we study the trend of CQL + DR3 when starting learning from a good initialization, which was studied in Figure 2.

Original: As shown in Figure A.15, while the performance for baseline CQL degrades signiﬁcantly (from 5000 at initialization on Asterix, performance degrades to ∼ 3200 by 100 iterations for base CQL), whereas the performance of DR3 only moves from 5000 to ∼ 4300.
Revised: As shown in Figure A.15, while the performance for baseline CQL degrades signiﬁcantly (from 5000 at initialization on Asterix, performance degrades to ∼ 3200 by 100 iterations for base CQL), whereas the performance of DR3 only moves from 5000 to ∼ 4300.

Original: A similar trend holds for Breakout.
Revised: A similar trend holds for Breakout.

Original: This means that the addition of DR3 does stabilize the learning
Revised: This means that the addition of DR3 does stabilize the learning

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: in the setup of Figure 2 to evaluate the stability of CQL + DR3 when starting training from a good solution.

Original: Observe that the performance of base CQL decays quickly from the good solution, but CQL + DR3 is relatively more stable.

Original: Additionally, the feature dot products for DR3 are much smaller compared to CQL.

Original: relative to the baseline algorithm.
Revised: relative to the baseline algorithm.

Original: Please note that we are not claiming that DR3 is unequivocally stable, but that improves stability relative to the base method.
Revised: Please note that we are not claiming that DR3 is unequivocally stable, but that improves stability relative to the base method.

Original: We also present the dot products for
Revised: We also present the dot products for

Original: DR3, which are also small, unlike increasing dot products for base CQL as shown in the ﬁgure.
Revised: DR3, which are also small, unlike increasing dot products for base CQL as shown in the ﬁgure.

Revised: A note about why the trend in Figure A.15 is different from Figure F.1: Note that these trends are different because when starting training from a good solution in Figure A.15, we do not retain the optimizer state.

Revised: This is because we intend to compare CQL and CQL + DR3 starting from a good

Revised: Q-function, without reasoning about how the Q-function was obtained.

Revised: Therefore, CQL + DR3 runs in these two ﬁgures are not comparable to each other.

Original: B E XTENDED R ELATED W ORK
Revised: Initial checkpoint we start from: We simply want a good Q-function checkpoint from some run of Q-learning to begin training from. Therefore, we obtained the initial Q-function from a single run of CQL + DR3 that attained good performance, and precisely, this corresponded to the one seed of CQL + DR3 with 0.3 multiplier on the DR3 term, which performed quite well. Note that CQL + DR3 with 0.3 multiplier on DR3 performs worse on an average across many runs, and the DR3 runs in Figure A.15 and Figure F.1 both use a DR3 multiplier of 0.03 (as mentioned in Appendix E.3), but we just used this single run for obtaining a good starting point.

Revised: As mentioned above in the previous paragraph, since the optimizer state is not carried over, the process of training to obtain the initial checkpoint does not affect the trends in Figure A.15.

Original: In this section, we brieﬂy review some extended related works, and in particular, try to connect feature co-adaptation and implicit regularization to various interesting results pertaining to RL lowerbounds with function approximation and self-supervised learning.
Revised: In this section, we brieﬂy review some extended related works, and in particular, try to connect feature co-adaptation and implicit regularization to various interesting results pertaining to RL lowerbounds with function approximation and self-supervised learning.

Original: Lower-bounds for ofﬂine RL.
Revised: Lower-bounds for ofﬂine RL.

Original: Zanette (2021) identiﬁes hard instances for ofﬂine TD learning of linear value functions when the provided features are “aliased”.
Revised: Zanette (2021) identiﬁes hard instances for ofﬂine TD learning of linear value functions when the provided features are “aliased”.

Original: Note that this work does not consider feature learning or implicit regularization, but their hardness result relies heavily on the fact the given linear features are aliased in a special sense.
Revised: Note that this work does not consider feature learning or implicit regularization, but their hardness result relies heavily on the fact the given linear features are aliased in a special sense.

Original: Aliased features utilized in the hard instance inhibit learning along certain dimensions of the feature space with TD-style updates, necessitating an exponential sample size for near-accurate value estimation, even under strong coverage assumptions.
Revised: Aliased features utilized in the hard instance inhibit learning along certain dimensions of the feature space with TD-style updates, necessitating an exponential sample size for near-accurate value estimation, even under strong coverage assumptions.

Original: A combination of Zanette (2021)’s argument, which provides a hard instance given aliased features, and our analysis, which studies the emergence of co-adapted/similar features in the ofﬂine deep RL setting, could imply that the co-adaptation can lead to failure modes from the hard instance, even on standard Ofﬂine RL problems, when provided with limited data.
Revised: A combination of Zanette (2021)’s argument, which provides a hard instance given aliased features, and our analysis, which studies the emergence of co-adapted/similar features in the ofﬂine deep RL setting, could imply that the co-adaptation can lead to failure modes from the hard instance, even on standard Ofﬂine RL problems, when provided with limited data.

Original: Connections to self-supervised learning (SSL).
Revised: Connections to self-supervised learning (SSL).

Original: Several modern self-supervised learning methods (Grill et al., 2020; Chen & He, 2021) can be viewwed as utilizing some form of bootstrapping where different augmentations of the same input ( x + Aug 1 , x + Aug 2 ) serve as consecutive stateaction tuples that appear on two sides of the backup.
Revised: Several modern self-supervised learning methods (Grill et al., 2020; Chen & He, 2021) can be viewwed as utilizing some form of bootstrapping where different augmentations of the same input ( x + Aug 1 , x + Aug 2 ) serve as consecutive stateaction tuples that appear on two sides of the backup.

Original: If we may extrapolate our reasoning of feature co-adaptation to this setting, it would suggest that performing noisy updates on a self-supervised bootstrapping loss will give us feature representations that are highly similar for consecutive stateaction tuples, i.e., the representations for ϕ ( x + Aug 1 )
Revised: If we may extrapolate our reasoning of feature co-adaptation to this setting, it would suggest that performing noisy updates on a self-supervised bootstrapping loss will give us feature representations that are highly similar for consecutive stateaction tuples, i.e., the representations for ϕ ( x + Aug 1 )

Original: ⊤ ϕ
Revised: ⊤ ϕ

Original: ( x + Aug 2 ) will be high.
Revised: ( x + Aug 2 ) will be high.

Original: Intuitively, an easy way for obtaining high feature dot products is for ϕ ( · ) to capture only that information in · , which is agnostic to data augmentation, thus giving rise to features that are invariant to transformations.
Revised: Intuitively, an easy way for obtaining high feature dot products is for ϕ ( · ) to capture only that information in · , which is agnostic to data augmentation, thus giving rise to features that are invariant to transformations.

Original: This aligns with what has been shown in self-supervised learning (Tian et al., 2020; 2021).
Revised: This aligns with what has been shown in self-supervised learning (Tian et al., 2020; 2021).

Original: Another inter- esting point to note is that while such an explanation would indicate that highly co-adapted features are beneﬁcial in SSL, such features can be adverse in value-based RL as discussed in Section 3.
Revised: Another interesting point to note is that while such an explanation would indicate that highly co-adapted features are beneﬁcial in SSL, such features can be adverse in value-based RL as discussed in Section 3.

Original: Preventing divergence in deep TD-learning.
Revised: Preventing divergence in deep TD-learning.

Original: Finally, we discuss Achiam et al.
Revised: Finally, we discuss Achiam et al.

Original: (2019) which proposes to pre-condition the TD-update using the inverse the neural tangent kernel (Jacot et al., 2018) matrix so that the TD-update is always a contraction, for every θ k found during TD-learning.
Revised: (2019) which proposes to pre-condition the TD-update using the inverse the neural tangent kernel (Jacot et al., 2018) matrix so that the TD-update is always a contraction, for every θ k found during TD-learning.

Original: Intuitively, this can be overly restrictive in several cases: we do not need to ensure that TD always contracts, but that is eventually stabilizes at good solution over long periods of running noisy TD updates, Our implicit regularizer (Equation‘4) derives this condition, and our theoretically-inspired
Revised: Intuitively, this can be overly restrictive in several cases: we do not need to ensure that TD always contracts, but that is eventually stabilizes at good solution over long periods of running noisy TD updates, Our implicit regularizer (Equation‘4) derives this condition, and our theoretically-inspired

Original: DR3 regularizer shows that empirically, it sufﬁces to penalize the dot product similarity in practice.
Revised: DR3 regularizer shows that empirically, it sufﬁces to penalize the dot product similarity in practice.

Original: C P ROOF OF T HEOREM 3. 
Revised: C P ROOF OF T HEOREM 3.

Original: In this section, we will derive our implicit regularizer R TD ( θ ) that emerges when performing TD updates with a stochastic noise model with covariance matrix M .
Revised: In this section, we will derive our implicit regularizer R TD ( θ ) that emerges when performing TD updates with a stochastic noise model with covariance matrix M .

Original: We ﬁrst introduce our notation that we will use throughout the proof, then present our assumptions and ﬁnally derive the regularizer.
Revised: We ﬁrst introduce our notation that we will use throughout the proof, then present our assumptions and ﬁnally derive the regularizer.

Original: Our proof utilizes the analysis techniques from Blanc et al.
Revised: Our proof utilizes the analysis techniques from Blanc et al.

Original: (2020) and Damian et al.
Revised: (2020) and Damian et al.

Original: (2021), which analyze label-noise SGD for supervised learning, however key modiﬁcations need to be made to their arguments to account for non-symmetric matrices that emerge in TD learning.
Revised: (2021), which analyze label-noise SGD for supervised learning, however key modiﬁcations need to be made to their arguments to account for non-symmetric matrices that emerge in TD learning.

Original: As a result, the form of the resulting regularizer is very different.
Revised: As a result, the form of the resulting regularizer is very different.

Original: To keep the proof concise, we will appeal to lemmas from these prior works which will allow us to bound certain concentration terms.
Revised: To keep the proof concise, we will appeal to lemmas from these prior works which will allow us to bound certain concentration terms.

Original: C. N OTATION
Revised: C. N OTATION

Original: The noisy TD-learning update for training the Q-function is given by:
Revised: The noisy TD-learning update for training the Q-function is given by:

Original: [Equation]
Revised: [Equation]

Original: where g ( θ ) denotes the parameter update.
Revised: where g ( θ ) denotes the parameter update.

Original: Note that g ( θ ) is not a full gradient of a scalar objective, but it is a form of a “pseudo”-gradient or “semi”-gradient.
Revised: Note that g ( θ ) is not a full gradient of a scalar objective, but it is a form of a “pseudo”-gradient or “semi”-gradient.

Original: Let ε k denote an i.i.d.random noise that is added to each update.
Revised: Let ε k denote an i.i.d.random noise that is added to each update.

Original: This noise is sampled from a zero-mean Gaussian random variable with covariance matrix M , i.e., N (0 , M ) .
Revised: This noise is sampled from a zero-mean Gaussian random variable with covariance matrix M , i.e., N (0 , M ) .

Original: Let θ ∗ denote a point in the parameter space such that in the vicinity of θ ∗ , g ( θ ) ≤ C , for a small enough C .
Revised: Let θ ∗ denote a point in the parameter space such that in the vicinity of θ ∗ , g ( θ ) ≤ C , for a small enough C .

Original: Let G ( θ ) denote the derivative of g ( θ ) w.r.t.
Revised: Let G ( θ ) denote the derivative of g ( θ ) w.r.t.

Original: θ : G ( θ ) =
Revised: θ : G ( θ ) =

Original: ∇ θ g ( θ ) and let ∇ G ( θ ) denote the third-order tensor ∇ 2 θ g ( θ ) .
Revised: ∇ θ g ( θ ) and let ∇ G ( θ ) denote the third-order tensor ∇ 2 θ g ( θ ) .

Original: For notation clarity, let G = G ( θ ∗ ) , ∇ G = ∇ G ( θ ∗ ) .
Revised: For notation clarity, let G = G ( θ ∗ ) , ∇ G = ∇ G ( θ ∗ ) .

Original: Let e i denote the signed TD error for a given transition ( s i , a i , s ′ i )
Revised: Let e i denote the signed TD error for a given transition ( s i , a i , s ′ i )

Original: ∈ D at θ ∗ :
Revised: ∈ D at θ ∗ :

Original: [Equation]
Revised: [Equation]

Original: Since θ ∗ is a ﬁxed point of the training TD error, e i = 0 .
Revised: Since θ ∗ is a ﬁxed point of the training TD error, e i = 0 .

Original: Following Blanc et al.
Revised: Following Blanc et al.

Original: (2020), we will assume that the learning rate in gradient descent, η , is small and we will ignore terms that scale as O ( η 1+ δ ) , for δ > 0 .
Revised: (2020), we will assume that the learning rate in gradient descent, η , is small and we will ignore terms that scale as O ( η 1+ δ ) , for δ > 0 .

Original: Our proof will rely on using a reference Ornstein-Uhlenbeck (OU) process which the TD parameter iterates will be compared to.
Revised: Our proof will rely on using a reference Ornstein-Uhlenbeck (OU) process which the TD parameter iterates will be compared to.

Original: Let ζ k denote the k -th iterate of an OU process, which is deﬁned as:
Revised: Let ζ k denote the k -th iterate of an OU process, which is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: We will drop θ from ∇ θ to indicate that the gradient is being computed at θ ∗ , and drop ( s i , a i ) from
Revised: We will drop θ from ∇ θ to indicate that the gradient is being computed at θ ∗ , and drop ( s i , a i ) from

Original: Q ( s i , a i ) and instead represent it as Q i for brevity; we will represent Q ( s ′ i , a ′ i ) as Q ′ i .
Revised: Q ( s i , a i ) and instead represent it as Q i for brevity; we will represent Q ( s ′ i , a ′ i ) as Q ′ i .

Original: We assume that ∇ 2 Q i is L 2 -Lipschitz and ∇ 3 Q i is L 3 -Lipschitz throughout the parameter space Θ .
Revised: We assume that ∇ 2 Q i is L 2 -Lipschitz and ∇ 3 Q i is L 3 -Lipschitz throughout the parameter space Θ .

Original: C. P ROOF S TRATEGY
Revised: C. P ROOF S TRATEGY

Original: For a given point θ ∗ to be an attractive ﬁxed point of TD-learning, our proof strategy would be to derive the condition under which it mimics a given OU noise process, which as we will show stays close to the parameter θ ∗ .
Revised: For a given point θ ∗ to be an attractive ﬁxed point of TD-learning, our proof strategy would be to derive the condition under which it mimics a given OU noise process, which as we will show stays close to the parameter θ ∗ .

Original: This condition would then be interpreted as the gradient of a “induced” implicit regularizer.
Revised: This condition would then be interpreted as the gradient of a “induced” implicit regularizer.

Original: If the point θ ∗ is not a stationary point of this regularizer, we will show that the movement θ is large when running the noisy TD updates, indicating that the regularizer, atleast in part guides the dynamics of TD-learning.
Revised: If the point θ ∗ is not a stationary point of this regularizer, we will show that the movement θ is large when running the noisy TD updates, indicating that the regularizer, atleast in part guides the dynamics of TD-learning.

Original: To show this, we would write out the gradient update, isolate some terms that will give rise to the implicit regularizer, and bound the remaining terms using contraction and concentration arguments.
Revised: To show this, we would write out the gradient update, isolate some terms that will give rise to the implicit regularizer, and bound the remaining terms using contraction and concentration arguments.

Original: The contraction arguments largely follow prior work (though with key exceptions in handling contraction with asymmetric and complex eigenvalue matrices), while the form of the implicit regularizer is different.
Revised: The contraction arguments largely follow prior work (though with key exceptions in handling contraction with asymmetric and complex eigenvalue matrices), while the form of the implicit regularizer is different.

Original: Finally, we will interpret the resulting update over large timescales to show that learning is indeed guided by the implicit regularizer.
Revised: Finally, we will interpret the resulting update over large timescales to show that learning is indeed guided by the implicit regularizer.

Original: C. A SSUMPTIONS AND C ONDITIONS
Revised: C. A SSUMPTIONS AND C ONDITIONS

Original: Next, we present some key assumptions we will need for the proof.
Revised: Next, we present some key assumptions we will need for the proof.

Original: Our ﬁrst assumption is that the matrix G ∈ R
Revised: Our ﬁrst assumption is that the matrix G ∈ R

Original: d × d is of maximal rank possible, which is equal to the number of datapoints n and n ≪ d , the dimensionality of the parameter space.
Revised: d × d is of maximal rank possible, which is equal to the number of datapoints n and n ≪ d , the dimensionality of the parameter space.

Original: Crucially, this assumption do not imply that G is of full rank – it cannot be, because we are in the overparameterized regime.
Revised: Crucially, this assumption do not imply that G is of full rank – it cannot be, because we are in the overparameterized regime.

Original: Assumption A1 ( G spans an n -dimensional basis.) .
Revised: Assumption A1 ( G spans an n -dimensional basis.) .

Original: Assume that the matrix G spans n -possible directions in the parameter space and hence, attains the maximal possible rank it can.
Revised: Assume that the matrix G spans n -possible directions in the parameter space and hence, attains the maximal possible rank it can.

Original: The second condition we require is that the matrices P i ∇ Q i ∇ Q ⊤ i and M share the same ndimensional basis as matrix G :
Revised: The second condition we require is that the matrices P i ∇ Q i ∇ Q ⊤ i and M share the same ndimensional basis as matrix G :

Original: [Equation]
Revised: Assumption A2. P i ∇ Q i ∇ Q ⊤ i , M , and G span identical n -dimensional subspaces.

Original: This is a technical condition that is required.
Revised: This is a technical condition that is required.

Original: If this condition is not met, as we will show the learning dynamics of noisy TD will not be a contraction in certain direction in the parameter space and TDlearning will not stabilize at such a solution θ ∗ .
Revised: If this condition is not met, as we will show the learning dynamics of noisy TD will not be a contraction in certain direction in the parameter space and TDlearning will not stabilize at such a solution θ ∗ .

Original: In fact, we will utilize a stronger version of this statement for TD-learning to converge, and we will discuss this shortly.
Revised: In fact, we will utilize a stronger version of this statement for TD-learning to converge, and we will discuss this shortly.

Original: C. L EMMAS U SED I N T HE P ROOF
Revised: C. L EMMAS U SED I N T HE P ROOF

Original: Next, we present some lemmas that would be useful for proving the theoretical result.
Revised: Next, we present some lemmas that would be useful for proving the theoretical result.

Original: Lemma C.1 (Expressions for the ﬁrst and and second-order derivatives of g ( θ ) .) .
Revised: Lemma C.1 (Expressions for the ﬁrst and and second-order derivatives of g ( θ ) .) .

Original: The following deﬁnitions and expansions apply to our proof:
Revised: The following deﬁnitions and expansions apply to our proof:

Original: [Equation]
Revised: [Equation]

Original: Lemma C.1 presents a decomposition of the matrix G and the directional derivative of the third order tensor ∇ G [ v , v ] in directions v and v , which will appear in the Taylor expansion layer.
Revised: Lemma C.1 presents a decomposition of the matrix G and the directional derivative of the third order tensor ∇ G [ v , v ] in directions v and v , which will appear in the Taylor expansion layer.

Original: Note that at θ ∗ since e i = 0 , the ﬁrst term in G ( θ ∗ ) and the third term in ∇ G ( θ ∗ )[ v , v ] vanish.
Revised: Note that at θ ∗ since e i = 0 , the ﬁrst term in G ( θ ∗ ) and the third term in ∇ G ( θ ∗ )[ v , v ] vanish.

Original: Lemma C. derives a ﬁxed-point recursion for the covariance matrix of the total noise accumulated in the OUprocess with covariance matrix M and this will appear in our proof.
Revised: Lemma C. derives a ﬁxed-point recursion for the covariance matrix of the total noise accumulated in the OUprocess with covariance matrix M and this will appear in our proof.

Original: Lemma C.2 (Covariance of the random noise process ζ k ) .
Revised: Lemma C.2 (Covariance of the random noise process ζ k ) .

Original: Let ζ k denote the OU process satisfying: ζ k +

Revised: Let ζ k denote the OU process satisfying: ζ k +

Original: =
Revised: =

Original: ( I − ηG ) ζ k
Revised: ( I − ηG ) ζ k

Original: + ηε k , where ε k ∼ N (0 , M ) , where M ≽ 0 .
Revised: + ηε k , where ε k ∼ N (0 , M ) , where M ≽ 0 .

Original: Then, ζ k +1 ∼ N (0 , Σ) , where Σ satisﬁes the discrete Lyapunov equation:
Revised: Then, ζ k +1 ∼ N (0 , Σ) , where Σ satisﬁes the discrete Lyapunov equation:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the OU process, ζ k +

Revised: For the OU process, ζ k +

Original: =
Revised: =

Original: ( I − ηG ) ζ k
Revised: ( I − ηG ) ζ k

Original: + ηε k , since ε k is a Gaussian random variable, by induction so is ζ k +1 , and therefore the covariance matrix of ζ k +1 is given by:
Revised: + ηε k , since ε k is a Gaussian random variable, by induction so is ζ k +1 , and therefore the covariance matrix of ζ k +1 is given by:

Original: [Equation]
Revised: [Equation]

Original: Solving for the ﬁxed point for Σ k gives the desired expression.
Revised: Solving for the ﬁxed point for Σ k gives the desired expression.

Original: In our proofs, we will require the following contraction lemmas to tightly bound the magnitude of some zero-mean terms that will appear in the noisy TD update under certain scenarios.
Revised: In our proofs, we will require the following contraction lemmas to tightly bound the magnitude of some zero-mean terms that will appear in the noisy TD update under certain scenarios.

Original: Unlike the analysis in Damian et al.
Revised: Unlike the analysis in Damian et al.

Original: (2021) and Blanc et al.
Revised: (2021) and Blanc et al.

Original: (2020) for supervised learning with label noise, where the contraction terms like ( I − ηG ) k G are bounded by ≈ 1 kη intuitively because I − ηG is a contraction in the subspace spanned by matrix G .
Revised: (2020) for supervised learning with label noise, where the contraction terms like ( I − ηG ) k G are bounded by ≈ 1 kη intuitively because I − ηG is a contraction in the subspace spanned by matrix G .

Original: However, this is not true for TD-learning directly since terms like ( I − ηG ) k S appear for a different matrix S .
Revised: However, this is not true for TD-learning directly since terms like ( I − ηG ) k S appear for a different matrix S .

Original: Therefore, TD-learning will diverge from θ ∗ unless matrices G and M have their corresponding eigenvectors assigned to the top eigenvalues be approximately “aligned”.
Revised: Therefore, TD-learning will diverge from θ ∗ unless matrices G and M have their corresponding eigenvectors assigned to the top eigenvalues be approximately “aligned”.

Original: We formalize this deﬁnition next, and then provide a proof of the concentration guarantee.
Revised: We formalize this deﬁnition next, and then provide a proof of the concentration guarantee.

Original: Deﬁnition 1 ( ( ω, C 0 ) -alignment) .
Revised: Deﬁnition 1 ( ( ω, C 0 ) -alignment) .

Original: Given a positive semideﬁnite matrix A , let A
Revised: Given a positive semideﬁnite matrix A , let A

Original: =
Revised: =

Original: U
Revised: U

Original: A Λ A
Revised: A Λ A

Original: U ⊤ A denote its eigendecomposition.
Revised: U ⊤ A denote its eigendecomposition.

Original: Without loss of generality assume that the eiegenvalues are arranged in decreasing order, i.e., ∀ i > j, Λ A ( i ) ≤ Λ A ( j ) .
Revised: Without loss of generality assume that the eiegenvalues are arranged in decreasing order, i.e., ∀ i > j, Λ A ( i ) ≤ Λ A ( j ) .

Original: Given another matrix B , let B =
Revised: Given another matrix B , let B =

Original: U B Λ B U HB denote its complex eigendecomposition, where eigenvalues in Λ B are arranged in decreasing order of their complex magnitudes, i.e., ∀ i > j, | Λ B ( i ) | ≤ | Λ B ( j ) | .
Revised: U B Λ B U HB denote its complex eigendecomposition, where eigenvalues in Λ B are arranged in decreasing order of their complex magnitudes, i.e., ∀ i > j, | Λ B ( i ) | ≤ | Λ B ( j ) | .

Original: Then the matrix pair ( A, B ) is said to be ( ω, C 0 ) -aligned
Revised: Then the matrix pair ( A, B ) is said to be ( ω, C 0 ) -aligned

Original: if | U HB ( i ) U
Revised: if | U HB ( i ) U

Original: A ( i ) | ≤ ω and if ∀ i, Λ A ( i ) ≤ C 

Revised: A ( i ) | ≤ ω and if ∀ i, Λ A ( i ) ≤ C

Original: | Λ B ( i ) | for a constant C 0 .
Revised: | Λ B ( i ) | for a constant C 0 .

Original: If two matrices are ( ω, C 0 ) -aligned, this means that the corresponding eigenvectors when arranged in decreasing order of eigenvalue magnitude roughly align with each other.
Revised: If two matrices are ( ω, C 0 ) -aligned, this means that the corresponding eigenvectors when arranged in decreasing order of eigenvalue magnitude roughly align with each other.

Original: This condition would be crucial while deriving the implicit regularizer as it will quantify the rate of contraction of certain terms that deﬁne the neighborhood that the iterates of noisy TD-learning will lie in with high probability.
Revised: This condition would be crucial while deriving the implicit regularizer as it will quantify the rate of contraction of certain terms that deﬁne the neighborhood that the iterates of noisy TD-learning will lie in with high probability.

Original: We will operate in the setting when the matrix G and P i ∇ Q i ∇ Q ⊤ i are ( ω, C 0 ) -aligned with each other, and matrix M and G are also ( ω, C 0 ) -aligned
Revised: We will operate in the setting when the matrix G and P i ∇ Q i ∇ Q ⊤ i are ( ω, C 0 ) -aligned with each other, and matrix M and G are also ( ω, C 0 ) -aligned

Original: (note that we can consider ω ′ , C ′ 0 ) , which will not change our bounds and therefore we go for less notational clutter).
Revised: (note that we can consider ω ′ , C ′ 0 ) , which will not change our bounds and therefore we go for less notational clutter).

Original: Next we utilize this notion of alignment to show a particular contraction bound that extends the weak contraction bound in Damian et al.
Revised: Next we utilize this notion of alignment to show a particular contraction bound that extends the weak contraction bound in Damian et al.

Original: Lemma C.3.
Revised: Lemma C.3.

Original: Assume we are given a matrix G such that |
Revised: Assume we are given a matrix G such that |

Original: λ i ( I − ηG ) | ≤ ρ 0 < 1 for all λ i such that
Revised: λ i ( I − ηG ) | ≤ ρ 0 < 1 for all λ i such that

Original: λ i ̸
Revised: λ i ̸

Original: = 0 .
Revised: = 0 .

Original: Let G
Revised: Let G

Original: = U Λ U H be the complex eigenvalue decomposition of G (since almost every matrix is complex-diagonalizable).
Revised: = U Λ U H be the complex eigenvalue decomposition of G (since almost every matrix is complex-diagonalizable).

Original: For a positive semi-deﬁnite matrix S that is ( ω, C 0 ) -aligned with G , if
Revised: For a positive semi-deﬁnite matrix S that is ( ω, C 0 ) -aligned with G , if

Original: S =
Revised: S =

Original: U S Λ S U ⊤ S is its eigenvalue decomposition, the following contraction bound holds:
Revised: U S Λ S U ⊤ S is its eigenvalue decomposition, the following contraction bound holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To prove this statement, we can expand ( I − ηG ) using its eigenvalue decomposition only in the subspace that is jointly shared by G and M , and then utilize the deﬁnition of ω -alignment to bound the terms.
Revised: To prove this statement, we can expand ( I − ηG ) using its eigenvalue decomposition only in the subspace that is jointly shared by G and M , and then utilize the deﬁnition of ω -alignment to bound the terms.

Original: [Equation]
Revised: [Equation]

Original: Now we need to solve for the inner maximization term.
Revised: Now we need to solve for the inner maximization term.

Original: When Λ( i ) is not complex for any i , the term above is ≲ 1 /ηk using the result from Damian et al.
Revised: When Λ( i ) is not complex for any i , the term above is ≲ 1 /ηk using the result from Damian et al.

Original: but when Λ( i ) is complex, this bound can only hold under certain conditions.
Revised: but when Λ( i ) is complex, this bound can only hold under certain conditions.

Original: To note when this quantity is bounded, we expand | 1 − ηx | k for some complex number x = r 
Revised: To note when this quantity is bounded, we expand | 1 − ηx | k for some complex number x = r (cos θ

Original: (cos θ + ι sin θ ) :
Revised: [Equation] sin θ ) :

Original: [Equation]
Revised: [Equation]

Original: Plugging back the above expression in the bound above completes the proof.
Revised: Plugging back the above expression in the bound above completes the proof.

Original: The proof of Lemma C.3 indicates that unless the learning rate η and the matrix G are such that the | λ i ( I − ηG ) | ≤ ρ < 1 in directions spanned by matrix S , such an expression may not converge.
Revised: The proof of Lemma C.3 indicates that unless the learning rate η and the matrix G are such that the | λ i ( I − ηG ) | ≤ ρ < 1 in directions spanned by matrix S , such an expression may not converge.

Original: This is expected since the matrix I − ηG will not contract in directions of non-zero eigenvalues if the real part r cos θ is negative or zero.
Revised: This is expected since the matrix I − ηG will not contract in directions of non-zero eigenvalues if the real part r cos θ is negative or zero.

Original: Additionally, we note that under Deﬁnition 1, we can extend several weak-contraction bounds from Damian et al.
Revised: Additionally, we note that under Deﬁnition 1, we can extend several weak-contraction bounds from Damian et al.

Original: (Lemmas 9-14 in Damian et al.
Revised: (Lemmas 9-14 in Damian et al.

Original: to our setting.
Revised: to our setting.

Original: Next, Lemma C.4 shows that the OU noise iterates are bounded with high probability when Deﬁnition 1 holds:
Revised: Next, Lemma C.4 shows that the OU noise iterates are bounded with high probability when Deﬁnition 1 holds:

Original: Lemma C.4 ( ζ k is bounded with high probability) .
Revised: Lemma C.4 ( ζ k is bounded with high probability) .

Original: With probability atleast 1 − δ and under Deﬁnition 1, 
Revised: With probability atleast 1 − δ and under Deﬁnition 1, || ζ k || ≤ nω √ ηC 0 log 1 δ

Original: [Equation]
Revised: = O ( √ η ) .

Original: Proof.
Revised: Proof.

Original: To prove this lemma, we ﬁrst bound the trace of the covariance matrix Σ k +1 and then apply high probability bounds on the Martingale norm concentration.
Revised: To prove this lemma, we ﬁrst bound the trace of the covariance matrix Σ k +1 and then apply high probability bounds on the Martingale norm concentration.

Original: The trace of the covariance matrix Σ k +1 can be bounded as follows (all the equations below are restricted to the dimensions of non-zero eigenvalues of G ):
Revised: The trace of the covariance matrix Σ k +1 can be bounded as follows (all the equations below are restricted to the dimensions of non-zero eigenvalues of G ):

Original: [Equation]
Revised: [Equation]

Original: Now, we can apply Corollary 1 from Damian et al.
Revised: Now, we can apply Corollary 1 from Damian et al.

Original: (2021) to obtain a bound on || ζ k || as with high probability, atleast 1 − δ , || ζ k || ≤
Revised: (2021) to obtain a bound on || ζ k || as with high probability, atleast 1 − δ , || ζ k || ≤

Original: [Equation]
Revised: [Equation]

Original: C. M AIN P ROOF OF T HEOREM 3. 
Revised: C. M AIN P ROOF OF T HEOREM 3.

Original: In this section, we present the main proof of Theorem 3.1.
Revised: In this section, we present the main proof of Theorem 3.1.

Original: The proof involves two components: (1) the part where we derive the regularizer, and (2) bounding additional terms via concentration inequalities.
Revised: The proof involves two components: (1) the part where we derive the regularizer, and (2) bounding additional terms via concentration inequalities.

Original: Part (1) is speciﬁc to TD-learning, while a lot of the machinery for part (2) is directly taken from prior work (Damian et al., 2021) and Blanc et al.
Revised: Part (1) is speciﬁc to TD-learning, while a lot of the machinery for part (2) is directly taken from prior work (Damian et al., 2021) and Blanc et al.

Original: We focus on part (1) here.
Revised: We focus on part (1) here.

Original: Our strategy is to analyze the learning dynamics of noisy TD updates that originate at θ ∗ .
Revised: Our strategy is to analyze the learning dynamics of noisy TD updates that originate at θ ∗ .

Original: In a small neighborhood around θ ∗ , we can expand the noisy TD update (Equation 3) using Taylor’s expansion around θ ∗ which gives:[Equation]
Revised: In a small neighborhood around θ ∗ , we can expand the noisy TD update (Equation 3) using Taylor’s expansion around θ ∗ which gives:

Original: [Equation]
Revised: [Equation]

Original: Denoting ν k := θ k − θ ∗ , using the fact that || g ( θ ∗ ) || ≤ C , we ﬁnd that ν k can be written as:
Revised: Denoting ν k := θ k − θ ∗ , using the fact that || g ( θ ∗ ) || ≤ C , we ﬁnd that ν k can be written as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Since the OU process ζ k stays in the vicinity of the point θ ∗ , and follows a similar recursion to the one above, our goal would be to design a regularizer so that Equation C.21 closely follows the OU process.
Revised: Since the OU process ζ k stays in the vicinity of the point θ ∗ , and follows a similar recursion to the one above, our goal would be to design a regularizer so that Equation C.21 closely follows the OU process.

Original: Thus, we would want to bound the difference between the variable ν k and the variable ζ k , denoted as r k to be within a small neighborhood:
Revised: Thus, we would want to bound the difference between the variable ν k and the variable ζ k , denoted as r k to be within a small neighborhood:

Original: [Equation]
Revised: [Equation]

Original: We can write down an expression for r k summing over all the terms:
Revised: We can write down an expression for r k summing over all the terms:

Original: [Equation]
Revised: [Equation]

Original: Term (a) in the above equation is the one that can induce a displacement in r k as k increases and would be used to derive the regularizer, whereas term (b) primarily consists of terms that concentrate to 0 .
Revised: Term (a) in the above equation is the one that can induce a displacement in r k as k increases and would be used to derive the regularizer, whereas term (b) primarily consists of terms that concentrate to 0 .

Original: We ﬁrst analyze term (a)
Revised: We ﬁrst analyze term (a)

Original: and then we will analyze the concentration terms later.
Revised: and then we will analyze the concentration terms later.

Original: To analyze term (a), note that the term ∇ G [ ν k , ν k ] , by Lemma C.1, only depends on ν k via the covariance matrix ν k ν ⊤ k .
Revised: To analyze term (a), note that the term ∇ G [ ν k , ν k ] , by Lemma C.1, only depends on ν k via the covariance matrix ν k ν ⊤ k .

Original: So we will partition this term into two terms: (i) a term that utilizes the asymptotic covariance matrix of the OU process and (ii) errors due to a ﬁnite k and stochasticity that will concentrate.
Revised: So we will partition this term into two terms: (i) a term that utilizes the asymptotic covariance matrix of the OU process and (ii) errors due to a ﬁnite k and stochasticity that will concentrate.

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is a “bias” term and doesn’t concentrate to 0 , and will give rise to the regularizer.
Revised: The ﬁrst term is a “bias” term and doesn’t concentrate to 0 , and will give rise to the regularizer.

Original: We can break this term using Lemma C.1 as:
Revised: We can break this term using Lemma C.1 as:

Original: [Equation]
Revised: [Equation]

Original: The regularizer R TD ( θ ) is the function such that:
Revised: The regularizer R TD ( θ ) is the function such that:

Original: [Equation]
Revised: [Equation]

Original: where [[ · ]] denotes the stop gradient operator.
Revised: where [[ · ]] denotes the stop gradient operator.

Original: If the point θ ∗ is a stationary point of the regularizer
Revised: If the point θ ∗ is a stationary point of the regularizer

Original: R TD ( θ ) , then Equations C.26 and C.27 imply that the ﬁrst term of Equation C.25 must be 0.
Revised: R TD ( θ ) , then Equations C.26 and C.27 imply that the ﬁrst term of Equation C.25 must be 0.

Original: Therefore in this case to show that θ ∗ is attractive, we need to show that the other terms in Equations C.25,
Revised: Therefore in this case to show that θ ∗ is attractive, we need to show that the other terms in Equations C.25,

Original: C.24 and term (b) in Equation C.22 concentrate around 0 and are bounded in magnitude.
Revised: C.24 and term (b) in Equation C.22 concentrate around 0 and are bounded in magnitude.

Original: The remaining part of the proof shown in Appendix C.7 provides these details, but we ﬁrst summarize the main takeaways in the proof to conclude the argument.
Revised: The remaining part of the proof shown in Appendix C.7 provides these details, but we ﬁrst summarize the main takeaways in the proof to conclude the argument.

Original: C. S UMMARY OF THE A RGUMENT
Revised: C. S UMMARY OF THE A RGUMENT

Original: We will show how to concentrate terms in Equation C.26 besides the regularizer largely following the techniques from prior work, but we ﬁrst summarize the entire proof.
Revised: We will show how to concentrate terms in Equation C.26 besides the regularizer largely following the techniques from prior work, but we ﬁrst summarize the entire proof.

Original: The overall update to the vector r k which measures the displacement between the parameter vector θ k − θ ∗ and the OUprocess ζ k can be written as follows, and it is governed by the derivative of the implicit regularizer (modulo error terms):
Revised: The overall update to the vector r k which measures the displacement between the parameter vector θ k − θ ∗ and the OUprocess ζ k can be written as follows, and it is governed by the derivative of the implicit regularizer (modulo error terms):

Original: [Equation]
Revised: [Equation]

Original: (C.28)
Revised: (C.28)

Original: An important detail to note here is that since the regularizer consists of Σ ∗ M and the size of Σ ∗ M (i.e, its eigenvalues), as shown in Lemma C.4 depends on one factor of η .
Revised: An important detail to note here is that since the regularizer consists of Σ ∗ M and the size of Σ ∗ M (i.e, its eigenvalues), as shown in Lemma C.4 depends on one factor of η .

Original: So, effectively the ﬁrst term in Equation C.28 does depend on two factors of η .
Revised: So, effectively the ﬁrst term in Equation C.28 does depend on two factors of η .

Original: Using Equation C.28, we can write the deviation between θ ∗ and θ k as:
Revised: Using Equation C.28, we can write the deviation between θ ∗ and θ k as:

Original: [Equation]
Revised: [Equation]

Original: The OU process ζ k converges to θ ∗ in the subspace spanned by G , since the condition ρ ( I − ηG ) <is active in this subspace (if the condition that ρ ( I − ηG ) < 1 in the subspace spanned by G is not true, then as Ghosh & Bellemare (2020) show, TD can diverge).
Revised: The OU process ζ k converges to θ ∗ in the subspace spanned by G , since the condition ρ ( I − ηG ) <is active in this subspace (if the condition that ρ ( I − ηG ) < 1 in the subspace spanned by G is not true, then as Ghosh & Bellemare (2020) show, TD can diverge).

Original: Now, given G satisﬁes this spectral radius condition, ζ k would converge to θ ∗ within a timescale of O (cid:16)η (cid:17) within this subspace, which as Blanc et al.
Revised: Now, given G satisﬁes this spectral radius condition, ζ k would converge to θ ∗ within a timescale of O (cid:16)η (cid:17) within this subspace, which as Blanc et al.

Original: (2020) put it is the strength of the “mean-reversion” term.
Revised: (2020) put it is the strength of the “mean-reversion” term.

Original: On the remaining directions (note that d ≫ n ), the dynamics is guided by the regularizer, although with a smaller weight of η 2 .
Revised: On the remaining directions (note that d ≫ n ), the dynamics is guided by the regularizer, although with a smaller weight of η 2 .

Original: C. A DDITIONAL P ROOF D ETAILS : C ONCENTRATING O THER T ERMS
Revised: C. A DDITIONAL P ROOF D ETAILS : C ONCENTRATING O THER T ERMS

Original: We ﬁrst concentrate the terms in Equation C.25.
Revised: We ﬁrst concentrate the terms in Equation C.25.

Original: The cumulative effect of the second term in Equation C.25 is given by:
Revised: The cumulative effect of the second term in Equation C.25 is given by:

Original: [Equation]
Revised: [Equation]

Original: which follows from the fact that ∇ 2 Q i is L 2 -Lipschitz, and using Lemma C.3 for contracting the remaining terms.
Revised: which follows from the fact that ∇ 2 Q i is L 2 -Lipschitz, and using Lemma C.3 for contracting the remaining terms.

Original: Next, we turn to concentrating the second term in Equation C.24.
Revised: Next, we turn to concentrating the second term in Equation C.24.

Original: This term corresponds to the contribution of difference between the empirical covariance matrix ν k ν ⊤ k and the asymptotic covariance matrix ζ ∗ ζ ∗⊤ .
Revised: This term corresponds to the contribution of difference between the empirical covariance matrix ν k ν ⊤ k and the asymptotic covariance matrix ζ ∗ ζ ∗⊤ .

Original: We expand this term below using the form of G from Lemma C.1, and bound it one by one.
Revised: We expand this term below using the form of G from Lemma C.1, and bound it one by one.

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: Now, we note that the term ∆ k
Revised: Now [Equation] we note that the term ∆ k

Original: +1 :=
Revised: +1 :=

Original: ν k +1 ν
Revised: ν k +1 ν

Original: ⊤ k +1 − ζ ∗ ζ ∗⊤ can itself be written as a recursion:
Revised: ⊤ k +1 − ζ ∗ ζ ∗⊤ can itself be written as a recursion:

Original: [Equation]
Revised: [Equation]

Original: Expanding the term ∆ k +1 in terms of a summation over k , and plugging it into the expression from
Revised: Expanding the term ∆ k +1 in terms of a summation over k , and plugging it into the expression from

Original: Equation C.35 we get X
Revised: Equation C.35 we get X

Original: [Equation]
Revised: [Equation]

Original:  Now by noting that if G and ∇ Q i are ( ω, C 0 ) -aligned, then so are G ⊤ and ∇ Q i , we can ﬁnish the proof by repeating the calculations used by Damian et al.
Revised: + B p )( I − ηG ⊤ ) j − p − Now by noting that if G and ∇ Q i are ( ω, C 0 ) -aligned, then so are G ⊤ and ∇ Q i , we can ﬁnish the proof by repeating the calculations used by Damian et al.

Original: (2021) (Appendix B, Equations 67-73) to bound the terms in Equation C.35 by O ( √ ηk ) , but with an additional factor of ω 2 C 20 .
Revised: (2021) (Appendix B, Equations 67-73) to bound the terms in Equation C.35 by O ( √ ηk ) , but with an additional factor of ω 2 C 20 .

Original: Term (b) in Equation C.22.
Revised: Term (b) in Equation C.22.

Original: When C is small enough, we can bound the term (b) using O ( √ ηk ) , similar to Damian et al.
Revised: When C is small enough, we can bound the term (b) using O ( √ ηk ) , similar to Damian et al.

Original: D P ROOF OF P ROPOSITION 3. 
Revised: D P ROOF OF P ROPOSITION 3.

Original: In this section, we will prove Proposition 3.2.
Revised: In this section, we will prove Proposition 3.2.

Original: First, we refer to Proposition 3.1 in Ghosh &
Revised: First, we refer to Proposition 3.1 in Ghosh &

Original: Bellemare (2020), which shows that TD-learning is stable and converges if and only if the matrix
Revised: Bellemare (2020), which shows that TD-learning is stable and converges if and only if the matrix

Original: M ϕ =
Revised: M ϕ =

Original: Φ ⊤ (Φ − γ Φ ′ ) has eigenvalues with all positive real entries.
Revised: Φ ⊤ (Φ − γ Φ ′ ) has eigenvalues with all positive real entries.

Original: Now note that if, 
Revised: Now note that if, X

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the trace of a real matrix is the sum of real components of eigenvalues, if for a given matrix
Revised: Since the trace of a real matrix is the sum of real components of eigenvalues, if for a given matrix

Original: M , trace( M ) ≤ 0 , then there exists at least one eigenvalue
Revised: M , trace( M ) ≤ 0 , then there exists at least one eigenvalue

Original: λ
Revised: λ

Original: i such that Re( λ i ) ≤ 0 .
Revised: i such that Re( λ i ) ≤ 0 .

Original: If λ i < 0 , then the learning dynamics of TD would diverge, while if λ i = 

Revised: If λ i < 0 , then the learning dynamics of TD would diverge, while if λ i =

Original: for all i , then learning will not contract towards the TD ﬁxed point.
Revised: for all i , then learning will not contract towards the TD ﬁxed point.

Original: This concludes the proof of this result.
Revised: This concludes the proof of this result.

Original: E E XPERIMENTAL D ETAILS OF A PPLYING DR 
Revised: E E XPERIMENTAL D ETAILS OF A PPLYING DR

Original: In this section, we discuss the practical experimental details and hyperparameters in applying our method, DR3 to various ofﬂine RL methods.
Revised: In this section, we discuss the practical experimental details and hyperparameters in applying our method, DR3 to various ofﬂine RL methods.

Original: We ﬁrst discuss an overview of the ofﬂine RL methods we considered in this paper, and then provide a discussion of hyperparameters for DR3.
Revised: We ﬁrst discuss an overview of the ofﬂine RL methods we considered in this paper, and then provide a discussion of hyperparameters for DR3.

Original: E. B ACKGROUND ON V ARIOUS O FFLINE RL A LGORITHMS
Revised: E. B ACKGROUND ON V ARIOUS O FFLINE RL A LGORITHMS

Original: In this paper, we consider four base ofﬂine RL algorithms that we apply DR3 on.
Revised: In this paper, we consider four base ofﬂine RL algorithms that we apply DR3 on.

Original: These methods are detailed below:
Revised: These methods are detailed below:

Original: REM .
Revised: REM .

Original: Random ensemble mixture (Agarwal et al., 2020) is an uncertainty-based ofﬂine RL algorithm uses multiple parameterized Q-functions to estimate the Q-values.
Revised: Random ensemble mixture (Agarwal et al., 2020) is an uncertainty-based ofﬂine RL algorithm uses multiple parameterized Q-functions to estimate the Q-values.

Original: During the Bellman backup,
Revised: During the Bellman backup,

Original: REM computes a random convex combination of the target Q-values and then trains the Q-function to match this randomized target estimate.
Revised: REM computes a random convex combination of the target Q-values and then trains the Q-function to match this randomized target estimate.

Original: The randomized target value estimate provides a robust estimate of target values, and delays unlearning and performance degradation that we typically see with standard DQN-style algorithms in the ofﬂine setting.
Revised: The randomized target value estimate provides a robust estimate of target values, and delays unlearning and performance degradation that we typically see with standard DQN-style algorithms in the ofﬂine setting.

Original: For instantiating REM, we follow the instantiation provided by the authors and instantiate a multi-headed Q-function with 200 heads, each of which serves as an estimate of the target value.
Revised: For instantiating REM, we follow the instantiation provided by the authors and instantiate a multi-headed Q-function with 200 heads, each of which serves as an estimate of the target value.

Original: These multiple heads branch off the last-but-one layer features of the base Q-network.
Revised: These multiple heads branch off the last-but-one layer features of the base Q-network.

Original: The objective for REM is given by:
Revised: The objective for REM is given by:

Original: [Equation]
Revised: [Equation]

Original: where l λ denotes the Huber loss while P ∆ denotes the probability distribution over the standard (K 1)-simplex.
Revised: where l λ denotes the Huber loss while P ∆ denotes the probability distribution over the standard (K 1)-simplex.

Original: CQL .
Revised: CQL .

Original: Conservative Q-learning (Kumar et al., 2020b) is an ofﬂine RL algorithm that learns a conservative value function such that the estimated performance of the policy under this learned value function lower-bounds its true value.
Revised: Conservative Q-learning (Kumar et al., 2020b) is an ofﬂine RL algorithm that learns a conservative value function such that the estimated performance of the policy under this learned value function lower-bounds its true value.

Original: CQL modiﬁes the Q-function training to incorporate a term that minimizes the overestimated Q-values in expectation, while maximizing the Q-values observed in the dataset, in addition to standard TD error.
Revised: CQL modiﬁes the Q-function training to incorporate a term that minimizes the overestimated Q-values in expectation, while maximizing the Q-values observed in the dataset, in addition to standard TD error.

Original: This CQL regularizer is typically multiplied by a coefﬁcient α , and we pick α = 0 .
Revised: This CQL regularizer is typically multiplied by a coefﬁcient α , and we pick α = 0 .

Original: 1 for all our Atari experiments following Kumar et al.
Revised: 1 for all our Atari experiments following Kumar et al.

Original: (2021) and α = 5 . 0 for all our kitchen and antmaze D4RL experiments.
Revised: (2021) and α = 5 . 0 for all our kitchen and antmaze D4RL experiments.

Original: Using y k ( s , a ) to denote the target values computed via the Bellman backup (we use actor-critic backup for D4RL experiments and the max a ′ backup for standard Q-learning in our Atari experiments following Kumar et al.
Revised: Using y k ( s , a ) to denote the target values computed via the Bellman backup (we use actor-critic backup for D4RL experiments and the max a ′ backup for standard Q-learning in our Atari experiments following Kumar et al.

Original: (2020b))
Revised: (2020b))

Original: , the objective for training CQL is given by:
Revised: , the objective for training CQL is given by:

Original: [Equation]
Revised: [Equation]

Original: The deep Q-network utilized by us is a ReLU network with four hidden layers of size (256 , 256 , 256 , 256) for the D4RL experiments, while for Atari we utilize the standard convolutional neural network from Agarwal et al. ; Kumar et al.
Revised: The deep Q-network utilized by us is a ReLU network with four hidden layers of size (256 , 256 , 256 , 256) for the D4RL experiments, while for Atari we utilize the standard convolu- the then

Original: (2021) with 3 convolutional layers borrowed from the nature DQN network and then a hidden feedforward layer of size 512 .

Original: BRAC .
Revised: BRAC .

Original: Behavior-regularized actor-critic (Wu et al., 2019) is a policy-constraint based actor-critic ofﬂine RL algorithm which regularizes the policy to stay close to the behavior policy π β to prevent the selection of “out-of-distribution” actions.
Revised: Behavior-regularized actor-critic (Wu et al., 2019) is a policy-constraint based actor-critic ofﬂine RL algorithm which regularizes the policy to stay close to the behavior policy π β to prevent the selection of “out-of-distribution” actions.

Original: In addition, BRAC subtracts this divergence estimate from the target Q-values when performing the backup, to speciﬁcally penalize target values that come from out-of-distribution action inputs at the next state ( s ′ , a ′ ) .
Revised: In addition, BRAC subtracts this divergence estimate from the target Q-values when performing the backup, to speciﬁcally penalize target values that come from out-of-distribution action inputs at the next state ( s ′ , a ′ ) .

Original: Q-function:
Revised: Q-function:

Original: [Equation]

Original: Policy

Original: [Equation]
Revised: [Equation]

Original: COG .
Revised: COG .

Original: COG (Singh et al., 2020) is an algorithmic framework for utilizing large, unlabeled datasets of diverse behavior to learn generalizable policies via ofﬂine RL.
Revised: COG (Singh et al., 2020) is an algorithmic framework for utilizing large, unlabeled datasets of diverse behavior to learn generalizable policies via ofﬂine RL.

Original: Similar to real-world scenarios where large unlabeled datasets are available alongside limited task-speciﬁc data, the agent is provided with two types of datasets.
Revised: Similar to real-world scenarios where large unlabeled datasets are available alongside limited task-speciﬁc data, the agent is provided with two types of datasets.

Original: The task-speciﬁc dataset consists of behavior relevant for the task, but the prior dataset can consist of a number of random or scripted behaviors being executed in the same environment/setting.
Revised: The task-speciﬁc dataset consists of behavior relevant for the task, but the prior dataset can consist of a number of random or scripted behaviors being executed in the same environment/setting.

Original: The goal in this task is to actually stitch together relevant and overlapping parts of different trajectories to obtain a good policy that can work from a new initial condition that was not seen in a trajectory that actually achieved the reward.
Revised: The goal in this task is to actually stitch together relevant and overlapping parts of different trajectories to obtain a good policy that can work from a new initial condition that was not seen in a trajectory that actually achieved the reward.

Original: COG utilizes CQL as the base ofﬂine RL algorithm, and following Singh et al.
Revised: COG utilizes CQL as the base ofﬂine RL algorithm, and following Singh et al.

Original: (2020), we ﬁx the hyperparameter α = 1 . 0 in the CQL part for both base COG and COG + DR3.
Revised: (2020), we ﬁx the hyperparameter α = 1 . 0 in the CQL part for both base COG and COG + DR3.

Original: All other hyperparameters including network sizes, etc are kept ﬁxed as the prior work Singh et al.
Revised: All other hyperparameters including network sizes, etc are kept ﬁxed as the prior work Singh et al.

Original: (2020) as well.
Revised: (2020) as well.

Original: E. T ASKS AND E NVIRONMENTS U SED
Revised: E. T ASKS AND E NVIRONMENTS U SED

Original: Atari 2600 games used .
Revised: Atari 2600 games used .

Original: For all our experiments, we used the same set of 17 games utilized by
Revised: For all our experiments, we used the same set of 17 games utilized by

Original: Kumar et al.
Revised: Kumar et al.

Original: (2021) to test rank collapse.
Revised: (2021) to test rank collapse.

Original: In the case of Atari, we used the 5 standard games (A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT ) for tuning the hyperparameters, a strategy followed by several prior works (Gulcehre et al., 2020; Agarwal et al., 2020; Kumar et al., 2021).
Revised: In the case of Atari, we used the 5 standard games (A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT ) for tuning the hyperparameters, a strategy followed by several prior works (Gulcehre et al., 2020; Agarwal et al., 2020; Kumar et al., 2021).

Original: The 17 games we test on are: A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT , D OUBLE D UNK ,
Revised: The 17 games we test on are: A STERIX , Q BERT , P ONG , S EAQUEST , B REAKOUT , D OUBLE D UNK ,

Original: J AMES B OND , M S .
Revised: J AMES B OND , M S .

Original: P ACMAN , S PACE I NVADERS , Z AXXON , W IZARD OF W OR , Y ARS ’ R EVENGE ,
Revised: P ACMAN , S PACE I NVADERS , Z AXXON , W IZARD OF W OR , Y ARS ’ R EVENGE ,

Original: E NDURO , R OAD R UNNER , B EAM R IDER , D EMON A TTACK , I CE H OCKEY .
Revised: E NDURO , R OAD R UNNER , B EAM R IDER , D EMON A TTACK , I CE H OCKEY .

Original: Following Agarwal et al.
Revised: Following Agarwal et al.

Original: (2021), we report interquartile mean (IQM) normalized scores across all runs as mean scores can be dominated by performance on a few outlier tasks while median is independent of performance on all except 1 task – zero score on half of the tasks would not affect the median.
Revised: (2021), we report interquartile mean (IQM) normalized scores across all runs as mean scores can be dominated by performance on a few outlier tasks while median is independent of performance on all except 1 task – zero score on half of the tasks would not affect the median.

Original: IQM which corresponds to 25% trimmed mean and considers the performance on middle 50% of the runs.
Revised: IQM which corresponds to 25% trimmed mean and considers the performance on middle 50% of the runs.

Original: IQM interpolates between mean and median, which correspond to 0% and almost 50% trimmed means across runs.
Revised: IQM interpolates between mean and median, which correspond to 0% and almost 50% trimmed means across runs.

Original: D4RL tasks used.
Revised: D4RL tasks used.

Original: For our experiments on D4RL, we utilize the Gym-MuJoCo-v0 environments for evaluating BRAC, since BRAC performed somewhat reasonably on these domains (Fu et al., 2020), whereas we use the harder AntMaze and Franka Kitchen domains for evaluating CQL, since these domains are challenging for CQL (Kumar et al., 2020b).
Revised: For our experiments on D4RL, we utilize the Gym-MuJoCo-v0 environments for evaluating BRAC, since BRAC performed somewhat reasonably on these domains (Fu et al., 2020), whereas we use the harder AntMaze and Franka Kitchen domains for evaluating CQL, since these domains are challenging for CQL (Kumar et al., 2020b).

Original: Robotic manipulation tasks from COG (Singh et al., 2020).
Revised: Robotic manipulation tasks from COG (Singh et al., 2020).

Original: These tasks consist of a 6-DoF
Revised: These tasks consist of a 6-DoF

Original: WidowX robot, placed in front of two drawers and a larger variety of objects.
Revised: WidowX robot, placed in front of two drawers and a larger variety of objects.

Original: The robot can open or close a drawer, grasp objects from inside the drawer or on the table, and place them anywhere in the scene.
Revised: The robot can open or close a drawer, grasp objects from inside the drawer or on the table, and place them anywhere in the scene.

Original: The task here consists of taking an object out of a drawer.
Revised: The task here consists of taking an object out of a drawer.

Original: A reward of +1 is obtained when the object has been taken out, and zero otherwise.
Revised: A reward of +1 is obtained when the object has been taken out, and zero otherwise.

Original: There are two variants of this domain: (1) in the ﬁrst variant, the drawer starts out closed, the top drawer starts out open (which blocks the handle for the lower drawer), and an object starts out in front of the closed drawer, which must be moved out of the way before opening, and (2) in the second variant, the drawer is blocked by an object, and this object must be removed before the drawer can be opened and the target object can be grasped from the drawer.
Revised: There are two variants of this domain: (1) in the ﬁrst variant, the drawer starts out closed, the top drawer starts out open (which blocks the handle for the lower drawer), and an object starts out in front of the closed drawer, which must be moved out of the way before opening, and (2) in the second variant, the drawer is blocked by an object, and this object must be removed before the drawer can be opened and the target object can be grasped from the drawer.

Original: The prior data for this environment is collected from a collection of scripted randomized policies.
Revised: The prior data for this environment is collected from a collection of scripted randomized policies.

Original: These policies are capable of opening and closing both drawers with 40-50% success rates, can grasp objects in the scene with about a 70% success rate, and place those objects at random places in the scene (with a slight bias for putting them in the tray). 
Revised: These policies are capable of opening and closing both drawers with 40-50% success rates, can grasp objects in the scene with about a 70% success rate, and place those objects at random places in the scene (with a slight bias for putting them in the tray). [Table] [Table]

Original: E. T HE DR3 R EGULARIZER C OEFFICIENT
Revised: E. T HE DR3 R EGULARIZER C OEFFICIENT

Original: We utilize identical hyperparameters of the base ofﬂine RL algorithms when DR3 is used, where the base hyper-parameters correspond to the ones provided in the corresponding publications.
Revised: We utilize identical hyperparameters of the base ofﬂine RL algorithms when DR3 is used, where the base hyper-parameters correspond to the ones provided in the corresponding publications.

Original: DR requires us to tune the additional coefﬁcient c 0 , that weights the DR3 explicit regularizer term.
Revised: DR requires us to tune the additional coefﬁcient c 0 , that weights the DR3 explicit regularizer term.

Original: In order to ﬁnd this value on our domains, we followed the tuning strategy typically followed on Atari, where we evaluated four different values of c 0 ∈ { 0 . 001 , 0 . 01 , 0 .
Revised: In order to ﬁnd this value on our domains, we followed the tuning strategy typically followed on Atari, where we evaluated four different values of c 0 ∈ { 0 . 001 , 0 . 01 , 0 .

Original: 03 , 0 . 3 } on 5 games (A STERIX ,
Revised: 03 , 0 . 3 } on 5 games (A STERIX ,

Original: S EAQUEST , B REAKOUT , P ONG and S PACE I NVADERS ) on the 5% replay dataset settings, picked cthat worked best on just these domains, and used it to report performance on all 17 games, across all dataset settings (1% replay and 10% initial replay) in Section 6.
Revised: S EAQUEST , B REAKOUT , P ONG and S PACE I NVADERS ) on the 5% replay dataset settings, picked cthat worked best on just these domains [Table] and used it to report performance on all 17 games, across all dataset settings (1% replay and 10% initial replay) in Section 6.

Original: This protocol is standard in Atari and has been used previously in Mnih et al.
Revised: This protocol is standard in Atari and has been used previously in Mnih et al.

Original: ; Agarwal et al.
Revised: ; Agarwal et al.

Original: ; Gulcehre et al.
Revised: ; Gulcehre et al.

Original: Kumar et al.
Revised: Kumar et al.

Original: (2021) in the context of ofﬂine RL.
Revised: (2021) in the context of ofﬂine RL.

Original: The value of the coefﬁcient found using this strategy was c 0 = 0 .
Revised: The value of the coefﬁcient found using this strategy was c 0 = 0 .

Original: 001 for REM and c 0 = 0 .
Revised: 001 for REM and c 0 = 0 .

Original: 03 for CQL.
Revised: 03 for CQL.

Original: [Table]

Original: For CQL on D4RL [Table] we ran DR3 with multiple values of c 0 ∈ { 0 . 0001 , 0 .
Revised: For CQL on D4RL, we ran DR3 with multiple values of c 0 ∈ { 0 . 0001 , 0 .

Original: [Table] and picked the smallest value of c 0 which did not lead to eventually divergent (either negatively diverging or positively diverging) Q-values, in average.
Revised: 001 , 0 . 01 , 0 . 5 , 1 . 0 , 10 . 0 } , and picked the smallest value of c 0 which did not lead to eventually divergent (either negatively diverging or positively diverging) Q-values, in average.

Original: For the antmaze domains, this corresponded to c 0 = 0 .
Revised: For the antmaze domains, this corresponded to c 0 = 0 .

Original: 001 and for the FrankaKitchen domains, this corresponded to c 0 = 1 . 0 .
Revised: 001 and for the FrankaKitchen domains, this corresponded to c 0 = 1 . 0 .

Original:  E 
Revised: E. C OMPUTE R EQUIREMENTS FOR O UR E XPERIMENTS

Original: C OMPUTE R EQUIREMENTS FOR O UR E XPERIMENTS

Original: Running 100 iterations (625k grad updates) takes about 2-2.5 days on a dedicated P100 GPU.
Revised: Running 100 iterations (625k grad updates) takes about 2-2.5 days on a dedicated P100 GPU.

Original: So, our results with one seed for one game take 4-5 days for 200 iterations on the Atari dataset.
Revised: So, our results with one seed for one game take 4-5 days for 200 iterations on the Atari dataset.

Original: We also note adding the practical DR3 regularizer doesn not have any signiﬁcant impact on compute time and takes almost the same amount of time as the base ofﬂine algorithm.
Revised: We also note adding the practical DR3 regularizer doesn not have any signiﬁcant impact on compute time and takes almost the same amount of time as the base ofﬂine algorithm.

Original: This is because the
Revised: This is because the

Original: DR3 regularizer can be computed by additionally logging last-but-one layer features when doing a forward pass through the Q-function.
Revised: DR3 regularizer can be computed by additionally logging last-but-one layer features when doing a forward pass through the Q-function.

Original: Since we ran 2 methods, with and without DR3, on 17 games with 5 seeds on 3 dataset settings, this amounts to 4080-5100 GPU days for reporting evaluation results on Atari.
Revised: Since we ran 2 methods, with and without DR3, on 17 games with 5 seeds on 3 dataset settings, this amounts to 4080-5100 GPU days for reporting evaluation results on Atari.

Original: Running 1M steps for one run of the MuJoCo tasks and the D4RL tasks takes about 16 hours on our implementation with a GPU.
Revised: Running 1M steps for one run of the MuJoCo tasks and the D4RL tasks takes about 16 hours on our implementation with a GPU.

Original: Note that this is partly because of using a not so optimized implementation, implementations on MuJoCo tasks can be made much faster otherwise.
Revised: Note that this is partly because of using a not so optimized implementation, implementations on MuJoCo tasks can be made much faster otherwise.

Original: F C OMPLETE R ESULTS ON A LL D OMAINS
Revised: F C OMPLETE R ESULTS ON A LL D OMAINS

Original: In this section, we present the results obtained by running DR3 on the Atari and D4RL domains which were not discussed in the main paper due to lack of space.
Revised: In this section, we present the results obtained by running DR3 on the Atari and D4RL domains which were not discussed in the main paper due to lack of space.

Original: We ﬁrst understand the effect of applying DR3 on BRAC (Wu et al., 2019), which was missing from the main paper, and then present the per-game Atari results. 
Revised: We ﬁrst understand the effect of applying DR3 on BRAC (Wu et al., 2019), which was missing from the main paper, and then present the per-game Atari results. In section F. [Table]

Original: In section F.1, we present the learning curves of CQL and REM with and without DR3 on the 5% uniform replay dataset per game for 200 × 62.5k gradient steps in Figures F.1 and Figure F. respectively. 
Revised: we present the learning curves of CQL and REM with and without DR3 on the 5% uniform replay dataset per game for 200 × 62.5k gradient steps in Figures F.1 and Figure F. respectively. [Table] [Table] [Table]

Revised: walker2d-expert-

Original: [Table]
Revised: [Table]

Revised: walker2d-medium-

Original: [Table]
Revised: [Table]

Revised: walker2d-medium-expert-

Original: [Table]
Revised: [Table]

Revised: walker2d-random-v

Original: [Table]
Revised: [Table]

Revised: walker2d-medium-replay-v

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: odu
Revised: odu

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]