Original: The Pitfalls of Regularization in Off-Policy TD Learning
Revised: The Pitfalls of Regularization in Off-Policy TD Learning

Original: Abstract
Revised: Abstract

Original: Temporal Difference (TD) learning is ubiquitous in reinforcement learning, where it is often combined with off-policy sampling and function approximation.
Revised: Temporal Difference (TD) learning is ubiquitous in reinforcement learning, where it is often combined with off-policy sampling and function approximation.

Original: Unfortunately learning with this combination (known as the deadly triad ), exhibits instability and unbounded error.
Revised: Unfortunately learning with this combination (known as the deadly triad ), exhibits instability and unbounded error.

Original: To account for this, modern RL methods often implicitly (or sometimes explicitly) assume that regularization is sufficient to mitigate the problem in practice; indeed, the standard deadly triad examples from the literature can be “fixed” via proper regularization.
Revised: To account for this, modern RL methods often implicitly (or sometimes explicitly) assume that regularization is sufficient to mitigate the problem in practice; indeed, the standard deadly triad examples from the literature can be “fixed” via proper regularization.

Original: In this paper, we introduce a series of new counterexamples to show that the instability and unbounded error of TD methods is not solved by regularization.
Revised: In this paper, we introduce a series of new counterexamples to show that the instability and unbounded error of TD methods is not solved by regularization.

Original: We demonstrate that, in the offpolicy setting with linear function approximation, TD methods can fail to learn a non-trivial value function under any amount of regularization; we further show that regularization can induce divergence under common conditions; and we show that one of the most promising methods to mitigate this divergence (Emphatic
Revised: We demonstrate that, in the offpolicy setting with linear function approximation, TD methods can fail to learn a non-trivial value function under any amount of regularization; we further show that regularization can induce divergence under common conditions; and we show that one of the most promising methods to mitigate this divergence (Emphatic

Original: TD algorithms) may also diverge under regularization.
Revised: TD algorithms) may also diverge under regularization.

Original: We further demonstrate such divergence when using neural networks as function approximators.
Revised: We further demonstrate such divergence when using neural networks as function approximators.

Original: Thus, we argue that the role of regularization in TD methods needs to be reconsidered, given that it is insufficient to prevent divergence and may itself introduce instability.
Revised: Thus, we argue that the role of regularization in TD methods needs to be reconsidered, given that it is insufficient to prevent divergence and may itself introduce instability.

Original: There needs to be much more care in the practical and theoretical application of regularization to RL methods.
Revised: There needs to be much more care in the practical and theoretical application of regularization to RL methods.

Original:  Introduction
Revised:  Introduction

Original: Temporal Difference (TD) learning is a method for learning expected future-discounted quantities from Markov processes, using transition samples to iteratively improve estimates.
Revised: Temporal Difference (TD) learning is a method for learning expected future-discounted quantities from Markov processes, using transition samples to iteratively improve estimates.

Original: This is most commonly used to estimate expected future-discounted rewards (the value function ) in Reinforcement
Revised: This is most commonly used to estimate expected future-discounted rewards (the value function ) in Reinforcement

Original: Learning (RL).
Revised: Learning (RL).

Original: Advances in RL allow us to use powerful function approximators, and also to use sampling strategies other than naively following the Markov process (MP).
Revised: Advances in RL allow us to use powerful function approximators, and also to use sampling strategies other than naively following the Markov process (MP).

Original: When TD, function approximation, and off-policy training are all combined, learned functions exhibit severe instability and divergence, as classically observed by Williams and Baird III [18], Tsitsiklis and Van Roy [15].
Revised: When TD, function approximation, and off-policy training are all combined, learned functions exhibit severe instability and divergence, as classically observed by Williams and Baird III [18], Tsitsiklis and Van Roy [15].

Original: This combination is known in the literature as the deadly triad [11, pg. 264], and while many contemporary variants of TD are designed to converge despite the instability, the quality of the solution at convergence may be arbitrarily poor.
Revised: This combination is known in the literature as the deadly triad [11, pg. 264], and while many contemporary variants of TD are designed to converge despite the instability, the quality of the solution at convergence may be arbitrarily poor.

Original: A common technique to avoid unbounded error is ℓ 2 regularization [14], i.e. penalizing the squared norm of the weights in addition to the TD error.
Revised: A common technique to avoid unbounded error is ℓ 2 regularization [14], i.e. penalizing the squared norm of the weights in addition to the TD error.

Original: This is generally understood to bound the worst-case error in exchange for biasing the model and potentially increasing the error everywhere else.
Revised: This is generally understood to bound the worst-case error in exchange for biasing the model and potentially increasing the error everywhere else.

Original: When used on three common examples of the deadly triad [6, 18, 11, pg.260], regularization appears to mitigate the worst aspects of the divergence in practice.
Revised: When used on three common examples of the deadly triad [6, 18, 11, pg.260], regularization appears to mitigate the worst aspects of the divergence in practice.

Original: Consequently, it has become an essential assumption made by many RL algorithms [1, 10, 12, 19, 20, 21, 8] and is seen as routine and innocuous.
Revised: Consequently, it has become an essential assumption made by many RL algorithms [1, 10, 12, 19, 20, 21, 8] and is seen as routine and innocuous.

Original: We argue that this perspective on regularization in off-policy TD is fundamentally mistaken.
Revised: We argue that this perspective on regularization in off-policy TD is fundamentally mistaken.

Original: While regularization is indeed reasonably well-behaved and innocuous in classic fully-supervised contexts, the use of bootstrapping in TD means that even small amounts of model bias induced by regularization can cause divergence.
Revised: While regularization is indeed reasonably well-behaved and innocuous in classic fully-supervised contexts, the use of bootstrapping in TD means that even small amounts of model bias induced by regularization can cause divergence.

Original: This is an oft-ignored phenomenon in the literature, and so we introduce a series of new counterexamples (summarized in Table 1) to show how regularization can have counterintuitive and destructive effects in TD.
Revised: This is an oft-ignored phenomenon in the literature, and so we introduce a series of new counterexamples (summarized in Table 1) to show how regularization can have counterintuitive and destructive effects in TD.

Original: We show that vacuous solutions and training instability are not solved by the use of regularization; that applying regularization can sometimes induce divergence and increase worst-case error; and that Emphatic TD algorithms–which are the most promising solution to this divergence–can themselves diverge when regularized.
Revised: We show that vacuous solutions and training instability are not solved by the use of regularization; that applying regularization can sometimes induce divergence and increase worst-case error; and that Emphatic TD algorithms–which are the most promising solution to this divergence–can themselves diverge when regularized.

Original: We finally also illustrate misbehaving regularization in the context of neural network value function approximation, demonstrating the general pitfalls of regularization possible in RL algorithms.
Revised: We finally also illustrate misbehaving regularization in the context of neural network value function approximation, demonstrating the general pitfalls of regularization possible in RL algorithms.

Original: Regularization needs to be treated cautiously in the context of RL, as it behaves differently than in supervised settings.
Revised: Regularization needs to be treated cautiously in the context of RL, as it behaves differently than in supervised settings.

Original: Our counterexamples demonstrate these core ideas:
Revised: Our counterexamples demonstrate these core ideas:

Original: TD learning off-policy can be unstable and/or have unbounded error even when it converges.
Revised: TD learning off-policy can be unstable and/or have unbounded error even when it converges.

Original: Following well-established methods we show there is some off-policy distribution under which TD with linear value function approximation diverges and learns a model with unbounded error (even if it were able to converge to the TD fixed point).
Revised: Following well-established methods we show there is some off-policy distribution under which TD with linear value function approximation diverges and learns a model with unbounded error (even if it were able to converge to the TD fixed point).

Original: This concisely demonstrates key features of the training error: the error is small when the distribution is close to on-policy, but the error diverges around specific off-policy distributions.
Revised: This concisely demonstrates key features of the training error: the error is small when the distribution is close to on-policy, but the error diverges around specific off-policy distributions.

Original: The intuition behind this, explained in Section 3, is that the off-policy 1 TD update involves a projection operation that depends on the sampling distribution and can be arbitrarily far away from the true value.
Revised: The intuition behind this, explained in Section 3, is that the off-policy 1 TD update involves a projection operation that depends on the sampling distribution and can be arbitrarily far away from the true value.

Original: This basic fact has already been established by past work [18, 6], but our example is based upon a particular simple three-state MP, drawn in Figure 1a.
Revised: This basic fact has already been established by past work [18, 6], but our example is based upon a particular simple three-state MP, drawn in Figure 1a.

Original: Regularization cannot always mitigate off-policy training error.
Revised: Regularization cannot always mitigate off-policy training error.

Original: We next introduce regularization into our setting, and show how it changes the relationship between training error and off-policy training.
Revised: We next introduce regularization into our setting, and show how it changes the relationship between training error and off-policy training.

Original: As explained in Section 2, we penalize the ℓ 2 -norm of learned (linear) weights with some coefficient η ; as η increases, the learned weights approach zero.
Revised: As explained in Section 2, we penalize the ℓ 2 -norm of learned (linear) weights with some coefficient η ; as η increases, the learned weights approach zero.

Original: However, in Example 1 , we show that there exists an off-policy distribution such that for any η ≥ 0 < ∞ , the regularized TD fixed point attains strictly higher approximation error than the zero solution (i.e., the infinitely regularized point).
Revised: However, in Example 1 , we show that there exists an off-policy distribution such that for any η ≥ 0 < ∞ , the regularized TD fixed point attains strictly higher approximation error than the zero solution (i.e., the infinitely regularized point).

Original: We call such examples vacuous .
Revised: We call such examples vacuous .

Original: In other words, vacuous value functions never do better than guessing zero for all states, for any amount of regularization .
Revised: In other words, vacuous value functions never do better than guessing zero for all states, for any amount of regularization .

Original: We further analyze this vacuous example in the context of the algorithm in [21].
Revised: We further analyze this vacuous example in the context of the algorithm in [21].

Original: In this work, the authors assume the use of regularization to derive bounds on the learned error under off-policy sampling.
Revised: In this work, the authors assume the use of regularization to derive bounds on the learned error under off-policy sampling.

Original: Although these bounds are technically correct in the case of our counterexample, they are very loose, at about 2000 times the threshold of vacuity.
Revised: Although these bounds are technically correct in the case of our counterexample, they are very loose, at about 2000 times the threshold of vacuity.

Original: This highlights the challenge of formally relying on regularization to bound model error.
Revised: This highlights the challenge of formally relying on regularization to bound model error.

Original: Small amounts of regularization can cause model divergence or large errors.
Revised: Small amounts of regularization can cause model divergence or large errors.

Original: There is a general implicit assumption in much ML literature that regularlization
Revised: There is a general implicit assumption in much ML literature that regularlization

Original: monotonically
Revised: monotonically

Original: shrinks learned weights.
Revised: shrinks learned weights.

Original: This intuition comes from classic fully-supervised machine learning where it typically holds.
Revised: This intuition comes from classic fully-supervised machine learning where it typically holds.

Original: But because TD bootstraps value estimates (i.e. learns values using its own output), it is possible for small amounts of bias to be arbitrarily magnified.
Revised: But because TD bootstraps value estimates (i.e. learns values using its own output), it is possible for small amounts of bias to be arbitrarily magnified.

Original: We dub this phenomenon “small-eta error” and illustrate it in Example 2 .
Revised: We dub this phenomenon “small-eta error” and illustrate it in Example 2 .

Original: We relate this to the presence of negative eigenvalues in an intermediate step of the solution and show that, in some settings, the error of the TD solution may be relatively small when applied with no regularization but adding regularization causes the model to have worse error than the zero solution.
Revised: We relate this to the presence of negative eigenvalues in an intermediate step of the solution and show that, in some settings, the error of the TD solution may be relatively small when applied with no regularization but adding regularization causes the model to have worse error than the zero solution.

Original: One common solution to this problem is to lower-bound η to guarantee that reguarlization behaves monotonically.
Revised: One common solution to this problem is to lower-bound η to guarantee that reguarlization behaves monotonically.

Original: However, we further show that such a lower bound may occur after the point of vacuity: a model that is not vacuous becomes vacuous for any regularization parameter above this lower bound.
Revised: However, we further show that such a lower bound may occur after the point of vacuity: a model that is not vacuous becomes vacuous for any regularization parameter above this lower bound.

Original: We also show that it is not always possible to select a single η a priori , with examples of
Revised: We also show that it is not always possible to select a single η a priori , with examples of

Original: Example
Revised: Example

Original: There exist off-policy distributions under which TD learns a vacuous model (one which– despite any amount of regularization–never does better than guessing zeros).
Revised: There exist off-policy distributions under which TD learns a vacuous model (one which– despite any amount of regularization–never does better than guessing zeros).

Original: Example
Revised: Example

Original: Small values of the regularization parameter η can make TD diverge in models that otherwise converge.
Revised: Small values of the regularization parameter η can make TD diverge in models that otherwise converge.

Original: This is an unavoidable effect of bootstrapping in TD, and setting a lower-bound to exclude this may render models vacuous.
Revised: This is an unavoidable effect of bootstrapping in TD, and setting a lower-bound to exclude this may render models vacuous.

Original: Example
Revised: Example

Original: Emphatic-TD-inspired algorithms are a promising way to reweight samples and mitigate the effects of training off-policy.
Revised: Emphatic-TD-inspired algorithms are a promising way to reweight samples and mitigate the effects of training off-policy.

Original: But if this reweighting is learned using TD, then using regularization can bias the emphasis model and cause the value model itself to diverge.
Revised: But if this reweighting is learned using TD, then using regularization can bias the emphasis model and cause the value model itself to diverge.

Original: Example
Revised: Example

Original: Training instability and increased error due to the deadly triad also occur when neural networks are used.
Revised: Training instability and increased error due to the deadly triad also occur when neural networks are used.

Original: We construct an empirical example and draw qualitative comparisons.
Revised: We construct an empirical example and draw qualitative comparisons.

Original: [Figure]
Revised: [Figure]

Original: -incompatible off-policy distributions
Revised: -incompatible off-policy distributions

Original: Emphatic-TD-based algorithms are vulnerable to instability from regularization.
Revised: Emphatic-TD-based algorithms are vulnerable to instability from regularization.

Original: EmphaticTD
Revised: EmphaticTD

Original: [13] fundamentally solves the problem of training off-policy by resampling TD updates so they appear to be on-policy.
Revised: [13] fundamentally solves the problem of training off-policy by resampling TD updates so they appear to be on-policy.

Original: This technique requires an emphasis model that decides how to scale each TD update, and learning this has been the key challenge preventing widespread adoption of Emphatic-TD.
Revised: This technique requires an emphasis model that decides how to scale each TD update, and learning this has been the key challenge preventing widespread adoption of Emphatic-TD.

Original: A recent paper [20] proposed learning this emphasis model using “reversed” TD while simultaneously learning the value model using regular TD.
Revised: A recent paper [20] proposed learning this emphasis model using “reversed” TD while simultaneously learning the value model using regular TD.

Original: The resultant algorithm is called COF-PAC, and employs regularization to ensure that the two TD models eventually converge.
Revised: The resultant algorithm is called COF-PAC, and employs regularization to ensure that the two TD models eventually converge.

Original: We show that regularization, while necessary, can be harmful for such models in Example 3 .
Revised: We show that regularization, while necessary, can be harmful for such models in Example 3 .

Original: Specifically, we construct a model that converges to the correct solution without regularization but to an arbitrarily poor solution when regularized.
Revised: Specifically, we construct a model that converges to the correct solution without regularization but to an arbitrarily poor solution when regularized.

Original: The intuition behind this is that regularizing the emphasis model changes the effective distribution of the TD updates to the value model, which can cause the value model to have arbitrarily large error.
Revised: The intuition behind this is that regularizing the emphasis model changes the effective distribution of the TD updates to the value model, which can cause the value model to have arbitrarily large error.

Original: We complete the example by showing that regularizing the value function separately does not restore performance.
Revised: We complete the example by showing that regularizing the value function separately does not restore performance.

Original: Regularization can cause model divergence in neural networks.
Revised: Regularization can cause model divergence in neural networks.

Original: So far most analysis of the deadly triad in the literature focuses on the linear case.
Revised: So far most analysis of the deadly triad in the literature focuses on the linear case.

Original: We extend our example to a nine-state Markov chain (shown in Figure 1b), and show how the previously identified problems persist into the neural network case in Example 4 .
Revised: We extend our example to a nine-state Markov chain (shown in Figure 1b), and show how the previously identified problems persist into the neural network case in Example 4 .

Original: We show two key similarities: first, models trained at certain off-policy distributions may be vacuous.
Revised: We show two key similarities: first, models trained at certain off-policy distributions may be vacuous.

Original: Second, small amounts of regularization counterintuitively increase error.
Revised: Second, small amounts of regularization counterintuitively increase error.

Original: This illustrates Example 2 in the NN case.
Revised: This illustrates Example 2 in the NN case.

Original:  Preliminaries and Notation
Revised:  Preliminaries and Notation

Original: Consider the n -state Markov chain ( S , P, R, γ ) , with state space S , state-dependent reward R :
Revised: Consider the n -state Markov chain ( S , P, R, γ ) , with state space S , state-dependent reward R :

Original: S → R , and discount factor γ ∈ [0 , 1] .
Revised: S → R , and discount factor γ ∈ [0 , 1] .

Original: P ∈ R n × n is the transition matrix, with P ij encoding the probability of moving from state i to j .
Revised: P ∈ R n × n is the transition matrix, with P ij encoding the probability of moving from state i to j .

Original: We wish to estimate the value function V : S → R , defined as the expected discounted future reward of being in each state: V ( s ) ˙= E
Revised: We wish to estimate the value function V : S → R , defined as the expected discounted future reward of being in each state: V ( s ) ˙= E

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t R ( s t ) | s 0 = s ] .
Revised: γ t R ( s t ) | s 0 = s ] .

Original: A key property is that it follows the Bellman equation:
Revised: A key property is that it follows the Bellman equation:

Original: [Equation]
Revised: [Equation]

Original: Using linear function approximation to learn V , we assume a matrix of feature-vectors Φ ∈ R n × k that is fixed, and a vector of parameters w ∈ R
Revised: Using linear function approximation to learn V , we assume a matrix of feature-vectors Φ ∈ R n × k that is fixed, and a vector of parameters w ∈ R

Original: k
Revised: k

Original: that is learned.
Revised: that is learned.

Original: The Bellman equation is then:
Revised: The Bellman equation is then:

Original: [Equation]
Revised: [Equation]

Original: When w is learned with TD, this equation is only valid if the TD updates are on-policy (that is, they are distributed according to the steady-state probability of visiting each state, written as π ∈ R n ).
Revised: When w is learned with TD, this equation is only valid if the TD updates are on-policy (that is, they are distributed according to the steady-state probability of visiting each state, written as π ∈ R n ).

Original: In the general case, where TD updates follow a (possibly) different distribution µ ∈ R n 0 , the TD solution is a fixed point of the Bellman operator followed by a projection [6]:
Revised: In the general case, where TD updates follow a (possibly) different distribution µ ∈ R n 0 , the TD solution is a fixed point of the Bellman operator followed by a projection [6]:

Original: [Equation]
Revised: [Equation]

Original: where the matrix Π µ = Φ(Φ ⊤ D Φ) − 1 Φ ⊤ D projects the Bellman backup onto the columnspace of Φ , reweighted by the diagonal matrix D = diag ( µ ) .
Revised: where the matrix Π µ = Φ(Φ ⊤ D Φ) − 1 Φ ⊤ D projects the Bellman backup onto the columnspace of Φ , reweighted by the diagonal matrix D = diag ( µ ) .

Original: This yields the closed-form solution:
Revised: This yields the closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: Where A = Φ ⊤ D ( I − γP )
Revised: Where A = Φ ⊤ D ( I − γP )

Original: Φ and ⃗b =
Revised: Φ and ⃗b =

Original: Φ ⊤ DR .
Revised: Φ ⊤ DR .

Original: When this solution is subject to ℓ 2 regularization, some non-negative η is added to ensure the matrix being inverted is positive definite:
Revised: When this solution is subject to ℓ 2 regularization, some non-negative η is added to ensure the matrix being inverted is positive definite:

Original: [Equation]
Revised: [Equation]

Original: As will be important later, we note that as η increases it drives w ∗ ( η ) towards zero.
Revised: As will be important later, we note that as η increases it drives w ∗ ( η ) towards zero.

Original:  Our Counterexamples
Revised:  Our Counterexamples

Original: When deadly triad conditions are present, TD may learn a value function with arbitrarily large error even if the true value function can be represented with low error.
Revised: When deadly triad conditions are present, TD may learn a value function with arbitrarily large error even if the true value function can be represented with low error.

Original: Consider the three-state MP in
Revised: Consider the three-state MP in

Original: Figure 1a, which we instantiate with the value function V =
Revised: Figure 1a, which we instantiate with the value function V =

Original: [1 , 2 . 2 , 1 . 05] ⊤ and discount factor γ = 0 .
Revised: [1 , 2 . 2 , 1 . 05] ⊤ and discount factor γ = 0 .

Original: The reward function is computed as R ← ( I − γP ) V .
Revised: The reward function is computed as R ← ( I − γP ) V .

Original: We choose a basis Φ with small representation error ∥
Revised: We choose a basis Φ with small representation error ∥

Original: Π µ V − V ∥ ≤
Revised: Π µ V − V ∥ ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ϵ >(6)
Revised: where ϵ >(6)

Original: We first consider the unregularized ( η = 0 ) case, closely following the derivation in [6].
Revised: We first consider the unregularized ( η = 0 ) case, closely following the derivation in [6].

Original: We wish to show there is some sampling distribution µ such that error in the learned value function is unbounded.
Revised: We wish to show there is some sampling distribution µ such that error in the learned value function is unbounded.

Original: To do this, we set µ = [0 .
Revised: To do this, we set µ = [0 .

Original: 56(1 − p ) , 0 .
Revised: 56(1 − p ) , 0 .

Original: 56 p, 0 . 44] , where p ∈ (0 , 1) .
Revised: 56 p, 0 . 44] , where p ∈ (0 , 1) .

Original: We set ϵ = 10 − 4 and find p around which A is ill-conditioned by solving det( A ) = 0 :
Revised: We set ϵ = 10 − 4 and find p around which A is ill-conditioned by solving det( A ) = 0 :

Original: [Equation]
Revised: [Equation]

Original: A − 1 (and consequently the error) can be made arbitrarily large by selecting p close to these values, which completes the introductory example.
Revised: A − 1 (and consequently the error) can be made arbitrarily large by selecting p close to these values, which completes the introductory example.

Original: Now we look at the behavior of TD under regularization, which is the main contribution of our paper.
Revised: Now we look at the behavior of TD under regularization, which is the main contribution of our paper.

Original: 3.1 Regularization cannot always mitigate off-policy training error.
Revised: 3.1 Regularization cannot always mitigate off-policy training error.

Original: There is a belief in the literature that regularization is a trade-off between reducing the blow-up of asymptotic errors and accurately learning the value function everywhere else [1, 21].
Revised: There is a belief in the literature that regularization is a trade-off between reducing the blow-up of asymptotic errors and accurately learning the value function everywhere else [1, 21].

Original: However, this belief does not accurately capture the nature of regularization: we show that it is possible to learn models that never perform better than always guessing zero despite any amount of regularization.
Revised: However, this belief does not accurately capture the nature of regularization: we show that it is possible to learn models that never perform better than always guessing zero despite any amount of regularization.

Original: That is, the TD error at all η is at least as much as the error as η → ∞ .
Revised: That is, the TD error at all η is at least as much as the error as η → ∞ .

Original: We call such models vacuous .
Revised: We call such models vacuous .

Original: Example 1.
Revised: Example 1.

Original: When TD is regularized, there may exist some off-policy distribution at which TD learns a vacuous model.
Revised: When TD is regularized, there may exist some off-policy distribution at which TD learns a vacuous model.

Original: In notation:
Revised: In notation:

Original: [Equation]
Revised: [Equation]

Original: Details.
Revised: Details.

Original: We use the same setting as in Section 3.
Revised: We use the same setting as in Section 3.

Original: A detailed derivation is provided in Appendix B.2.
Revised: A detailed derivation is provided in Appendix B.2.

Original: We begin by noting that we can easily find the solution ˆ w that minimizes the least-squares error ∥
Revised: We begin by noting that we can easily find the solution ˆ w that minimizes the least-squares error ∥

Original: Φ ˆ w − V ∥ .
Revised: Φ ˆ w − V ∥ .

Original: If we consider this solution as a vector (as drawn in Figure 2a), we can immediately see that there is an ℓ 2 -ball around ˆ w corresponding to the set of w ∗ ( η ) with no more than ∥ V ∥ error.
Revised: If we consider this solution as a vector (as drawn in Figure 2a), we can immediately see that there is an ℓ 2 -ball around ˆ w corresponding to the set of w ∗ ( η ) with no more than ∥ V ∥ error.

Original: Similarly, we can trace the trajectory that the TD solution w ∗ ( η ) takes as η is increased from 0 to ∞ .
Revised: Similarly, we can trace the trajectory that the TD solution w ∗ ( η ) takes as η is increased from 0 to ∞ .

Original: We know that, as η → ∞ , w ∗ ( η ) is crushed to zero and so all trajectories must eventually terminate at the origin.
Revised: We know that, as η → ∞ , w ∗ ( η ) is crushed to zero and so all trajectories must eventually terminate at the origin.

Original: When regularized models are not vacuous, the trajectory intersects the non-vacuous-error ball.
Revised: When regularized models are not vacuous, the trajectory intersects the non-vacuous-error ball.

Original: We see this in trajectory 2, where the error briefly dips below ∥ V ∥ in Figure 2b.
Revised: We see this in trajectory 2, where the error briefly dips below ∥ V ∥ in Figure 2b.

Original: Intuitively, a sufficient condition for a solution to be vacuous is that it remains in the half-space that is tangent to and excludes the non-vacuous parameter ball.
Revised: Intuitively, a sufficient condition for a solution to be vacuous is that it remains in the half-space that is tangent to and excludes the non-vacuous parameter ball.

Original: This is equivalent to finding some distribution µ such that ˆ w ⊤ w ∗ ( η ) ≤ 0 for all η , which we numerically solve to obtain the model in trajectory 1.
Revised: This is equivalent to finding some distribution µ such that ˆ w ⊤ w ∗ ( η ) ≤ 0 for all η , which we numerically solve to obtain the model in trajectory 1.

Original: From Figure 2a we can see the trajectory remains in the half-space, and from Figure 2b we can see that the error is never less than ∥ V ∥ .
Revised: From Figure 2a we can see the trajectory remains in the half-space, and from Figure 2b we can see that the error is never less than ∥ V ∥ .

Original: Trajectory 1 is a vacuous example.
Revised: Trajectory 1 is a vacuous example.

Original: We observe that Example 1, because it remains entirely in the halfspace ˆ w ⊤ w ∗ ( η ) ≤ 0 , could easily be generalized to other forms of regularization.
Revised: We observe that Example 1, because it remains entirely in the halfspace ˆ w ⊤ w ∗ ( η ) ≤ 0 , could easily be generalized to other forms of regularization.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Breaking the Deadly Triad and our counterexample.
Revised: Breaking the Deadly Triad and our counterexample.

Original: In light of our example we examine the work of [21] in which the authors derive a bound for the regularized TD error under a novel doubleprojection update rule.
Revised: In light of our example we examine the work of [21] in which the authors derive a bound for the regularized TD error under a novel doubleprojection update rule.

Original: We apply our example to their bound and show that their method may produce loose bounds on TD solutions, and so doesn’t quite break the deadly triad:
Revised: We apply our example to their bound and show that their method may produce loose bounds on TD solutions, and so doesn’t quite break the deadly triad:

Original: [Equation]
Revised: [Equation]

Original: for ξ ∈ [0 , 1] , where σ max and σ min denote the largest and smallest singular value respectively.
Revised: for ξ ∈ [0 , 1] , where σ max and σ min denote the largest and smallest singular value respectively.

Original: Theorem 2 from [21] bounds η , and therefore also b :
Revised: Theorem 2 from [21] bounds η , and therefore also b :

Original: [Equation]
Revised: [Equation]

Original: Their method bounds the error in our example by 2000 ∗ ∥ V ∥ , which is tremendously loose.
Revised: Their method bounds the error in our example by 2000 ∗ ∥ V ∥ , which is tremendously loose.

Original: (We analyze a different example in Appendix B.3, showing a still-loose but improved bound of 8 ∗ ∥ V ∥ .)
Revised: (We analyze a different example in Appendix B.3, showing a still-loose but improved bound of 8 ∗ ∥ V ∥ .)

Original: This illustrates the risk of relying on regularization to formally bound model error.
Revised: This illustrates the risk of relying on regularization to formally bound model error.

Original: 3.2 Small amounts of regularization can cause large increases in training error.
Revised: 3.2 Small amounts of regularization can cause large increases in training error.

Original: There is a general assumption in the literature that ℓ 2 regularization monotonically shrinks the learned weights.
Revised: There is a general assumption in the literature that ℓ 2 regularization monotonically shrinks the learned weights.

Original: While this is true in classification, regression, and other non-bootstrapping contexts, this is not true in TD.
Revised: While this is true in classification, regression, and other non-bootstrapping contexts, this is not true in TD.

Original: Because TD bootstraps values, it is possible for model bias to be arbitrarily magnified.
Revised: Because TD bootstraps values, it is possible for model bias to be arbitrarily magnified.

Original: This can be understood in terms of the eigenvalues of the matrix A in Equation 5.
Revised: This can be understood in terms of the eigenvalues of the matrix A in Equation 5.

Original: By increasing values along the diagonal, ℓ 2 regularization increases eigenvalues of the matrix ( A + ηI ) to ensure it is positive definite.
Revised: By increasing values along the diagonal, ℓ 2 regularization increases eigenvalues of the matrix ( A + ηI ) to ensure it is positive definite.

Original: Under off-policy distributions, it is possible for A to have eigenvalues that are negative or zero.
Revised: Under off-policy distributions, it is possible for A to have eigenvalues that are negative or zero.

Original: This implies that there are η for which det( A + ηI ) = 0 , and selecting η close to these values allows us to achieve arbitrarily high error.
Revised: This implies that there are η for which det( A + ηI ) = 0 , and selecting η close to these values allows us to achieve arbitrarily high error.

Original: We show one such case in Example 2.
Revised: We show one such case in Example 2.

Original: This is not merely theoretical–we demonstrate this in the neural network case in Section 3.4.
Revised: This is not merely theoretical–we demonstrate this in the neural network case in Section 3.4.

Original: Example 2.
Revised: Example 2.

Original: When TD is regularized, the model may diverge around (typically small) values of η .
Revised: When TD is regularized, the model may diverge around (typically small) values of η .

Original: Lower-bounding η , a common mitigation, can make well-behaved models vacuous.
Revised: Lower-bounding η , a common mitigation, can make well-behaved models vacuous.

Original: It is not always possible to select a single value of η that makes models vacuous at different sampling distributions.
Revised: It is not always possible to select a single value of η that makes models vacuous at different sampling distributions.

Original: Details.
Revised: Details.

Original: Using our three-state example, we set µ = [0 . 05 , 0 . 05 , 0 . 9] and solve for det( A + ηI ) = 0 :
Revised: Using our three-state example, we set µ = [0 . 05 , 0 . 05 , 0 . 9] and solve for det( A + ηI ) = 0 :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ˆ w (1) (2) (3)
Revised: ˆ w (1) (2) (3)

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 10 −1010 2 10 −
Revised: 10 −1010 2 10 −

Original: [Figure]
Revised: [Figure]

Original: 101010∥ V ∥
Revised: 101010∥ V ∥

Original: ϵ (1) (2) (3)
Revised: ϵ (1) (2) (3)

Original: Regularization parameter η
Revised: Regularization parameter η

Original: E rr
Revised: E rr

Original: o r a t c onv e r g e n ce
Revised: o r a t c onv e r g e n ce

Original: As in the introductory example, the error can be made arbitrarily large by setting η ≈ 0 . 0634 .
Revised: As in the introductory example, the error can be made arbitrarily large by setting η ≈ 0 . 0634 .

Original: This small- η divergence effect can appear in several ways, illustrated in Figure 3a.
Revised: This small- η divergence effect can appear in several ways, illustrated in Figure 3a.

Original: Typically, this appears as one or more points at which TD error diverges before the region at which regularization reduces the model error below ∥ V ∥ .
Revised: Typically, this appears as one or more points at which TD error diverges before the region at which regularization reduces the model error below ∥ V ∥ .

Original: The first and second plot in Figure 3a show two such cases, where the error increases sharply at two and one points respectively.
Revised: The first and second plot in Figure 3a show two such cases, where the error increases sharply at two and one points respectively.

Original: In the literature it is commonly assumed that A is “nearly” positive definite, where only a few eigenvalues are non-positive, and those are close to zero.
Revised: In the literature it is commonly assumed that A is “nearly” positive definite, where only a few eigenvalues are non-positive, and those are close to zero.

Original: This gives rise to the common mitigation of setting a lower-bound η 0 such that ( A + ηI ) is positive definite for η > η 0 .
Revised: This gives rise to the common mitigation of setting a lower-bound η 0 such that ( A + ηI ) is positive definite for η > η 0 .

Original: This may render an otherwise well-behaved model vacuous.
Revised: This may render an otherwise well-behaved model vacuous.

Original: The third plot in Figure 3a illustrates this: the model is not vacuous when unregularized, but is vacuous in the domain η > 10 − 2 where divergence is prohibited.
Revised: The third plot in Figure 3a illustrates this: the model is not vacuous when unregularized, but is vacuous in the domain η > 10 − 2 where divergence is prohibited.

Original: A common practice in the literature is to set η before training, without regard for the sampling distribution.
Revised: A common practice in the literature is to set η before training, without regard for the sampling distribution.

Original: This is ill advised, as the value may be under- or over-regularizing depending on the sampling distribution.
Revised: This is ill advised, as the value may be under- or over-regularizing depending on the sampling distribution.

Original: One such example is illustrated in Figure 3b, where selecting an η that minimizes the error for one distribution will lead to vacuous or nearly-vacuous results in the other two.
Revised: One such example is illustrated in Figure 3b, where selecting an η that minimizes the error for one distribution will lead to vacuous or nearly-vacuous results in the other two.

Original: A second example in Figure 2b has no single η for which trajectories 2 and 3 are both non-vacuous.
Revised: A second example in Figure 2b has no single η for which trajectories 2 and 3 are both non-vacuous.

Original: This is especially relevant as regularization is commonly used to permit distribution drift during training, as discussed in Section 4.
Revised: This is especially relevant as regularization is commonly used to permit distribution drift during training, as discussed in Section 4.

Original: If the training distribution changes while η is fixed, then algorithms that can be proven to converge to good solutions under some original distribution may converge to poor solutions as the distribution drifts.
Revised: If the training distribution changes while η is fixed, then algorithms that can be proven to converge to good solutions under some original distribution may converge to poor solutions as the distribution drifts.

Original: 3.3 Emphatic approaches and our counterexample
Revised: 3.3 Emphatic approaches and our counterexample

Original: Emphatic-TD eliminates instability from off-policy sampling by reweighting incoming data (via an importance function)
Revised: Emphatic-TD eliminates instability from off-policy sampling by reweighting incoming data (via an importance function)

Original: so it appears to be on-policy.
Revised: so it appears to be on-policy.

Original: There is considerable interest in making this more practical, especially by learning the importance and value models simultaneously.
Revised: There is considerable interest in making this more practical, especially by learning the importance and value models simultaneously.

Original: A leading example of this work is COF-PAC [20], which uses ℓ 2 -regularized versions of GTD2 [12] to learn both the value and emphasis models.
Revised: A leading example of this work is COF-PAC [20], which uses ℓ 2 -regularized versions of GTD2 [12] to learn both the value and emphasis models.

Original: The authors rely on regularization, particularly because the target policy changes during learning.
Revised: The authors rely on regularization, particularly because the target policy changes during learning.

Original: This makes COF-PAC vulnerable to regularization-caused error.
Revised: This makes COF-PAC vulnerable to regularization-caused error.

Original: We illustrate this with Example 3 in which COF-PAC learns correctly when unregularized, but has large error when regularized.
Revised: We illustrate this with Example 3 in which COF-PAC learns correctly when unregularized, but has large error when regularized.

Original: (Mathematical details are deferred to Appendix B.5.)
Revised: (Mathematical details are deferred to Appendix B.5.)

Original: Example 3.
Revised: Example 3.

Original: COF-PAC may learn the value function with low error when unregularized, but with arbitrarily high error when regularized.
Revised: COF-PAC may learn the value function with low error when unregularized, but with arbitrarily high error when regularized.

Original: Details.
Revised: Details.

Original: Conceptually, COF-PAC maintains two separate models that are each updated by TD: the emphasis and the value models.
Revised: Conceptually, COF-PAC maintains two separate models that are each updated by TD: the emphasis and the value models.

Original: This emphasis model is used to reweight TD updates to the value
Revised: This emphasis model is used to reweight TD updates to the value

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Emphasis Model Error
Revised: Emphasis Model Error

Original: [Figure]
Revised: [Figure]

Original: Specific values of η m induce the value function to diverge (Figure 4b).
Revised: Specific values of η m induce the value function to diverge (Figure 4b).

Original: The resultant value function is vacuous (Figure 4c).
Revised: The resultant value function is vacuous (Figure 4c).

Original: Under COF-PAC, regularization can greatly increase model error.
Revised: Under COF-PAC, regularization can greatly increase model error.

Original: function so they appear to come from the on-policy distribution.
Revised: function so they appear to come from the on-policy distribution.

Original: Our strategy is to first show how regularization biases the emphasis model, and then how this bias causes the value model to diverge.
Revised: Our strategy is to first show how regularization biases the emphasis model, and then how this bias causes the value model to diverge.

Original: We begin with our three-state MP, noting its on-policy distribution is π = [ . 25 .
Revised: We begin with our three-state MP, noting its on-policy distribution is π = [ . 25 .

Original: We wish to learn the values using COF-PAC while sampling off-policy at µ = [ . 2 . 2 . 6] .
Revised: We wish to learn the values using COF-PAC while sampling off-policy at µ = [ . 2 . 2 . 6] .

Original: Now we introduce a key conceptual tool: υ ( η m ) , which is the effective distribution seen by the
Revised: Now we introduce a key conceptual tool: υ ( η m ) , which is the effective distribution seen by the

Original: TD-updates, influenced by the emphasis regularization parameter η m .
Revised: TD-updates, influenced by the emphasis regularization parameter η m .

Original: Unregularized, COF-PAC is able to resample off-policy updates to the on-policy distribution: υ (0) ≡ π .
Revised: Unregularized, COF-PAC is able to resample off-policy updates to the on-policy distribution: υ (0) ≡ π .

Original: If the model is regularized, then the effective distribution moves away from π .
Revised: If the model is regularized, then the effective distribution moves away from π .

Original: Figure 4a illustrates the distance between υ ( η m ) and π as the regularlization parameter increases.
Revised: Figure 4a illustrates the distance between υ ( η m ) and π as the regularlization parameter increases.

Original: We can use the effective distribution to compute the error in the value model.
Revised: We can use the effective distribution to compute the error in the value model.

Original: Plotting the relationship between the value function error and η m in Figure 4b, we can see the value function has asymptotic error around η m = 2 × 10 − 4 .
Revised: Plotting the relationship between the value function error and η m in Figure 4b, we can see the value function has asymptotic error around η m = 2 × 10 − 4 .

Original: This shows how COF-PAC may diverge with specific regularization.
Revised: This shows how COF-PAC may diverge with specific regularization.

Original: COF-PAC also allows for the value function to be separately regularized with parameter η v .
Revised: COF-PAC also allows for the value function to be separately regularized with parameter η v .

Original: We show the effect of this in Figure 4c, where the value function never does much better than ∥ V ∥ making it (nearly) vacuous.
Revised: We show the effect of this in Figure 4c, where the value function never does much better than ∥ V ∥ making it (nearly) vacuous.

Original: We can conclude that regularizing the emphasis model may cause the value model to diverge, and this cannot be fixed by regularizing the value function separately.
Revised: We can conclude that regularizing the emphasis model may cause the value model to diverge, and this cannot be fixed by regularizing the value function separately.

Original: [Figure]
Revised: [Figure]

Original: COF-PAC makes the strong assumption that Kolter’s relaxed-contraction condition [6, eqn.
Revised: COF-PAC makes the strong assumption that Kolter’s relaxed-contraction condition [6, eqn.

Original: 10] holds in both the emphasis and value models [20, asm. 4].
Revised: 10] holds in both the emphasis and value models [20, asm. 4].

Original: We discuss this in Appendix B.5.1.
Revised: We discuss this in Appendix B.5.1.

Original: 3.4 Applied to multi-layer networks
Revised: 3.4 Applied to multi-layer networks

Original: We use a 9-state variant of our example to study the deadly triad in multi-layer neural networks (NNs).
Revised: We use a 9-state variant of our example to study the deadly triad in multi-layer neural networks (NNs).

Original: A deterministic observation function is chosen so we can control the degree of function approximation.
Revised: A deterministic observation function is chosen so we can control the degree of function approximation.

Original: We train a simple two-layer neural network with 3 neurons in the hidden layer.
Revised: We train a simple two-layer neural network with 3 neurons in the hidden layer.

Original: The value function is assigned pseudo-randomly in range [ − 1 , 1] .
Revised: The value function is assigned pseudo-randomly in range [ − 1 , 1] .

Original: (See Appendix C for details.)
Revised: (See Appendix C for details.)

Original: Example 4.
Revised: Example 4.

Original: Vacuous models and small- η error also occur in neural network conditions.
Revised: Vacuous models and small- η error also occur in neural network conditions.

Original: Details.
Revised: Details.

Original: We train 100 models using simple semi-gradient TD updates under a fixed learning rate.
Revised: We train 100 models using simple semi-gradient TD updates under a fixed learning rate.

Original: We plot the mean and the 10 th – 90 th percentile range in Figure 5a, with and without regularization.
Revised: We plot the mean and the 10 th – 90 th percentile range in Figure 5a, with and without regularization.

Original: TD is known to exhibit high variance, and regularization is the traditional remedy for that.
Revised: TD is known to exhibit high variance, and regularization is the traditional remedy for that.

Original: We corroborate this by noting that the performance of the unregularized model varies widely, but regularization leads to similar performance across initializations at the cost of increased error.
Revised: We corroborate this by noting that the performance of the unregularized model varies widely, but regularization leads to similar performance across initializations at the cost of increased error.

Original: First, we show that vacuous models may exist in the neural network case.
Revised: First, we show that vacuous models may exist in the neural network case.

Original: In Figure 5a, note how there are some off-policy distributions under which both the regularized and unregularized models perform worse than the threshold of vacuity.
Revised: In Figure 5a, note how there are some off-policy distributions under which both the regularized and unregularized models perform worse than the threshold of vacuity.

Original: We can numerically verify that vacuous models exist.
Revised: We can numerically verify that vacuous models exist.

Original: Second, we show the small- η error problem in the neural network case in Figure 5b, where we plot the TD error against η at a fixed off-policy distribution.
Revised: Second, we show the small- η error problem in the neural network case in Figure 5b, where we plot the TD error against η at a fixed off-policy distribution.

Original: We observe that around η ≈ 10 − 3 the TD
Revised: We observe that around η ≈ 10 − 3 the TD

Original: Error unexpectedly increases before decreasing, which clearly illustrates this phenomenon.
Revised: Error unexpectedly increases before decreasing, which clearly illustrates this phenomenon.

Original: These qualitative links show a clear connection between the neural network case and the linear case, and highlights the importance of correctly handling off-policy sampling.
Revised: These qualitative links show a clear connection between the neural network case and the linear case, and highlights the importance of correctly handling off-policy sampling.

Original:  Related Work
Revised:  Related Work

Original: Three examples of the deadly triad are common in the literature: the classic Tsitsiklis and Van Roy ( w, 2 w ) example [11, p. 260], Kolter’s example [6], and Baird’s counterexample which shows how training instability can exist despite overparameterization [18].
Revised: Three examples of the deadly triad are common in the literature: the classic Tsitsiklis and Van Roy ( w, 2 w ) example [11, p. 260], Kolter’s example [6], and Baird’s counterexample which shows how training instability can exist despite overparameterization [18].

Original: ℓ 2 regularization is common when proving that an algorithm converges under a changing sampling policy.
Revised: ℓ 2 regularization is common when proving that an algorithm converges under a changing sampling policy.

Original: This is seen in GTD (analyzed in [19]), GTD2 [12], RO-TD [10], and COF-PAC [20].
Revised: This is seen in GTD (analyzed in [19]), GTD2 [12], RO-TD [10], and COF-PAC [20].

Original: This assumption may also be used to ensure convergence when training with a target network [21].
Revised: This assumption may also be used to ensure convergence when training with a target network [21].

Original: Despite the prevalence of regularization, the induced bias from using it is not well studied.
Revised: Despite the prevalence of regularization, the induced bias from using it is not well studied.

Original: It is often dismissed as a mere technical assumption, as in [1].
Revised: It is often dismissed as a mere technical assumption, as in [1].

Original: We contradict that: using regularization for convergence proofs may induce catastrophic bias.
Revised: We contradict that: using regularization for convergence proofs may induce catastrophic bias.

Original: By showing concrete examples, this work hopes to inspire further investigation into regularization-induced bias in the same vein as [19].
Revised: By showing concrete examples, this work hopes to inspire further investigation into regularization-induced bias in the same vein as [19].

Original: Alternatives to regularization and TD We focus on ℓ 2 regularization in this paper, which penalizes the ℓ 2 -norm of the learned weights; it is also possible to use ℓ 1 regularization with a proximal operator/saddle point formulation as in [10], or any convex regularization term under a fixed target policy [19].
Revised: Alternatives to regularization and TD We focus on ℓ 2 regularization in this paper, which penalizes the ℓ 2 -norm of the learned weights; it is also possible to use ℓ 1 regularization with a proximal operator/saddle point formulation as in [10], or any convex regularization term under a fixed target policy [19].

Original: Instead of directly regularizing the weights, COP-TD uses a discounted update [4].
Revised: Instead of directly regularizing the weights, COP-TD uses a discounted update [4].

Original: DisCor [7] propagates bounds on Q-value estimates to quickly converge TD learning in the face of large bootstrapping error; it is not clear if DisCor can overcome off-policy sampling.
Revised: DisCor [7] propagates bounds on Q-value estimates to quickly converge TD learning in the face of large bootstrapping error; it is not clear if DisCor can overcome off-policy sampling.

Original: A separate primal-dual saddle point method has also been adapted to ℓ 2 regularization [2] and is known to converge under deadly triad conditions, and recent work [17] has derived error bounds with improved scaling properties in the linear setting, offering a promising line of research.
Revised: A separate primal-dual saddle point method has also been adapted to ℓ 2 regularization [2] and is known to converge under deadly triad conditions, and recent work [17] has derived error bounds with improved scaling properties in the linear setting, offering a promising line of research.

Original: Emphatic-TD [13] fixes the fundamental problem in off-policy TD by reweighting updates so they appear on-policy.
Revised: Emphatic-TD [13] fixes the fundamental problem in off-policy TD by reweighting updates so they appear on-policy.

Original: The core idea underlying these techniques is to estimate the “followon trace” for each state, the (weighted, λ - and γ − discounted) probability mass of all states whose value estimates it influences.
Revised: The core idea underlying these techniques is to estimate the “followon trace” for each state, the (weighted, λ - and γ − discounted) probability mass of all states whose value estimates it influences.

Original: This trace is then used to estimate the emphasis, which is the reweighting factor for each update.
Revised: This trace is then used to estimate the emphasis, which is the reweighting factor for each update.

Original: While this family of methods is provably optimal in expectation, it is subject to tremendous variance in theory and practice, especially when the importance is estimated using Monte-Carlo sampling.
Revised: While this family of methods is provably optimal in expectation, it is subject to tremendous variance in theory and practice, especially when the importance is estimated using Monte-Carlo sampling.

Original: In practice, these methods learn the follow-on trace using TD [5, 20] or similar [16], which makes them vulnerable to bias induced by the use of regularization.
Revised: In practice, these methods learn the follow-on trace using TD [5, 20] or similar [16], which makes them vulnerable to bias induced by the use of regularization.

Original:  Conclusion
Revised:  Conclusion

Original: There is a tremendous focus in the RL literature on proving convergence of novel algorithms, but not on the error at convergence.
Revised: There is a tremendous focus in the RL literature on proving convergence of novel algorithms, but not on the error at convergence.

Original: Papers like [21] are laudable because they provide error bounds; even if the current bounds are loose, future work will no doubt tighten them.
Revised: Papers like [21] are laudable because they provide error bounds; even if the current bounds are loose, future work will no doubt tighten them.

Original: In this work, we show that the popular technique of ℓ 2 regularization does not always prevent singularities and could even introduce catastrophic divergence.
Revised: In this work, we show that the popular technique of ℓ 2 regularization does not always prevent singularities and could even introduce catastrophic divergence.

Original: We show this with a new counterexample that elegantly illustrates the problems with learning off-policy and how it persists into the NN case.
Revised: We show this with a new counterexample that elegantly illustrates the problems with learning off-policy and how it persists into the NN case.

Original: Even though regularization can catastrophically fail in the ways we illustrate, it remains a reasonable method that may offer a fair tradeoff – as long as we are careful to check that we are not running afoul of the failure modes we explain in the paper.
Revised: Even though regularization can catastrophically fail in the ways we illustrate, it remains a reasonable method that may offer a fair tradeoff – as long as we are careful to check that we are not running afoul of the failure modes we explain in the paper.

Original: It may be possible to design an adaptive regularization scheme that can avoid these pathologies.
Revised: It may be possible to design an adaptive regularization scheme that can avoid these pathologies.

Original: For now, testing the model performance over a range of regularization parameters (spanning several orders of magnitude) is the best option we have to detect such pathological behavior.
Revised: For now, testing the model performance over a range of regularization parameters (spanning several orders of magnitude) is the best option we have to detect such pathological behavior.

Original: Emphatic-TD is perhaps the most promising area of research for mitigating off-policy TD-learning.
Revised: Emphatic-TD is perhaps the most promising area of research for mitigating off-policy TD-learning.

Original: The key problem preventing its widespread adoption is the difficulty in estimating the emphasis function, but future work in this area may be able to overcome this.
Revised: The key problem preventing its widespread adoption is the difficulty in estimating the emphasis function, but future work in this area may be able to overcome this.

Original: Our example shows the risk of relying on regularization in practical implementations of such methods.
Revised: Our example shows the risk of relying on regularization in practical implementations of such methods.

Original: It is absolutely critical that
Revised: It is absolutely critical that

Original: Emphatic algorithms correctly manage regularization to avoid the risks that we highlight in this paper.B Example Details We provide a more detailed explanation of our examples.
Revised: Emphatic algorithms correctly manage regularization to avoid the risks that we highlight in this paper.

Original: B.1 “Vacuous” models

Original: Without ℓ 2 regularization

Original: [Table]

Original: our linear model fails with asymptotic error.

Original: As this penalizes the ℓ 2 -norm of the learned weights, this removes the asymptote and so we can no longer use the existence of an asymptote as evidence of failure.

Original: Instead, we propose a different definition of failure by noting that, in the limiting case, regularization drives the learned weights to zero ( lim η →∞ w ∗ ( η ) = ⃗ 0 ).

Original: [-The learned value function Φ · ⃗ 
-]
Revised: [-The learned value function Φ · ⃗ 
-]

Original: = ⃗ 0 has no information about the true value function.

Original: We argue that if the error with any η ∈ R + is never better than this case then the model is vacuous and hence adopt the threshold error of ∥ Φ · ⃗ 0 − V ∥

Original: =

Original: ∥ V ∥ to call a model vacuous.

Original: This explains the failure condition in Equation 8.

Original: B.2 Details of regularization example

Original: This provides numeric details for Example 1.

Original: Example 5.

Original: When TD is regularized, there may exist some off-policy distribution at which TD learns a vacuous model.

Original: In notation:

Original: [Equation]

Original: Details.

Original: We use the same setting as in Section 3.

Original: We observe that ˆ w =

Original: [1 , − 1]

Original: ⊤ minimizes the least-squares error ∥

Original: Φ ˆ w − V ∥ , and further observe that a sufficient (but not necessary) condition for a solution to be vacuous is that ˆ w ⊤ w ∗ ( η ) ≤ 0 .

Original: Solving:

Original: [Equation]

Original: We verify that TD is vacuous at p = 0 . 102636 by computing the TD error at convergence:

Original: [Equation]

Original: Since the fraction term in Equation 15 is obviously improper, we can conclude that our example will always have at least

Original: ∥ V ∥ error over all η , and is therefore vacuous.

Original: We note that the error is not defined at η = 0 because this corresponds to a model divergence similar to our introductory example.

Original: In practice, the TD fixed point will still converge to a vacuous solution:

Original: [Equation]

Original: B.3 Additional simple regularization example

Original: We present a second example where the error is stationary with respect to the regularization parameter.

Original: This is worse than Example 5 because we are able to show that the point the model converges to is independent of regularization.

Original: Example 6.

Original: When TD is regularized, there may exist some off-policy distribution at which the TD fixed point is independent of the regularization parameter.

Original: Details.

Original: We use the same setting as in Section 3, except the value function is V =

Original: [1 , 1 , 1 . 05] ⊤ and basis Φ selected to have small representation error ∥

Original: Π D V − V ∥ ≤ ϵ :

Original: [Equation]

Original: where ϵ >

Original: [Equation]

Original: We set ϵ = 10 − 4 and write down w ∗ ( η ) in terms of g , a scalar function of η and p :

Original: [Equation]

Original: When g ( p, η ) ≤ 0 , the TD solution is vacuous.

Original: We show that directly:

Original: [Equation]

Original: When g ( p, η ) ≤ 0 , then ∥ g ( p, η ) − 1 ∥ ≥ 1 for all η and the TD solution is vacuous.

Original: We find such a solution by noting the numerator has two roots in p , one of which corresponds to a vacuous solution: g (0 .

Original: [-715083 , η ) = 
-]
Revised: [-715083 , η ) = 
-]

Original: ( ∀ η ) , and this completes the example!

Original: In this setting, when TD updates follow the sampling distribution p ≈ 0 . 715083 , the error of the model at convergence is always ∥ V ∥ regardless of regularization.

Original: Our example converges to the same vacuous value regardless η .

Original: In Figure 6, we can see that the TD error intersects the η → ∞ line immediately before and after the singularity.

Original: Our counterexample corresponds to the second root (that is, the intersection point at higher p .)

Original: This is because that corresponds to the stationary point between the asymptote that is crushed and the error on the right that increases.

Original: If our simpler derivation proved unsatisfying, we can also derive this counterexample using this fact:

Original: [Equation]

Original: From this, we can easily see that the counterexample is at p = 0 .

Original: And this completes the example!

Original: We have discovered some p at which the TD error is always at least ∥ V ∥ , regardless of regularization, and so our example learns a vacuous value function.

Original: η → ∞ η =p = 0 .

Original: p in sampling distribution µ = [ p / 2 , p / 2 , 1 − p ]

Original: [Figure]

Original: B.3.1 Breaking the Deadly Triad and our counterexample.

Original: In light of our counterexample we examine the work of [21] in which the authors derive a bound for the regularized TD error under a novel double-projection update rule.

Original: We apply our example to their bound b to show that their method permits vacuous TD solutions and doesn’t quite break the deadly triad.

Original: Starting from Equation 9:

Original: [Equation]

Original: (22) for ξ ∈ [0 , 1] , where σ max and σ min denote the largest and smallest singular value respectively.

Original: Theorem 2 from [21] bounds η , and therefore also b :

Original: [Equation]

Original: [Equation]

Original: Under our example, their method bounds the error at no more than 7 .

Original: 86 ∗ ∥ V ∥ , which is a very loose bound that permits vacuous solutions.

Original: This illustrates the risk of trying to regularize away singularities, particularly in theoretical work.

Original: Investigating the cause of the loose bounds reveals that the presence of σ min ( D ) 2 . 5 in 9 is largely responsible.

Original: As D is a diagonal matrix encoding the sampling distribution, σ min ( D ) is the smallest sampling rate of any state, and so the bound must be at least η ξn 2 .

Original: 5 for any perfectly representable n -state MP.

Original: Unfortunately, this appears to be fundamental limit caused by finding a linear bound to an error that scales non-linearly, and following their derivation in the appendix does not readily admit a way to improve this.

Original: B.4 Small-Eta Error

Original: Our simplified example allows us to show this easily.

Original: Example 7.

Original: When TD is regularized, the model may diverge around (typically small) values of η .

Original: η =η = 2 · 10 −

Original: With Regul.

Original: distrib.

Original: param.

Original: p d i s t r i b .

Original: e rr .

Original: ∥ υ ( η ) − π ∥

Original: Emphasis Model Error

Original: [Figure]

Original: η =υ ( 0 ) υ (·− 4 )

Original: With Regul.

Original: distrib.

Original: param.

Original: q v a l u e e rr .

Original: ∥

Original: Φ v w ∗ v − V ∥

Original: Value Model Error

Original: [Figure]

Original: Details.

Original: We set p = 0 .

Original: 9 and solve for det( A + ηI ) = 0 :

Original: [Equation]

Original: [Equation]

Original: Note that the denominator of g ( p, η ) is proportional to det( A + ηI ) , and so g (0 . 9 , η ) –and the error at the TD fixed point–can be made arbitrarily large by selecting η close to 4 .

Original: 83 × 10 − 3 .

Original: As this is the only positive root, the model does not diverge at other values.

Original: B.5 Emphatic approaches and our counterexample

Original: We use an MP with the same transition function as in Figure 1a, with separate bases Φ m and Φ v for the emphasis and value stages respectively.

Original: We assume that our interest in all states is uniformly i = 1 .

Original: We begin by setting the off-policy sampling distribution of µ = [ . 2 . 2 . 6] , used as the diagonal matrix D µ = diag ( µ ) .

Original: Thanks to the simple structure of our example, we know the emphasis is m =

Original: i 1 − γ · πD − 1 µ ∝ ( 5 / 4 , 5 / 4 , 5 / 6 ) .

Original: We select a basis that allows us to represent this emphasis:

Original: [Equation]

Original: We deliberately choose Φ m to have a poor condition number for reasons that will become apparent later.

Original: We can represent c · ( 5 / 4 , 5 / 4 , 5 / 6 ) exactly for any constant c :

Original: [Equation]

Original: [Equation]

Original: Using Equation 5 from [20], we define the matrices:

Original: [Equation]

Original: And we apply these to the formulation in Lemma 3 and compute the emphasis weights as a function of the regularization w m :

Original: [-R +
-]
Revised: [-R +
-]

Original: → R + :

Original: [Equation]

Original: [Equation]

Original: We can then use this to compute the new apparent distribution υ , which is the effective distribution that the updates to the value model see, and it is equal to the emphasis multiplied by the off-policy distribution.

Original: [Equation]

Original: Without any regularization, this should be exactly equal to the on-policy distribution.

Original: [Equation]

Original: When we compute this value with a small amount of regularization η = 2 × 10 − 4 , we observe that the apparent distribution drifts far away from the on-policy distribution.

Original: [Equation]

Original: The proximate cause of this is the poor condition number of C , caused by the 1100 scale factor applied to the second column of Φ m .

Original: This allows η to affect different columns by different (relative) amounts in the definition of w ∗ ( η ) , which pushes it away from the symmetric solution.

Original: See this error shift in

Original: Figure 7a.

Original: So far, we have shown how regularization causes a shift in the apparent distribution that the TD updates see.

Original: To complete the example we show how this moves the fixed point of the value function away from a stable point into an asymptote where it may grow without bounds.

Original: This second phase follows in the same pattern as the first phase, starting with the desired value function: V = [1 2 .

Original: 69 1 . 05] and a basis that can almost exactly represent the value function:

Original: [Equation]

Original: We use this basis to compute the state-rewards R =

Original: ( I − γP )

Original: V =

Original: [ − 0 .

Original: 43 1 . 26 − 0 .

Original: 38] and define the matrices A v and C v and the solution w ∗ v ( η ) :

Original: [Equation]

Original: We can use this solution to compute the error between the value function and the true values, ∥ Φ v w v ( η ) − V ∥ .

Original: First, under the corrected distribution without regularization υ (0) ≡ π :

Original: [Equation]

Original: Then, with regularization in the value function (but not in the emphasis function):

Original: [Equation]

Original: Then, under the apparent distribution υ induced by use of regularization in the emphasis function, without and with regularization:

Original: [Equation]

Original: It is immediately obvious that the use of regularization in the emphasis function causes the learned value function to be incorrect.

Original: Including a regularizing term in the value estimate is not sufficient to fix the value function.

Original: This completes the example.

Original: B.5.1 Kolter’s non-expansion condition and our counterexample.

Original: In the construction of COF-PAC, a key assumption made is that both the emphasis and value models are not subject to runaway TD [20, asm. 4].

Original: Specifically, they make the strong assumption that

Original: Kolter’s relaxed-contraction condition [6, eqn.

Original: 10] holds in the emphasis model at µ and value model at υ .

Original: Kolter’s condition selects a convex subset of distributions under which one-step transition followed by projection onto Φ is non-expansive.

Original: We illustrate these regions in Figure 8.

Original: Even in the one-dimensional parameterization shown, this condition only holds in a small sub-region of the space and therefore appears to be a very strong condition.

Original: Empirically determining if such a condition holds (or training models to enforce it) may be possible with TD-DO [6, sec 4.1], but it is not clear how that method interacts with regularization.

Original: From Figure 0 .0 .0 .0 .

Original: From Figure 7(a) 0 .0 .0 .0 .

Original: From Figure 7(b)

Original: C Applied to multi-layer networks

Original: We also use a variant of our example to study how the deadly triad appears in multi-layer networks.

Original: As illustrated in Figure 11, we replace each self-loop with two additional states, forming a clique with the original state.

Original: The resultant MP has n = 9 states; we define a deterministic obervation function o : S

Original: → B 6 .

Original: where each state is encoded as the concatenation of the one-hot vector of its subscripts.

Original: The value function is assigned pseudo-randomly in range [ − 1 , 1] , and a consistent reward function is assigned.

Original: We select the family of sampling distributions µ ∝ [4 p, 1 p, 1 p, 4 p, 1 p, 1 p, 8(1 − p ) , 4(1 − p ) , 4(1 − p )] , where the on-policy distribution is at p = 0 .

Original: We wish to learn the model with a two-layer network with k < n nodes in the inner layer.

Original: We define the network as f ( o ( s i,j )) =

Original: tan − 1 ( o ( s i,j ) ∗ ω 1 ) ∗ ω 2 .

Original: The parameters ω 1 ∈ R 6 × k , ω 2 ∈ R k × 1 are trained to convergence using simple TD updates with semi-gradient updates, a fixed learning rate, and without a target network.

Original: In addition to the example in Figure 5b, we present an additional example in Figure 9.

Original: The same

Original: Markov process, at a different off-policy distribution, attains a curve where the non-vacuous region lies before the divergent region, similar to the second row in Figure 3a.

Original: An added observation is that these two graphs are mutually incompatible – there is no fixed η that can simultaneously do better than vacuity in both, which promotes the idea of testing multiple regularization parameters or using an adaptive regularization scheme.

Original: C.1 Overparameterization does not solve this problem

Original: Baird’s counterexample [18] shows how, in the linear case, that off-policy divergence can also happen under overparameterization, as long as some amount of function approximation occurs.

Original: It is not obvious that this conclusion persists in the neural network case, so we include an additional example showing that the parameterization doesn’t resolve small- η divergence.

Original: [Figure]

Original: 5b

Original: [Figure]

Original: In Figure 10 we plot models with 3 to 13 nodes in the hidden layer.

Original: For reference, the MDP hasstates, so some models under-parameterize and some models over-parameterize.

Original: We observe that, in the low-regularization regime, increasing the number of parameters improves the error slightly.

Original: However, increasing the number of parameters in the hidden layer does not change the behavior in the the small- η divergence region.

Original: C.2 Relationship to modern RL algorithms

Original: It is still not obvious how strongly this instability affects modern RL algorithms, which are also sensitive to a variety of other failure modes.

Original: Unlike our analysis, the sampling distribution changes during training, and regularization mechanisms are more complex than simple ℓ 2 penalities.

Original: The exact relationship between the instabilities we study and RL algorithms is an open problem, but we offer two pieces of indirect evidence suggesting there is a link.

Original: First, in the offline/batch RL literature, it is well-known that online RL algorithms naively applied can catastrophically fail if the learned policy is not consistent with the data distribution.

Original: This is known as the distribution shift problem, [9, p. 26] and offline RL algorithms are generally constructed to explicitly address this.

Original: Second, when using experience replay buffers in online RL algorithms, policy quality generally improves when older transitions are more quickly evicted [3].

Original: However, there are multiple factors at work here, and it is not possible to separate out the instability from off-policy sampling from the remaining factors.

Original: D Markov Processes

Original: We use a three-, five- and nine-state Markov Processes to generate examples for this paper.

Original: Here we give details of the construction of each example.

Original: Mathematica code for all examples is included in the supplementary material.

Original: D.1 Three-state

Original: The construction of the three-state MDP is described in Section 3 and illustrated in Figure 1a.

Original: This example is used in Examples 1 and 3 For completeness, the transition matrix is:

Original: [Equation]

Original: 10 − 5 10 − 4 10 − 3 10 − 2 10 −

Original: [Equation]

Original: 10 0   Regularization parameter

Original: [Equation]

Original: 10 − 5 10 −

Original: [Figure]

Original: s 3 ,s 1 ,s 2 , 1 1 ⁄1 ⁄1 ⁄1 ⁄1 ⁄ 4 1 ⁄s 2 ,s 2 ,s 2 ,e ⁄e ⁄e ⁄

Original: D

Original: 2 Nine-state

Original: This example is used to train neural networks.

Original: The construction is based on the three-state example and the construction is illustrated in Figures 1b and 11.

Original: [-The transition matrix (with omitted zeros) is:   1 1 1 
-]
Revised: [-The transition matrix (with omitted zeros) is:   1 1 1 
-]

Original: 24 44 4  (36) and the observation function that forces the neural network to approximate is: o : S

Original: → R 6 =   1 0 0 1 00 1 0 1 00 0 1 1 01 0 0 0 10 1 0 0 10 0 1 0 11 0 0 0 00 1 0 0 00 0 1 0 0  (37)

Original: D.3 Five-state

Original: We use this to generate Figure 3a.

Original: The transition matrix is:   .

Original: 2 . 4 .  (38)

Original: We set value function V =

Original: [1 , 1 , 1 . 05 , 1 , 1] ⊤ , γ = 0 .

Original: 99 , ϵ = 0 .

Original: and set basis: Φ = 1  3 00 31 10 01 1  ∗ (cid:34)  0 0 .

Original: 0 .(cid:35) (39)

Original: We also parameterize the off-policy distribution as:

Original: [Equation]

Original: where p, q ∈ (0 , 1) .

Original: We verify that (cid:80) D

Original: = 1 over this domain.

Original: The on-policy distribution is π = 1 / 12 [2 , 3 , 2 , 3 , 2] .

Original: The plots correspond to the off-policy distributions:

Original: p → 0 . 77 , q → 0 .2.

Original: p → 0 . 4 , q → 0 .3.

Original: p → 0 . 02 , q → 0 .