Original: The Pitfalls of Regularization in Off-Policy TD Learning
Revised: The Pitfalls of Regularization in Off-Policy TD Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Temporal Difference (TD) learning is ubiquitous in reinforcement learning, whereit is often combined with off-policy sampling and function approximation.
Revised: Temporal Difference (TD) learning is ubiquitous in reinforcement learning, whereit is often combined with off-policy sampling and function approximation.

Original: Unfortunately learning with this combination (known as the deadly triad ), exhibitsinstability and unbounded error.
Revised: Unfortunately learning with this combination (known as the deadly triad ), exhibitsinstability and unbounded error.

Original: To account for this, modern RL methods oftenimplicitly (or sometimes explicitly) assume that regularization is sufficient to mitigate the problem in practice; indeed, the standard deadly triad examples from theliterature can be “fixed” via proper regularization.
Revised: To account for this, modern RL methods oftenimplicitly (or sometimes explicitly) assume that regularization is sufficient to mitigate the problem in practice; indeed, the standard deadly triad examples from theliterature can be “fixed” via proper regularization.

Original: In this paper, we introduce aseries of new counterexamples to show that the instability and unbounded errorof TD methods is not solved by regularization.
Revised: In this paper, we introduce aseries of new counterexamples to show that the instability and unbounded errorof TD methods is not solved by regularization.

Original: We demonstrate that, in the offpolicy setting with linear function approximation, TD methods can fail to learn anon-trivial value function under any amount of regularization; we further show thatregularization can induce divergence under common conditions; and we show thatone of the most promising methods for mitigating this divergence (emphatic TDalgorithms) may themselves diverge under regularization.
Revised: We demonstrate that, in the offpolicy setting with linear function approximation, TD methods can fail to learn anon-trivial value function under any amount of regularization; we further show thatregularization can induce divergence under common conditions; and we show thatone of the most promising methods for mitigating this divergence (emphatic TDalgorithms) may themselves diverge under regularization.

Original: We also demonstratesuch divergence when using neural networks as function approximations.
Revised: We also demonstratesuch divergence when using neural networks as function approximations.

Original: Thus,we argue that the role of regularization in TD methods needs to be reconsidered,given that it is insufficient to prevent divergence and may itself introduce instability.
Revised: Thus,we argue that the role of regularization in TD methods needs to be reconsidered,given that it is insufficient to prevent divergence and may itself introduce instability.

Original: There needs to be much more care, both practically and theoretically, in theapplication of regularization to RL methods.
Revised: There needs to be much more care, both practically and theoretically, in theapplication of regularization to RL methods.

Original:  Introduction
Revised:  Introduction

Original: Temporal Difference (TD) learning is a method for learning expected future-discounted quantitiesfrom Markov processes, using transition samples to iteratively improve estimates.
Revised: Temporal Difference (TD) learning is a method for learning expected future-discounted quantitiesfrom Markov processes, using transition samples to iteratively improve estimates.

Original: This is mostcommonly used to estimate expected future-discounted rewards (the value function ) in Reinforcement
Revised: This is mostcommonly used to estimate expected future-discounted rewards (the value function ) in Reinforcement

Original: Learning (RL).
Revised: Learning (RL).

Original: Advances in RL allow us to use powerful function approximators, and also to usesampling strategies other than naively following the Markov process (MP).
Revised: Advances in RL allow us to use powerful function approximators, and also to usesampling strategies other than naively following the Markov process (MP).

Original: When TD, functionapproximation and off-policy training are all combined, learned functions exhibit severe instabilityand divergence, as classically observed by Williams and Baird III [15], Tsitsiklis and Van Roy[13].
Revised: When TD, functionapproximation and off-policy training are all combined, learned functions exhibit severe instabilityand divergence, as classically observed by Williams and Baird III [18], Tsitsiklis and Van Roy[15].

Original: This combination is known in the literature as the deadly triad [9, pg. 264], and while manycontemporary variants of TD are designed to converge despite the instability, the quality of thesolution at convergence may be arbitrarily poor.
Revised: This combination is known in the literature as the deadly triad [11, pg. 264], and while manycontemporary variants of TD are designed to converge despite the instability, the quality of thesolution at convergence may be arbitrarily poor.

Original: A common technique to avoid unbounded error is ℓ 2 regularization [12], i.e. penalizing the squarednorm of the weights in addition to the TD error.
Revised: A common technique to avoid unbounded error is ℓ 2 regularization [14], i.e. penalizing the squarednorm of the weights in addition to the TD error.

Original: This is generally understood to bound the worst-caseerror in exchange for biasing the model and potentially increasing the error everywhere else.
Revised: This is generally understood to bound the worst-caseerror in exchange for biasing the model and potentially increasing the error everywhere else.

Original: When used on three common examples of the deadly triad [5, 15, 9, pg.260], regularization appears tomitigate the worst aspects of the divergence in practice.
Revised: When used on three common examples of the deadly triad [6, 18, 11, pg.260], regularization appears tomitigate the worst aspects of the divergence in practice.

Original: Consequently, it has become an essentialassumption made by many RL algorithms [1,8, 10, 16, 17, 18, 7] and is seen as routine and innocuous.
Revised: Consequently, it has become an essential assumption made by many RL algorithms [1, 10, 12, 19, 20, 21, 8] and is seen as routine and innocuous.

Original: We argue that this perspective on regularization in off-policy TD is fundamentally mistaken.
Revised: We argue that this perspective on regularization in off-policy TD is fundamentally mistaken.

Original: Whileregularization is indeed reasonably well-behaved and innocuous in classic fully-supervised contexts,the use of bootstrapping in TD means that even small amounts of model bias induced by regularizationcan cause divergence.
Revised: Whileregularization is indeed reasonably well-behaved and innocuous in classic fully-supervised contexts,the use of bootstrapping in TD means that even small amounts of model bias induced by regularizationcan cause divergence.

Original: This is an oft-ignored phenomenon in the literature, and so we introducea series of new counterexamples (summarized in Table 1) to show how regularization can havecounterintuitive and destructive effects in TD.
Revised: This is an oft-ignored phenomenon in the literature, and so we introducea series of new counterexamples (summarized in Table 1) to show how regularization can havecounterintuitive and destructive effects in TD.

Original: We show that vacuous solutions and training instabilityare not solved by the use of regularization; that applying regularization can sometimes inducedivergence and increase worst-case error; and that Emphatic TD algorithms–which are the mostpromising solution to this divergence–can themselves diverge when regularized.
Revised: We show that vacuous solutions and training instabilityare not solved by the use of regularization; that applying regularization can sometimes inducedivergence and increase worst-case error; and that Emphatic TD algorithms–which are the mostpromising solution to this divergence–can themselves diverge when regularized.

Original: We finally alsoillustrate misbehaving regularization in the context of neural network value function approximation,demonstrating the general pitfalls of regularization possible in RL algorithms.
Revised: We finally alsoillustrate misbehaving regularization in the context of neural network value function approximation,demonstrating the general pitfalls of regularization possible in RL algorithms.

Original: Regularization needsto be treated cautiously in the context of RL, as it  differently than in supervised settings.
Revised: Regularization needsto be treated cautiously in the context of RL, as it behaves differently than in supervised settings.

Original: 1.1 Summary of our counterexamples

Original: Our counterexamples demonstrate these core ideas:
Revised: Our counterexamples demonstrate these core ideas:

Original: TD learning off-policy can be unstable and/or have unbounded error even when it converges
Revised: TD learning off-policy can be unstable and/or have unbounded error even when it converges.

Original: Following well-established methods we show there is some off-policy distribution under which TDwith linear value function approximation diverges and learns a model with unbounded error (evenif it were able to converge to the TD fixed point).
Revised: Following well-established methods we show there is some off-policy distribution under which TDwith linear value function approximation diverges and learns a model with unbounded error (evenif it were able to converge to the TD fixed point).

Original: This concisely demonstrates key features of thetraining error: the error is small when the distribution is close to on-policy, but the error divergesaround specific off-policy distributions.
Revised: This concisely demonstrates key features of thetraining error: the error is small when the distribution is close to on-policy, but the error divergesaround specific off-policy distributions.

Original: The intuition behind this, explained in Section 3, is that theoff-policy 1 TD update involves a projection operation that depends on the sampling distribution andcan be arbitrarily far away from the true value.
Revised: The intuition behind this, explained in Section 3, is that theoff-policy 1 TD update involves a projection operation that depends on the sampling distribution andcan be arbitrarily far away from the true value.

Original: This basic fact has already been established by pastwork [15, 5], but our example is based upon a particular simple three-state MP, drawn in Figure 1a.
Revised: This basic fact has already been established by pastwork [18, 6], but our example is based upon a particular simple three-state MP, drawn in Figure 1a.

Original: Regularization cannot always mitigate off-policy training error.
Revised: Regularization cannot always mitigate off-policy training error.

Original: We next introduce regularization into our setting, and show how it changes the relationship between training error and off-policytraining.
Revised: We next introduce regularization into our setting, and show how it changes the relationship between training error and off-policytraining.

Original: As explained in Section 2, we penalize the ℓ 2 -norm of learned (linear) weights with somecoefficient η ; as η increases, the learned weights approach zero.
Revised: As explained in Section 2, we penalize the ℓ 2 -norm of learned (linear) weights with somecoefficient η ; as η increases, the learned weights approach zero.

Original: However, in Example 1 , we showthat there exists an off-policy distribution such that for any η ≥ 0 < ∞ , the regularized TD fixedpoint attains strictly higher approximation error than the zero solution (i.e., the infinitely regularizedpoint).
Revised: However, in Example 1 , we showthat there exists an off-policy distribution such that for any η ≥ 0 < ∞ , the regularized TD fixedpoint attains strictly higher approximation error than the zero solution (i.e., the infinitely regularizedpoint).

Original: We call such examples vacuous .
Revised: We call such examples vacuous .

Revised: In other words, vacuous value functions never do better thanguessing zero for all states, for any amount of regularization .

Original: We further analyze this vacuous example in the context of the algorithm in [18].
Revised: We further analyze this vacuous example in the context of the algorithm in [21].

Original: In this work, theauthors assume the use of regularization to derive bounds on the learned error under off-policysampling.
Revised: In this work, theauthors assume the use of regularization to derive bounds on the learned error under off-policysampling.

Original: Although these bounds are “correct” in the case of our counterexample, they are alsovacuous, at about 2000 times the error of the zero model.
Revised: Although these bounds are technically correct in the case of our counterexample, they arevery loose, at about 2000 times the threshold of vacuity.

Original: This highlights the challenge of formallyrelying on regularization to bound model error.
Revised: This highlights the challenge of formallyrelying on regularization to bound model error.

Original: Small amounts of regularization can cause model divergence or large errors.
Revised: Small amounts of regularization can cause model divergence or large errors.

Original: There is ageneral implicit assumption in much ML literature that regularlization monotonically shrinks learnedweights.
Revised: There is ageneral implicit assumption in much ML literature that regularlization monotonically shrinks learnedweights.

Original: This intuition comes from classic fully-supervised machine learning where it typically holds.
Revised: This intuition comes from classic fully-supervised machine learning where it typically holds.

Original: But because TD bootstraps value estimates (i.e. learns values using its own output), it is possible forsmall amounts of bias to be arbitrarily magnified.
Revised: But because TD bootstraps value estimates (i.e. learns values using its own output), it is possible forsmall amounts of bias to be arbitrarily magnified.

Original: We dub this phenomenon “small-eta error” andillustrate it in Example 2 .
Revised: We dub this phenomenon “small-eta error” andillustrate it in Example 2 .

Original: We relate this to the presence of negative eigenvalues in an intermediatestep of the solution and show that, in some settings, the error of the TD solution may be relatively
Revised: We relate this to the presence of negative eigenvalues in an intermediatestep of the solution and show that, in some settings, the error of the TD solution may be relativelysmall when applied with no regularization but adding regularization causes the model to have worseerror than the zero solution.

Original: Example
Revised: Example

Original: There exist off-policy distributions under which TD learns a vacuous model (one which– despite any amount of regularization–never does better than guessing zeros).
Revised: There exist off-policy distributions under which TD learns a vacuous model (one which– despite any amount of regularization–never does better than guessing zeros).

Original: Example
Revised: Example

Original: Small values of the regularization parameter η can make TD diverge in models that otherwise converge.
Revised: Small values of the regularization parameter η can make TD diverge in models that otherwise converge.

Original: This is an unavoidable effect of bootstrapping in TD, and setting a lower-bound to exclude this may render models vacuous.
Revised: This is an unavoidable effect of bootstrapping in TD, and setting a lower-bound to exclude this may render models vacuous.

Original: Example
Revised: Example

Original: Emphatic-TD-inspired algorithms are a promising way to reweight samples and mitigate the effects of training off-policy.
Revised: Emphatic-TD-inspired algorithms are a promising way to reweight samples and mitigate the effects of training off-policy.

Original: But if this reweighting is learned using TD, then using regularization can bias the emphasis model and cause the value model itself to diverge.
Revised: But if this reweighting is learned using TD, then using regularization can bias the emphasis model and cause the value model itself to diverge.

Original: Example
Revised: Example

Original: Training instability and increased error due to the deadly triad also occur when neural networks are used.
Revised: Training instability and increased error due to the deadly triad also occur when neural networks are used.

Original: We construct an empirical example and draw qualitative comparisons.
Revised: We construct an empirical example and draw qualitative comparisons.

Original: small when applied with no regularization but adding regularization causes the model to have worseerror than the zero solution.
Revised: [Figure]

Original: One common solution tothis problem is tolower-bound η to guarantee that reguarlization behavesmonotonically.
Revised: to is to

Original: However, we furthershow that such a lower bound may occur after the point ofvacuity: a model that is not vacuous becomes vacuous for any regularization parameter above thislower bound. We also show that it isnot always possible to select a single η a priori , with examples ofmutually-incompatible off-policy distributions where there is no η that achieves better than vacuousor nearly-vacuous results at different distributions.
Revised: However further that it is

Original: Emphatic-TD-based algorithms are vulnerable to instability from regularization.
Revised: Emphatic-TD-based algorithms are vulnerable to instability from regularization.

Original: EmphaticTD
Revised: EmphaticTD

Original: [11] fundamentally solves the problem of training off-policy by resampling TD updates so theyappear to be on-policy.
Revised: [13] fundamentally solves the problem of training off-policy by resampling TD updates so theyappear to be on-policy.

Original: This technique requires an emphasis model that decides how to scale each TDupdate, and learning this has been the key challenge preventing widespread adoption of Emphatic-TD.
Revised: This technique requires an emphasis model that decides how to scale each TDupdate, and learning this has been the key challenge preventing widespread adoption of Emphatic-TD.

Original: A recent paper [17] proposed learning this emphasis model using “reversed” TD while simultaneouslylearning the value model using regular TD.
Revised: A recent paper [20] proposed learning this emphasis model using “reversed” TD while simultaneouslylearning the value model using regular TD.

Original: The resultant algorithm is called COF-PAC, and employsregularization to ensure that the two TD models eventually converge.
Revised: The resultant algorithm is called COF-PAC, and employsregularization to ensure that the two TD models eventually converge.

Original: We show that regularization, while necessary, can be harmful for such models in Example 3 .
Revised: We show that regularization, while necessary, can be harmful for such models in Example 3 .

Original: Specifically, we construct a model that converges to the correct solution without regularization but toan arbitrarily poor error when reguarlization is used.
Revised: Specifically, we construct a model that converges to the correct solution without regularization butto an arbitrarily poor solution when regularized.

Original: The intuition behind this is that regularizingthe emphasis model changes the effective distribution of the TD updates to the value model, whichcan cause the value model to have arbitrarily large error.
Revised: The intuition behind this is that regularizing theemphasis model changes the effective distribution of the TD updates to the value model, which cancause the value model to have arbitrarily large error.

Original: We complete the example by showing thatregularizing the value function separately does not restore performance.
Revised: We complete the example by showing thatregularizing the value function separately does not restore performance.

Original: Regularization can cause model divergence in neural networks.
Revised: Regularization can cause model divergence in neural networks.

Original: So far most analysis of thedeadly triad in the literature focuses on the linear case.
Revised: So far most analysis of thedeadly triad in the literature focuses on the linear case.

Original: We extend our example to a nine-state Markovchain (shown in Figure 1b), and show how the previously identified problems persist into the neuralnetwork case in Example 4 .
Revised: We extend our example to a nine-state Markovchain (shown in Figure 1b), and show how the previously identified problems persist into the neuralnetwork case in Example 4 .

Original: We show two key similarities: first, models trained at certain off-policydistributions may be vacuous.
Revised: We show two key similarities: first, models trained at certain off-policydistributions may be vacuous.

Original: Second, small amounts of regularization counterintuitively increaseerror.
Revised: Second, small amounts of regularization counterintuitively increaseerror.

Original: This illustrates Example 2 in the NN case.
Revised: This illustrates Example 2 in the NN case.

Original:  Preliminaries and Notation
Revised:  Preliminaries and Notation

Original: Consider the n -state Markov chain ( S , P, R, γ ) , with state space S , state-dependent reward R ∈ R n ,and discount factor γ ∈ [0 .. 1] .
Revised: Consider the n -state Markov chain ( S , P, R, γ ) , with state space S , state-dependent reward R : S → R , and discount factor γ ∈ [0 , 1] .

Original: P ∈ R n × n is the transition matrix, with P ij encoding the probabilityof moving from state i to j .
Revised: P ∈ R n × n is the transition matrix, with P ij encoding theprobability of moving from state i to j .

Original: We wish to estimate the value function V ∈ Rn , defined as the expecteddiscounted future reward of being in each state: V ( s ) ˙= E
Revised: We wish to estimate the value function V : S → R , definedas the expected discounted future reward of being in each state: V ( s ) ˙= E

Original: [ (cid:80) ∞ t =

Revised: [ (cid:80) ∞ t =

Original: γ t R ( s t ) | s 0 = s ] .
Revised: γ t R ( s t ) | s 0 = s ] .

Original: A key property isthat it follows the Bellman equation:
Revised: Akey property is that it follows the Bellman equation:

Original: [Equation]
Revised: [Equation]

Original: [Figure] Using linear function approximation to learn V , we assume a matrix of feature-vectors Φ ∈ R n × k that is fixed, and a vector of parameters w ∈ R
Revised: Using linear function approximation to learn V , we assume a matrix of feature-vectors Φ ∈ R n × k that is fixed, and a vector of parameters w ∈ R

Original: k
Revised: k

Original: that is learned.
Revised: that is learned.

Original: The Bellman equation is then:
Revised: The Bellman equation is then:

Original: [Equation]
Revised: [Equation]

Original: When w is learned with TD, this equation is only valid if the TD updates are on-policy (that is, they are distributed according to the steady-state probability of visiting each state, written as π ∈ R n ).
Revised: When w is learned with TD, this equation is only valid if the TD updates are on-policy (that is, they are distributed according to the steady-state probability of visiting each state, written as π ∈ R n ).

Original: In the general case, where TD updates follow a (possibly) different distribution µ ∈ R n 0 , the TD solution is a fixed point of the Bellman operator followed by a projection [5]:
Revised: In the general case, where TD updates follow a (possibly) different distribution µ ∈ R n 0 , the TD solution is a fixed point of the Bellman operator followed by a projection [6]:

Original: [Equation]
Revised: [Equation]

Original: (3) where the matrix Π µ = Φ(Φ ⊤ D Φ) − 1 Φ ⊤ D projects the Bellman backup onto the columnspace of Φ , reweighted by the diagonal matrix D = diag ( µ ) .
Revised: where the matrix Π µ = Φ(Φ ⊤ D Φ) − 1 Φ ⊤ D projects the Bellman backup onto the columnspace of Φ , reweighted by the diagonal matrix D = diag ( µ ) .

Original: This yields the closed-form solution:
Revised: This yields the closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: Where A = Φ ⊤ D ( I − γP )
Revised: Where A = Φ ⊤ D ( I − γP )

Original: Φ and ⃗b =
Revised: Φ and ⃗b =

Original: Φ ⊤ DR .
Revised: Φ ⊤ DR .

Original: When this solution is subject to ℓ 2 regularization, some non-negative η is added to ensure the matrix being inverted is positive definite:
Revised: When this solution is subject to ℓ 2 regularization, some non-negative η is added to ensure the matrix being inverted is positive definite:

Original: [Equation]
Revised: [Equation]

Original: As will be important later, we note that as η increases it drives w ∗ ( η ) towards zero.
Revised: As will be important later, we note that as η increases it drives w ∗ ( η ) towards zero.

Original:  Our Counterexamples
Revised:  Our Counterexamples

Original: When deadly triad conditions are present, TD may learn a value function with arbitrarily large erroreven if the true value function can be represented with low error.
Revised: When deadly triad conditions are present, TD may learn a value function with arbitrarily large erroreven if the true value function can be represented with low error.

Original: Consider the three-state MP in
Revised: Consider the three-state MP in

Original: Figure 1a, which we instantiate with the value function V =
Revised: Figure 1a, which we instantiate with the value function V =

Original: [1 , 2 . 2 , 1 . 05] ⊤ and discount factorγ = 0 .
Revised: [1 , 2 . 2 , 1 . 05] ⊤ and discount factorγ = 0 .

Original: The reward function is computed as R ← ( I − γP ) V .
Revised: The reward function is computed as R ← ( I − γP ) V .

Original: We choose a basis Φ with smallrepresentation error ∥
Revised: We choose a basis Φ with smallrepresentation error ∥

Original: Π D V − V ∥ ≤ ϵ :
Revised: Π µ V − V ∥ ≤ ϵ :

Original: [Equation]
Revised: [Equation]

Original:  ϵ > [Equation]
Revised: where ϵ >(6)

Original: We first consider the unregularized ( η = 0 ) case, closely following the derivation in [5].
Revised: We first consider the unregularized ( η = 0 ) case, closely following the derivation in [6].

Original: We wish toshow there is some sampling distribution µ such that error in the learned value function is unbounded.
Revised: We wish toshow there is some sampling distribution µ such that error in the learned value function is unbounded.

Original: To do this, we set µ = [0 .
Revised: To do this, we set µ = [0 .

Original: 56(1 − p ) , 0 .
Revised: 56(1 − p ) , 0 .

Original: 56 p, 0 . 44] , where p ∈ (0 , 1) .
Revised: 56 p, 0 . 44] , where p ∈ (0 , 1) .

Original: We set ϵ = 10 − 4 and find paround which A is ill-conditioned by solving det( A ) = 0 :
Revised: We set ϵ = 10 − 4 and find paround which A is ill-conditioned by solving det( A ) = 0 :

Original: [Equation]
Revised: [Equation]

Original: A − 1 (and consequently the error) can be made arbitrarily large by selecting p close to these values,which completes the introductory example.
Revised: A − 1 (and consequently the error) can be made arbitrarily large by selecting p close to these values,which completes the introductory example.

Original: Now we look at the behavior of TD under regularization,which is the main contribution of our paper:
Revised: Now we look at the behavior of TD under regularization,which is the main contribution of our paper.

Original:  Regularization cannot always mitigate off-policy training error.
Revised: 3.1 Regularization cannot always mitigate off-policy training error.

Original: There is a belief in the literature that regularization is a trade-off between reducing the blow-up ofasymptotic errors and accurately learning the value function everywhere else [1, 18].
Revised: There is a belief in the literature that regularization is a trade-off between reducing the blow-up ofasymptotic errors and accurately learning the value function everywhere else [1, 21].

Original: However, thisbelief does not accurately capture the nature of regularization: we show that it is possible to learnmodels that never perform better than always guessing zero despite any amount of regularization.
Revised: However, thisbelief does not accurately capture the nature of regularization: we show that it is possible to learnmodels that never perform better than always guessing zero despite any amount of regularization.

Original: That is, the TD error at all η is at least as much as the error as η → ∞ .
Revised: That is, the TD error at all η is at least as much as the error as η → ∞ .

Original: We call such models vacuous .
Revised: We call such models vacuous .

Original: Example 1.
Revised: Example 1.

Original: When TD is regularized, there may exist some off-policy distribution at which TD learnsa vacuous model.
Revised: When TD is regularized, there may exist some off-policy distribution at which TD learnsa vacuous model.

Original: In notation:
Revised: In notation:

Original: [Equation]
Revised: [Equation]

Original: Details.
Revised: Details.

Original: We use the same setting as in Section 3.
Revised: We use the same setting as in Section 3.

Original: A detailed derivation is provided in Appendix B.2.
Revised: A detailed derivation is provided in Appendix B.2.

Original: We begin by noting that we can easily find the solution ˆ w that minimizes the least-squares error∥ Φ ˆ w − V ∥ .
Revised: We begin by noting that we can easily find the solution ˆ w that minimizes the least-squares error∥ Φ ˆ w − V ∥ .

Original: If we consider this solution as a vector (as drawn in Figure 2a), we can immediately seethat there is an ℓ 2 -ball around ˆ w corresponding to the set of w ∗ ( η ) with no more than ∥ V ∥ error.
Revised: If we consider this solution as a vector (as drawn in Figure 2a), we can immediately seethat there is an ℓ 2 -ball around ˆ w corresponding to the set of w ∗ ( η ) with no more than ∥ V ∥ error.

Original: Similarly, we can trace the trajectory that the TD solution w ∗ ( η ) takes as η is increased from 0 to ∞ .
Revised: Similarly, we can trace the trajectory that the TD solution w ∗ ( η ) takes as η is increased from 0 to ∞ .

Original: We know that, as η → ∞ , w ∗ ( η ) is crushed to zero and so all trajectories must eventually terminate atthe origin.
Revised: We know that, as η → ∞ , w ∗ ( η ) is crushed to zero and so all trajectories must eventually terminate atthe origin.

Original: When regularized models are not vacuous, the trajectory intersects the non-vacuous-errorball.
Revised: When regularized models are not vacuous, the trajectory intersects the non-vacuous-errorball.

Original: We see this in trajectory (2), where the error briefly dips below ∥ V ∥ in Figure 2b.
Revised: We see this in trajectory 2, where the error briefly dips below ∥ V ∥ in Figure 2b.

Original: Intuitively, a sufficient condition for a solution to be vacuous is that it remains in the half-spacethat is tangent to and excludes the non-vacuous parameter ball.
Revised: Intuitively, a sufficient condition for a solution to be vacuous is that it remains in the half-spacethat is tangent to and excludes the non-vacuous parameter ball.

Original: This is equivalent to finding somedistribution µ such that ˆ w ⊤ w ∗ ( η ) ≤ 0 for all η , which we numerically solve to obtain the model intrajectory (1).
Revised: This is equivalent to finding somedistribution µ such that ˆ w ⊤ w ∗ ( η ) ≤ 0 for all η , which we numerically solve to obtain the model intrajectory 1.

Original: From Figure 2a we can see the trajectory remains in the half-space, and from Figure 2bwe can see that the error is never less than ∥ V ∥ .
Revised: From Figure 2a we can see the trajectory remains in the half-space, and from Figure 2bwe can see that the error is never less than ∥ V ∥ .

Original: Trajectory (1) is a vacuous example.
Revised: Trajectory 1 is a vacuous example.

Original: We observe that Example 1, because it remains entirely in the halfspace ˆ w ⊤ w ∗ ( η ) ≤ 0 , could easilybe generalized to other forms of regularization.
Revised: We observe that Example 1, because it remains entirely in the halfspace ˆ w ⊤ w ∗ ( η ) ≤ 0 , could easilybe generalized to other forms of regularization.

Original: We leave this for future work.
Revised: We leave this for future work.

Original: Breaking the Deadly Triad and our counterexample.
Revised: Breaking the Deadly Triad and our counterexample.

Original: In light of our example we examine thework of [18] in which the authors derive a bound for the regularized TD error under a novel doubleprojection update rule.
Revised: In light of our example we examine thework of [21] in which the authors derive a bound for the regularized TD error under a novel doubleprojection update rule.

Original: We apply our example to their bound and show that their method may produceloose bounds on TD solutions, and so doesn’t quite break the deadly triad:
Revised: We apply our example to their bound and show that their method may produceloose bounds on TD solutions, and so doesn’t quite break the deadly triad:

Original: [Equation]
Revised: [Equation]

Original: for ξ ∈ [0 .. 1] , where σ max and σ min denote the largest and smallest singular value respectively.
Revised: for ξ ∈ [0 , 1] , where σ max and σ min denote the largest and smallest singular value respectively.

Original: Theorem 2 from [18] bounds η , and therefore also b :
Revised: Theorem 2 from [21] bounds η , and therefore also b :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Their method bounds the error in our example by 2000 ∗ ∥ V ∥ , which is tremendously loose.
Revised: Their method bounds the error in our example by 2000 ∗ ∥ V ∥ , which is tremendously loose.

Original: (Weanalyze a different example in Appendix B.3, showing a still-loose but improved bound of 8 ∗ ∥ V ∥ .)
Revised: (Weanalyze a different example in Appendix B.3, showing a still-loose but improved bound of 8 ∗ ∥ V ∥ .)

Original: This illustrates the risk of relying on regularization to formally bound model error. 3.2 Small amounts of regularization can cause large increases in training error.
Revised: This illustrates the risk of relying on regularization to formally bound model error.3.2 Small amounts of regularization can cause large increases in training error.

Original: There is a general assumption in the literature that ℓ 2 regularization monotonically shrinks the learnedweights.
Revised: There is a general assumption in the literature that ℓ 2 regularization monotonically shrinks the learnedweights.

Original: While this is true in classification, regression, and other non-bootstrapping contexts, this isnot true in TD.
Revised: While this is true in classification, regression, and other non-bootstrapping contexts, this isnot true in TD.

Original: Because TD bootstraps values, it is possible for model bias to be arbitrarily magnified.
Revised: Because TD bootstraps values, it is possible for model bias to be arbitrarily magnified.

Original: This can be understood in terms of the eigenvalues of the matrix A in Equation 5.
Revised: This can be understood in terms of the eigenvalues of the matrix A in Equation 5.

Original: By increasingvalues along the diagonal, ℓ 2 regularization increases eigenvalues of the matrix ( A + ηI ) to ensure it
Revised: By increasingvalues along the diagonal, ℓ 2 regularization increases eigenvalues of the matrix ( A + ηI ) to ensure it

Original: ˆ w (1) (2) (3)
Revised: ˆ w (1) (2) (3)

Original: 10 −10 −10 −10 −1010 2 10 −101010∥ V ∥
Revised: 10 −10 −10 −10 −1010 2 10 −101010∥ V ∥

Original: ϵ (1) (2) (3)
Revised: ϵ (1) (2) (3)

Original: Regularization parameter η
Revised: Regularization parameter η

Original: E rr
Revised: E rr

Revised: o r a t c onv e r g e n ce comparison, we show trajectory 2 that is improved by regularization, and 3, which exhibits small- errors.

Original: o r a t c onv e r g e n ce is positive definite.
Revised: ( are stretched, so the errors in the is positive definite.

Original: Under off-policy distributions, it is possible for A to have eigenvalues that arenegative or zero.
Revised: Under off-policy distributions, it is possible for A to have eigenvalues that arenegative or zero.

Original: This implies that there are η for which det( A + ηI ) = 0 , and selecting η close tothese values allows us to achieve arbitrarily high error.
Revised: This implies that there are η for which det( A + ηI ) = 0 , and selecting η close tothese values allows us to achieve arbitrarily high error.

Original: We show one such case in Example 2.
Revised: We show one such case in Example 2.

Original: This isnot merely theoretical–we demonstrate this in the neural network case in Section 3.4.
Revised: This isnot merely theoretical–we demonstrate this in the neural network case in Section 3.4.

Original: Example 2.
Revised: Example 2.

Original: When TD is regularized, the model may diverge around (typically small) values of η .
Revised: When TD is regularized, the model may diverge around (typically small) values of η .

Original: Lower-bounding η , a common mitigation, can make well-behaved models vacuous.
Revised: Lower-bounding η , a common mitigation, can make well-behaved models vacuous.

Original: It is not alwayspossible to select a single value of η that makes models vacuous at different sampling distributions.
Revised: It is not alwayspossible to select a single value of η that makes models vacuous at different sampling distributions.

Original: Details.
Revised: Details.

Original: Using our three-state example, we set µ = [0 . 05 , 0 . 05 , 0 . 9] and solve for det( A + ηI ) {+= 
+}
Revised: Using our three-state example, we set µ = [0 . 05 , 0 . 05 , 0 . 9] and solve for det( A + ηI ) {+= 
+}

Original: [Equation]
Revised: :0 =

Original: [Equation]
Revised: [Equation]

Original: As in the introductory example, the error can be made arbitrarily large by setting η ≈ 0 . 0634 .
Revised: As in the introductory example, the error can be made arbitrarily large by setting η ≈ 0 . 0634 .

Original: This small- η divergence effect can appear in several ways, illustrated in Figure 3a.
Revised: This small- η divergence effect can appear in several ways, illustrated in Figure 3a.

Original: Typically, thisappears as one or more points at which TD error diverges before the region at which regularizationreduces the model error below ∥ V ∥ .
Revised: Typically, thisappears as one or more points at which TD error diverges before the region at which regularizationreduces the model error below ∥ V ∥ .

Original: The first and second plot in Figure 3a show two such cases,where the error increases sharply at two and one points respectively.
Revised: The first and second plot in Figure 3a show two such cases,where the error increases sharply at two and one points respectively.

Original: In the literature, it is commonly assumed that A is “nearly” positive definite, where only a feweigenvalues are non-positive, and those are close to zero.
Revised: In the literature, it is commonly assumed that A is “nearly” positive definite, where only a feweigenvalues are non-positive, and those are close to zero.

Original: This gives rise to the common mitigationof setting a lower-bound η 0 such that ( A + ηI ) is positive definite for η > η 0 .
Revised: This gives rise to the common mitigationof setting a lower-bound η 0 such that ( A + ηI ) is positive definite for η > η 0 .

Original: This may render anotherwise well-behaved model vacuous.
Revised: This may render anotherwise well-behaved model vacuous.

Original: The third plot in Figure 3a illustrates this: the model is notvacuous when unregularized, but is vacuous in the domain η > 10 − 2 where divergence is prohibited.
Revised: The third plot in Figure 3a illustrates this: the model is notvacuous when unregularized, but is vacuous in the domain η > 10 − 2 where divergence is prohibited.

Original: A common practice in the literature is to set η before training, without regard for the samplingdistribution.
Revised: A common practice in the literature is to set η before training, without regard for the samplingdistribution.

Original: This is ill advised, as the value may be under- or over-regularizing depending onthe sampling distribution.
Revised: This is ill advised, as the value may be under- or over-regularizing depending onthe sampling distribution.

Original: One such example is illustrated in Figure 3b, where selecting an η thatminimizes the error for one distribution will lead to vacuous or nearly-vacuous results in the othertwo.
Revised: One such example is illustrated in Figure 3b, where selecting an η thatminimizes the error for one distribution will lead to vacuous or nearly-vacuous results in the other two.

Original: A second example in Figure 2b has no single η for which trajectories (2) and (3) are bothnon-vacuous.
Revised: A second example in Figure 2b has no single η for which trajectories 2 and 3 are both non-vacuous.

Original: This is especially relevant as regularization is commonly used to permit distributiondrift during training, as discussed in Section 4.
Revised: This is especially relevant as regularization is commonly used to permit distribution drift duringtraining, as discussed in Section 4.

Original: If the training distribution changes while η is fixed,then algorithms that can be proven to converge to good solutions under some original distributionmay converge to poor solutions as the distribution drifts.
Revised: If the training distribution changes while η is fixed, then algorithmsthat can be proven to converge to good solutions under some original distribution may converge topoor solutions as the distribution drifts.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Emphasis Model Error
Revised: Emphasis Model Error

Original: [Figure]
Revised: [Figure]

Original: Specific values of η m induce the value function to diverge (Figure 4b).
Revised: Specific values of η m induce the value function to diverge (Figure 4b).

Original: The resultant value function is vacuous (Figure 4c).
Revised: The resultant value function is vacuous (Figure 4c).

Original: Under COF-PAC, regularization can greatly increase model error.
Revised: Under COF-PAC, regularization can greatly increase model error.

Original: 3.3 Emphatic approaches and our counterexample
Revised: 3 Emphatic approaches and our counterexample

Original: Emphatic-TD eliminates instability from off-policy sampling by reweighing incoming data (via animportance function)
Revised: Emphatic-TD eliminates instability from off-policy sampling by reweighting incoming data (via animportance function)

Original: so it appears to be on-policy.
Revised: so it appears to be on-policy.

Original: There is considerable interest in making thismore practical, especially by learning the importance and value models simultaneously.
Revised: There is considerable interest in making thismore practical, especially by learning the importance and value models simultaneously.

Original: A leadingexample of this work is COF-PAC [17], which uses ℓ 2 -regularized versions of GTD2 [10] to learnboth the value and emphasis models.
Revised: A leadingexample of this work is COF-PAC [20], which uses ℓ 2 -regularized versions of GTD2 [12] to learnboth the value and emphasis models.

Original: The authors rely on regularization, particularly because thetarget policy changes during learning.
Revised: The authors rely on regularization, particularly because thetarget policy changes during learning.

Original: This makes COF-PAC vulnerable to regularization-causederror.
Revised: This makes COF-PAC vulnerable to regularization-causederror.

Original: We illustrate this with Example 3 in which COF-PAC learns correctly when unregularized, buthas large error when regularized.
Revised: We illustrate this with Example 3 in which COF-PAC learns correctly when unregularized, buthas large error when regularized.

Original: (Mathematical details are deferred to Appendix B.5.)
Revised: (Mathematical details are deferred to Appendix B.5.)

Original: Example 3.
Revised: Example 3.

Original: COF-PAC may learn the value function with low error when unregularized, but witharbitrarily high error when regularized.
Revised: COF-PAC may learn the value function with low error when unregularized, but witharbitrarily high error when regularized.

Original: Details.
Revised: Details.

Original: Conceptually, COF-PAC maintains two separate models that are each updated by TD: theemphasis and the value models.
Revised: Conceptually, COF-PAC maintains two separate models that are each updated by TD: theemphasis and the value models.

Original: This emphasis model is used to reweigh TD updates to the valuefunction so they appear to come from the on-policy distribution.
Revised: This emphasis model is used to reweight TD updates to the valuefunction so they appear to come from the on-policy distribution.

Original: Our strategy is to first show howregularization biases the emphasis model, and then how this bias causes the value model to diverge.
Revised: Our strategy is to first show howregularization biases the emphasis model, and then how this bias causes the value model to diverge.

Original: We begin with our three-state MP, noting its on-policy distribution is π = [ . 25 .
Revised: We begin with our three-state MP, noting its on-policy distribution is π = [ . 25 .

Original: We wish tolearn the values using COF-PAC while sampling off-policy at µ = [ . 2 . 2 . 6] .
Revised: We wish tolearn the values using COF-PAC while sampling off-policy at µ = [ . 2 . 2 . 6] .

Original: [Figure]
Revised: [Figure]

Original: value  case.
Revised: value problems persist case.

Original: Now we introduce a key conceptual tool: υ ( η m ) , which is the effective distribution seen by the
Revised: Now we introduce a key conceptual tool: υ ( η m ) , which is the effective distribution seen by the

Original: TD-updates, influenced by the emphasis regularization parameter η m .
Revised: TD-updates, influenced by the emphasis regularization parameter η m .

Original: Unregularized, COF-PACis able to resample off-policy updates to the on-policy distribution: υ (0) ≡ π .
Revised: Unregularized, COF-PACis able to resample off-policy updates to the on-policy distribution: υ (0) ≡ π .

Original: If the model isregularized, then the effective distribution moves away from π .
Revised: If the model isregularized, then the effective distribution moves away from π .

Original: Figure 4a illustrates the distancebetween υ ( η m ) and π as the regularlization parameter increases.
Revised: Figure 4a illustrates the distancebetween υ ( η m ) and π as the regularlization parameter increases.

Original: We can use the effective distribution to compute the error in the value model.
Revised: We can use the effective distribution to compute the error in the value model.

Original: Plotting the relationshipbetween the value function error and η m in Figure 4b, we can see the value function has asymptoticerror around η m = 2 × 10 − 4 .
Revised: Plotting the relationshipbetween the value function error and η m in Figure 4b, we can see the value function has asymptoticerror around η m = 2 × 10 − 4 .

Original: This shows how COF-PAC may diverge with specific regularization.
Revised: This shows how COF-PAC may diverge with specific regularization.

Original: COF-PAC also allows for the value function to be separately regularized with parameter η v .
Revised: COF-PAC also allows for the value function to be separately regularized with parameter η v .

Original: We showthe effect of this in Figure 4c, where the value function never does much better than ∥ V ∥ making it(nearly) vacuous.
Revised: We showthe effect of this in Figure 4c, where the value function never does much better than ∥ V ∥ making it(nearly) vacuous.

Original: We can conclude that regularizing the emphasis model may cause the value modelto diverge, and this cannot be fixed by regularizing the value function separately.
Revised: We can conclude that regularizing the emphasis model may cause the value modelto diverge, and this cannot be fixed by regularizing the value function separately.

Original: COF-PAC makes the strong assumption that Kolter’s relaxed-contraction condition [5, eqn.
Revised: COF-PAC makes the strong assumption that Kolter’s relaxed-contraction condition [6, eqn.

Original: 10] holdsin both the emphasis and value models [17, asm. 4].
Revised: 10] holdsin both the emphasis and value models [20, asm. 4].

Original: We discuss this in Appendix B.5.1.
Revised: We discuss this in Appendix B.5.1.

Original: Applied to multi-layer networks
Revised: Applied to multi-layer networks

Original: We use a 9-state variant of our example to study the deadly triad in multi-layer neural networks(NNs).
Revised: We use a 9-state variant of our example to study the deadly triad in multi-layer neural networks(NNs).

Original: A deterministic observation function is chosen so we can control the degree of functionapproximation.
Revised: A deterministic observation function is chosen so we can control the degree of functionapproximation.

Original: We train a simple two-layer neural network with 3 neurons in the hidden layer.
Revised: We train a simple two-layer neural network with 3 neurons in the hidden layer.

Original: Thevalue function is assigned pseudo-randomly in range [ − 1 , 1] .
Revised: Thevalue function is assigned pseudo-randomly in range [ − 1 , 1] .

Original: (See Appendix C for details.)
Revised: (See Appendix C for details.)

Original: Example 4.
Revised: Example 4.

Original: Vacuous models and small-eta error also occur in neural network conditions.
Revised: Vacuous models and small- η error also occur in neural network conditions.

Original: Details.
Revised: Details.

Original: Werandomly initialize 100 models, and train them to convergence using simple semigradient TD updates under a fixed learning rate and without a target network.
Revised: We train 100 models using simple semi-gradient TD updates under a fixed learning rate.

Original: We plot the mean andthe 10 th – 90 th percentile range in Figure 5a, with and without regularization.
Revised: Weplot the mean and the 10 th – 90 th percentile range in Figure 5a, with and without regularization.

Original: TD is known to exhibithigh variance, and regularization is the traditional remedy for that.
Revised: TD isknown to exhibit high variance, and regularization is the traditional remedy for that.

Original: We corroborate this by notingthat the performance of the unregularized model varies widely, but regularization leads to similarperformance across initializations at the cost of increased error.
Revised: We corroboratethis by noting that the performance of the unregularized model varies widely, but regularization leadsto similar performance across initializations at the cost of increased error.

Original: First, we show that vacuous models may exist in the neural network case.
Revised: First, we show that vacuous models may exist in the neural network case.

Original: In Figure 5a, note howthere are some off-policy distributions under which both the regularized and unregularized modelsperform worse than the threshold of vacuity.
Revised: In Figure 5a, note howthere are some off-policy distributions under which both the regularized and unregularized modelsperform worse than the threshold of vacuity.

Original: We can numerically verify that there are some off-policydistributions that are vacuous.
Revised: We can numerically verify that vacuous models exist.

Original: Second, we show the small- η error problem in the neural network casein Figure 5b, where we plot the TD error against η at a fixed off-policy distribution.
Revised: Second, we show the small- η error problem in the neural network case in Figure 5b, where we plotthe TD error against η at a fixed off-policy distribution.

Original: We observe thataround η ≈ 10 − 3 the TD Error unexpectedly increases before decreasing, which clearly illustratesthis phenomenon.
Revised: We observe that around η ≈ 10 − 3 the TD Error unexpectedly increases before decreasing, which clearly illustrates this phenomenon.

Original: These qualitative links show a clear connection between the neural network case and the linear case,and highlights the importance of correctly handling off-policy sampling.
Revised: These qualitative links show a clear connection between the neural network case and the linear case,and highlights the importance of correctly handling off-policy sampling.

Original: Understanding the dynamicsof off-policy training will be useful for informing training strategies of practical RL algorithms, andis a key part of the future of the field.

Original:  Related Work
Revised:  Related Work

Original: Prior to this work, three examples  are common in the literature: the classic Tsitsiklis and Van Roy( w, 2 w ) example [9, p. 260], Kolter’s example [5], and Baird’s counterexample which shows howtraining instability can exist despite overparameterization [15]. All these examples can be shown tohave bounded error under regularization.ℓ 2 regularization isparticularly common when proving that some training method converges despitea changing sampling policy.
Revised: Three examples of the deadly triad are common in the literature: the classic Tsitsiklis and Van Roy( w, 2 w ) example [11, p. 260], Kolter’s example [6], and Baird’s counterexample which shows howtraining instability can exist despite overparameterization [18].ℓ 2 regularization is common when proving that an algorithm converges under a changing samplingpolicy.

Original: This is seen in GTD (analyzed in [16]), GTD2 [10], RO-TD [8], and COF-PAC [17].
Revised: This is seen in GTD (analyzed in [19]), GTD2 [12], RO-TD [10], and COF-PAC [20].

Original: This assumption may also be used to ensure convergence when training with a targetnetwork [18].
Revised: Thisassumption may also be used to ensure convergence when training with a target network [21].

Original: Despite the prevalence of regularization, the induced bias from using it is not wellstudied in the literature.
Revised: Despitethe prevalence of regularization, the induced bias from using it is not well studied.

Original: It is sometimes even dismissed as a mere technical assumption, as in [1].
Revised: It is often dismissedas a mere technical assumption, as in [1].

Original: A key conclusion of this work contradicts that; using regularization for convergence proofs inducesbias that may hinder performance.
Revised: We contradict that: using regularization for convergenceproofs may induce catastrophic bias.

Original: By showing concrete examples, this work hopes to inspire furtherinvestigation into regularization-induced bias in the same vein as [16].
Revised: By showing concrete examples, this work hopes to inspirefurther investigation into regularization-induced bias in the same vein as [19].

Original: Alternatives to regularization and TD We focus on ℓ 2 regularization in this paper, which penalizesthe ℓ 2 -norm of the learned weights; it is also possible to use ℓ 1 regularization with a proximaloperator/saddle point formulation as in [8], or any convex regularization term under a fixed targetpolicy [16].
Revised: Alternatives to regularization and TD We focus on ℓ 2 regularization in this paper, which penalizesthe ℓ 2 -norm of the learned weights; it is also possible to use ℓ 1 regularization with a proximaloperator/saddle point formulation as in [10], or any convex regularization term under a fixed targetpolicy [19].

Original: Instead of directly regularizing the weights, COP-TD uses a discounted update [3].
Revised: Instead of directly regularizing the weights, COP-TD uses a discounted update [4].

Original: DisCor [6] propagates bounds on Q-value estimates to quickly converge TD learning in the face oflarge bootstrapping error; it is not clear if DisCor can overcome off-policy sampling.
Revised: DisCor [7] propagates bounds on Q-value estimates to quickly converge TD learning in the face oflarge bootstrapping error; it is not clear if DisCor can overcome off-policy sampling.

Original: A separateprimal-dual saddle point method has also been adapted to ℓ 2 regularization [2] and is known toconverge under deadly triad conditions but error bounds at convergence are not yet known.
Revised: A separateprimal-dual saddle point method has also been adapted to ℓ 2 regularization [2] and is known toconverge under deadly triad conditions, and recent work [17] has derived error bounds with improvedscaling properties in the linear setting, offering a promising line of research.

Original: Emphatic-TD [11] fixes the fundamental problem in off-policy TD by reweighing updates so theyappear on-policy.
Revised: Emphatic-TD [13] fixes the fundamental problem in off-policy TD by reweighting updates so theyappear on-policy.

Original: The core idea underlying these techniques is to estimate the “followon trace” foreach state, the (weighted, λ - and γ − discounted) probability mass of all states whose value estimatesit influences.
Revised: The core idea underlying these techniques is to estimate the “followon trace” foreach state, the (weighted, λ - and γ − discounted) probability mass of all states whose value estimatesit influences.

Original: This trace is then used to estimate the emphasis, which is the reweighting factor for eachupdate.
Revised: This trace is then used to estimate the emphasis, which is the reweighting factor for eachupdate.

Original: While this family of methods is provably optimal in expectation, it is subject to tremendousvariance in theory and practice, especially when the importance is estimated using Monte-Carlosampling. 

Revised: While this family of methods is provably optimal in expectation, it is subject to tremendousvariance in theory and practice, especially when the importance is estimated using Monte-Carlosampling.

Original: In practice, these methods learn the follow-on trace using TD [4, 17] or similar [14],which makes them vulnerable to bias induced by the use of regularization.
Revised: In practice, these methods learn the follow-on trace using TD [5, 20] or similar [16],which makes them vulnerable to bias induced by the use of regularization.

Original:  Conclusion
Revised:  Conclusion

Original: There is a tremendous focus in the RL literature on proving convergence of novel algorithms, butnot on the error at convergence.
Revised: There is a tremendous focus in the RL literature on proving convergence of novel algorithms, butnot on the error at convergence.

Original: Papers like [18] are laudable because they provide error bounds;even if the current bounds are loose, future work will no doubt tighten them.
Revised: Papers like [21] are laudable because they provide error bounds;even if the current bounds are loose, future work will no doubt tighten them.

Original: In this work, we showthat the popular technique of ℓ 2 regularization does not always prevent singularities and could evenintroduce catastrophic divergence.
Revised: In this work, we showthat the popular technique of ℓ 2 regularization does not always prevent singularities and could evenintroduce catastrophic divergence.

Original: We show this with a new counterexample that elegantly illustratesthe problems with learning off-policy and how it persists into the NN case.
Revised: We show this with a new counterexample that elegantly illustratesthe problems with learning off-policy and how it persists into the NN case.

Revised: Even though regularization can catastrophically fail in the ways we illustrate, it remains a reasonablemethod that may offer a fair tradeoff – as long as we are careful to check that we are not running afoulof the failure modes we explain in the paper.

Revised: It may be possible to design an adaptive regularizationscheme that can avoid pathological behavior, or even trial-and-error.

Original: Emphatic-TD is perhaps the most promising area of research for mitigating off-policy TD-learning.
Revised: Emphatic-TD is perhaps the most promising area of research for mitigating off-policy TD-learning.

Original: The key problem preventing its widespread adoption is the difficulty in estimating the emphasisfunction, but future work in this area may be able to overcome this.
Revised: The key problem preventing its widespread adoption is the difficulty in estimating the emphasisfunction, but future work in this area may be able to overcome this.

Original: Our example shows the risk ofrelying on regularization in practical implementations of such methods.
Revised: Our example shows the risk ofrelying on regularization in practical implementations of such methods.

Original: It is absolutely critical that
Revised: It is absolutely critical that

Original: Emphatic algorithms correctly manage regularization to avoid the risks that we highlight in this paper.
Revised: Emphatic algorithms correctly manage regularization to avoid the risks that we highlight in this paper.

Original: B Example Details
Revised: B Example Details

Original: We provide a more detailed explanation of our examples.
Revised: We provide a more detailed explanation of our examples.

Original: B.1 “Vacuous” models
Revised: B.1 “Vacuous” models

Original: Without ℓ 2 regularization, our linear modelto fails with asymptotic error.
Revised: Without ℓ 2 regularization, our linear model fails with asymptotic error.

Original: As this penalizes theℓ 2 -norm of the learned weights, this removes the asymptote and so we can no longer use the existenceof an asymptote as evidence of failure.
Revised: As this penalizes the ℓ 2 -normof the learned weights, this removes the asymptote and so we can no longer use the existence of anasymptote as evidence of failure.

Original: Instead, we propose a different definition of failure by notingthat, in the limiting case, regularization drives the learned weights to zero ( lim η →∞ w ∗ ( η ) = ⃗ 0 ).
Revised: Instead, we propose a different definition of failure by noting that,in the limiting case, regularization drives the learned weights to zero ( lim η →∞ w ∗ ( η ) = ⃗ 0 ).

Original: Thelearned value function Φ · ⃗ 

Revised: Thelearned value function Φ · ⃗

Original: = ⃗ 0 has no information about the true value function.
Revised: = ⃗ 0 has no information about the true value function.

Original: We argue that ifthe error with any η ∈ R + is never better than this case then the model is vacuous and hence adoptthe threshold error of ∥ Φ · ⃗ 0 − V ∥
Revised: We argue that ifthe error with any η ∈ R + is never better than this case then the model is vacuous and hence adoptthe threshold error of ∥ Φ · ⃗ 0 − V ∥

Original: =
Revised: =

Original: ∥ V ∥ to call a model vacuous.
Revised: ∥ V ∥ to call a model vacuous.

Original: This explains the failure conditionin Equation 8.
Revised: This explains the failure conditionin Equation 8.

Original: B.2 Details of regularization example
Revised: B.2 Details of regularization example

Original: This provides numeric details for Example 1.
Revised: This provides numeric details for Example 1.

Original: Example 5.
Revised: Example 5.

Original: When TD is regularized, there may exist some off-policy distribution at which TD learnsa vacuous model.
Revised: When TD is regularized, there may exist some off-policy distribution at which TD learnsa vacuous model.

Original: In notation:
Revised: In notation:

Original: [Equation]
Revised: [Equation]

Original: Details.
Revised: Details.

Original: We use the same setting as in Section 3.
Revised: We use the same setting as in Section 3.

Original: We observe that ˆ w =
Revised: We observe that ˆ w =

Original: [1 , − 1]
Revised: [1 , − 1]

Original: ⊤ minimizes theleast-squares error ∥
Revised: ⊤ minimizes theleast-squares error ∥

Original: Φ ˆ w − V ∥ , and further observe that a sufficient (but not necessary) condition for a solution to be vacuous is that ˆ w ⊤ w ∗ ( η ) ≤ 0 .
Revised: Φ ˆ w − V ∥ , and further observe that a sufficient (but not necessary) condition for a solution to be vacuous is that ˆ w ⊤ w ∗ ( η ) ≤ 0 .

Original: Solving:
Revised: Solving:

Original: [Equation]
Revised: [Equation]

Original: We verify that TD is vacuous at p = 0 . 102636 by computing the TD error at convergence:
Revised: We verify that TD is vacuous at p = 0 . 102636 by computing the TD error at convergence:

Original: [Equation]
Revised: [Equation]

Original: Since the fraction term in Equation 15 is obviously improper, we can conclude that our example willalways have at least ∥ V ∥ error over all η , and is therefore vacuous.
Revised: Since the fraction term in Equation 15 is obviously improper, we can conclude that our example willalways have at least ∥ V ∥ error over all η , and is therefore vacuous.

Original: We note that the error is not defined at η = 0 because this corresponds to a model divergence similarto our introductory example.
Revised: We note that the error is not defined at η = 0 because this corresponds to a model divergence similarto our introductory example.

Original: In practice, the TD fixed point will still converge to a vacuous solution:
Revised: In practice, the TD fixed point will still converge to a vacuous solution:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B.3 Additional simple regularization example
Revised: B.3 Additional simple regularization example

Original: We present a second example where the error is stationary with respect to the regularization parameter.
Revised: We present a second example where the error is stationary with respect to the regularization parameter.

Original: This is worse than Example 5 because we are able to show that the point the model converges to isindependent of regularization.
Revised: This is worse than Example 5 because we are able to show that the point the model converges to isindependent of regularization.

Original: Example 6.
Revised: Example 6.

Original: When TD is regularized, there may exist some off-policy distribution at which the TDfixed point is independent of the regularization parameter.
Revised: When TD is regularized, there may exist some off-policy distribution at which the TDfixed point is independent of the regularization parameter.

Original: Details.
Revised: Details.

Original: We use the same setting as in Section 3, except the value function is V =
Revised: We use the same setting as in Section 3, except the value function is V =

Original: ⊤ andbasis Φ selected to have small representation error ∥
Revised: ⊤ andbasis Φ selected to have small representation error ∥

Original: Π D V − V ∥
Revised: Π D V − V ∥

Original: ≤ ϵ :Φ = (cid:34)   
Revised: ≤ ϵ :Φ = (cid:34)   −

Original: [Equation]
Revised: [Equation]

Original: where ϵ >
Revised: where ϵ >

Original: [Equation]
Revised: [Equation]

Original: We set ϵ = 10 − 4 and write down w ∗ ( η ) in terms of g , a scalar function of η and p :
Revised: We set ϵ = 10 − 4 and write down w ∗ ( η ) in terms of g , a scalar function of η and p :

Original: [Equation]
Revised: [Equation]

Original: When g ( p, η ) ≤ 0 , the TD solution is vacuous.
Revised: When g ( p, η ) ≤ 0 , the TD solution is vacuous.

Original: We show that directly:
Revised: We show that directly:

Original: [Equation]
Revised: [Equation]

Original: When g ( p, η ) ≤ 0 , then ∥ g ( p, η ) − 1 ∥ ≥ 1 for all η and the TD solution is vacuous.
Revised: When g ( p, η ) ≤ 0 , then ∥ g ( p, η ) − 1 ∥ ≥ 1 for all η and the TD solution is vacuous.

Original: We findsuch a solution by noting the numerator has two roots in p , one of which corresponds to a vacuoussolution: g (0 . 715083 , η ) = 

Revised: We findsuch a solution by noting the numerator has two roots in p , one of which corresponds to a vacuoussolution: g (0 . 715083 , η ) =

Original: ( ∀ η ) , and this completes the example!
Revised: ( ∀ η ) , and this completes the example!

Original: In this setting, when TD updatesfollow the sampling distribution p ≈ 0 . 715083 , the error of the model at convergence is always ∥ V ∥regardless of regularization.
Revised: In this setting, when TD updatesfollow the sampling distribution p ≈ 0 . 715083 , the error of the model at convergence is always ∥ V ∥regardless of regularization.

Original: Our example converges to the same vacuous value regardless η .
Revised: Our example converges to the same vacuous value regardless η .

Original: In Figure 6, we can see that the TD error intersects the η → ∞ line immediately before and afterthe singularity.
Revised: In Figure 6, we can see that the TD error intersects the η → ∞ line immediately before and afterthe singularity.

Original: Our counterexample corresponds to the second root (that is, the intersection pointat higher p .)
Revised: Our counterexample corresponds to the second root (that is, the intersection pointat higher p .)

Original: This is because that corresponds to the stationary point between the asymptote that iscrushed and the error on the right that increases.
Revised: This is because that corresponds to the stationary point between the asymptote that iscrushed and the error on the right that increases.

Original: If our simpler derivation proved unsatisfying, we can also derive this counterexample using this fact:
Revised: If our simpler derivation proved unsatisfying, wecan also derive this counterexample using this fact:

Original: [Equation]
Revised: [Equation]

Original: From this, we can easily see that the counterexample is at p = 0 .
Revised: From this, we can easily see that the counterexample is at p = 0 .

Original: And this completes theexample!
Revised: And this completes theexample!

Original: We have discovered some p at which the TD error is always at least ∥ V ∥ , regardless ofregularization, and so our example learns a vacuous value function.
Revised: We have discovered some p at which the TD error is always at least ∥ V ∥ , regardless ofregularization, and so our example learns a vacuous value function.

Original: η → ∞ η =p = 0 .
Revised: η → ∞ η =p = 0 .

Original: p in sampling distribution µ = [ p / 2 , p / 2 , 1 − p ]
Revised: p in sampling distribution µ = [ p / 2 , p / 2 , 1 − p ]

Original: [Figure]
Revised: [Figure]

Original: B.3.1 Breaking the Deadly Triad and our counterexample.
Revised: B.3.1 Breaking the Deadly Triad and our counterexample.

Original: In light of our counterexample we examine the work of [18] in which the authors derive a bound forthe regularized TD error under a novel double-projection update rule.
Revised: In light of our counterexample we examine the work of [21] in which the authors derive a bound forthe regularized TD error under a novel double-projection update rule.

Original: We apply our example to theirbound b to show that their method permits vacuous TD solutions and doesn’t quite break the deadlytriad.
Revised: We apply our example to theirbound b to show that their method permits vacuous TD solutions and doesn’t quite break the deadlytriad.

Original: Starting from Equation 9:
Revised: Starting from Equation 9:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (22) for ξ ∈ [0 .. 1] , where σ max and σ min denote the largest and smallest singular value respectively.
Revised: (22) for ξ ∈ [0 , 1] , where σ max and σ min denote the largest and smallest singular value respectively.

Original: Theorem 2 from [18] bounds η , and therefore also b :
Revised: Theorem 2 from [21] bounds η , and therefore also b :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Under our example, their method bounds the error at no more than 7 .
Revised: Under our example, their method bounds the error at no more than 7 .

Original: 86 ∗ ∥ V ∥ , which is a veryloose bound that permits vacuous solutions.
Revised: 86 ∗ ∥ V ∥ , which is a veryloose bound that permits vacuous solutions.

Original: This illustrates the risk of trying to regularize awaysingularities, particularly in theoretical work.
Revised: This illustrates the risk of trying to regularize awaysingularities, particularly in theoretical work.

Original: Investigating the cause of the loose bounds reveals that the presence of σ min ( D ) 2 . 5 in 9 is largelyresponsible.
Revised: Investigating the cause of the loose bounds reveals that the presence of σ min ( D ) 2 . 5 in 9 is largelyresponsible.

Original: As D is a diagonal matrix encoding the sampling distribution, σ min ( D ) is the smallestsampling rate of any state, and so the bound must be at least η ξn 2 .
Revised: As D is a diagonal matrix encoding the sampling distribution, σ min ( D ) is the smallestsampling rate of any state, and so the bound must be at least η ξn 2 .

Original: 5 for any perfectly representablen -state MP.
Revised: 5 for any perfectly representablen -state MP.

Original: Unfortunately, this appears to be fundamental limit caused by finding a linear bound toan error that scales non-linearly, and following their derivation in the appendix does not readily admita way to improve this.
Revised: Unfortunately, this appears to be fundamental limit caused by finding a linear bound toan error that scales non-linearly, and following their derivation in the appendix does not readily admita way to improve this.

Original: B.4 Small-Eta Error
Revised: B.4 Small-Eta Error

Original: Our simplified example allows us to show this easily.
Revised: Our simplified example allows us to show this easily.

Original: Example 7.
Revised: Example 7.

Original: When TD is regularized, the model may diverge around (typically small) values of η .
Revised: When TD is regularized, the model may diverge around (typically small) values of η .

Original: η =η = 2 · 10 −
Revised: η =η = 2 · 10 −

Original: With Regul.
Revised: With Regul.

Original: distrib.
Revised: distrib.

Original: param.
Revised: param.

Original: p d i s t r i b .
Revised: p d i s t r i b .

Original: e rr .
Revised: e rr .

Original: ∥ υ ( η ) − π ∥
Revised: ∥ υ ( η ) − π ∥

Original: Emphasis Model Error
Revised: Emphasis Model Error

Original: [Figure]
Revised: [Figure]

Original: η =υ ( 0 ) υ (·− 4 )
Revised: η =υ ( 0 ) υ (·− 4 )

Original: With Regul.
Revised: With Regul.

Original: distrib.
Revised: distrib.

Original: param.
Revised: param.

Original: q v a l u e e rr .
Revised: q v a l u e e rr .

Original: ∥
Revised: ∥

Original: Φ v w ∗ v − V ∥
Revised: Φ v w ∗ v − V ∥

Original: Value Model Error
Revised: Value Model Error

Original: [Figure]
Revised: [Figure]

Original: Details.
Revised: Details.

Original: We set p = 0 .
Revised: We set p = 0 .

Original: 9 and solve for det( A + ηI ) = 0 :
Revised: 9 and solve for det( A + ηI ) = 0 :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that the denominator of g ( p, η ) is proportional to det( A + ηI ) , and so g (0 . 9 , η ) –and the errorat the TD fixed point–can be made arbitrarily large by selecting η close to 4 .
Revised: Note that the denominator of g ( p, η ) is proportional to det( A + ηI ) , and so g (0 . 9 , η ) –and the errorat the TD fixed point–can be made arbitrarily large by selecting η close to 4 .

Original: 83 × 10 − 3 .
Revised: 83 × 10 − 3 .

Original: As this is theonly positive root, the model does not diverge at other values.
Revised: As this is theonly positive root, the model does not diverge at other values.

Original: B.5 Emphatic approaches and our counterexample
Revised: B.5 Emphatic approaches and our counterexample

Original: We use an MP with the same transition function as in Figure 1a, with separate bases Φ m and Φ v forthe emphasis and value stages respectively.
Revised: We use an MP with the same transition function as in Figure 1a, with separate bases Φ m and Φ v forthe emphasis and value stages respectively.

Original: We assume that our interest in all states is uniformlyi = 1 .
Revised: We assume that our interest in all states is uniformlyi = 1 .

Original: We begin by setting the off-policy sampling distribution of µ = [ . 2 . 2 . 6] , used as the diagonalmatrix D µ = diag ( µ ) .
Revised: We begin by setting the off-policy sampling distribution of µ = [ . 2 . 2 . 6] , used as the diagonalmatrix D µ = diag ( µ ) .

Original: Thanks to the simple structure of our example, we know the emphasis ism =
Revised: Thanks to the simple structure of our example, we know the emphasis ism =

Original: i 1 − γ · πD − 1 µ ∝ ( 5 / 4 , 5 / 4 , 5 / 6 ) .
Revised: i 1 − γ · πD − 1 µ ∝ ( 5 / 4 , 5 / 4 , 5 / 6 ) .

Original: We select a basis that allows us to represent this emphasis:
Revised: We select a basis that allows us to represent this emphasis:

Original: [Equation]
Revised: [Equation]

Original: We deliberately choose Φ m to have a poor condition number for reasons that will become apparent later.
Revised: We deliberately choose Φ m to have a poor condition number for reasons that will become apparent later.

Original: We can represent c · ( 5 / 4 , 5 / 4 , 5 / 6 ) exactly for any constant c :
Revised: We can represent c · ( 5 / 4 , 5 / 4 , 5 / 6 ) exactly for any constant c :

Original: [Equation]
Revised: [Equation]

Original:  Using Equation 5 from [17], we define the matrices:
Revised: [Equation] Using Equation 5 from [20], we define the matrices:

Original: [Equation]
Revised: [Equation]

Original: And we apply these to the formulation in Lemma 3 and compute the emphasis weights as a function of the regularization w m :
Revised: And we apply these to the formulation in Lemma 3 and compute the emphasis weights as a function of the regularization w m :

Original: R +

Revised: R +

Original: → R + :
Revised: → R + :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can then use this to compute the new apparent distribution υ , which is the effective distributionthat the updates to the value model see, and it is equal to the emphasis multiplied by the off-policydistribution. 
Revised: We can then use this to compute the new apparent distribution υ , which is the effective distributionthat the updates to the value model see, and it is equal to the emphasis multiplied by the off-policydistribution. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Without any regularization, this should be exactly equal to the on-policy distribution.
Revised: Without any regularization, this should be exactly equal to the on-policy distribution.

Original: [Equation]
Revised: [Equation]

Original: When we compute this value with a small amount of regularization η = 2 × 10 − 4 , we observe that the apparent distribution drifts far away from the on-policy distribution.
Revised: When we compute this value with a small amount of regularization η = 2 × 10 − 4 , we observe that the apparent distribution drifts far away from the on-policy distribution.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The proximate cause of this is the poor condition number of C , caused by the 1100 scale factor appliedto the second column of Φ m .
Revised: The proximate cause of this is the poor condition number of C , caused by the 1100 scale factor appliedto the second column of Φ m .

Original: This allows η to affect different columns by different (relative) amountsin the definition of w ∗ ( η ) , which pushes it away from the symmetric solution.
Revised: This allows η to affect different columns by different (relative) amountsin the definition of w ∗ ( η ) , which pushes it away from the symmetric solution.

Original: See this error shift in
Revised: See this error shift in

Original: Figure 7a.
Revised: Figure 7a.

Original: So far, we have shown how regularization causes a shift in the apparent distribution that the TD updatessee.
Revised: So far, we have shown how regularization causes a shift in the apparent distribution that the TD updatessee.

Original: To complete the example we show how this moves the fixed point of the value function awayfrom a stable point into an asymptote where it may grow without bounds.
Revised: To complete the example we show how this moves the fixed point of the value function awayfrom a stable point into an asymptote where it may grow without bounds.

Original: This second phase followsin the same pattern as the first phase, starting with the desired value function: V = [1 2 .
Revised: This second phase followsin the same pattern as the first phase, starting with the desired value function: V = [1 2 .

Original: 69 1 . 05] anda basis that can almost exactly represent the value function:
Revised: 69 1 . 05] anda basis that can almost exactly represent the value function:

Original: [Equation]
Revised: [Equation]

Original: We use this basis to compute the state-rewards R =
Revised: We use this basis to compute the state-rewards R =

Original: ( I − γP )
Revised: ( I − γP )

Original: V =
Revised: V =

Original: [ − 0 .
Revised: [ − 0 .

Original: 43 1 . 26 − 0 .
Revised: 43 1 . 26 − 0 .

Original: 38] and define the matrices A v and C v and the solution w ∗ v ( η ) :
Revised: 38] and define the matrices A v and C v and the solution w ∗ v ( η ) :

Original: [Equation]
Revised: [Equation]

Original: We can use this solution to compute the error between the value function and the true values, ∥ Φ v w v [Equation] η ) − V ∥ .
Revised: We can use this solution to compute the error between the value function and the true values, ∥ Φ v w v ( η ) − V ∥ .

Original: First, under the corrected distribution without regularization υ (0) ≡ π :
Revised: First, under the corrected distribution without regularization υ (0) ≡ π :

Original: [Equation]
Revised: [Equation]

Original: Then, with regularization in the value function (but not in the emphasis function):
Revised: Then, with regularization in the value function (but not in the emphasis function):

Original: [Equation]
Revised: [Equation]

Original: Then, under the apparent distribution υ induced by use of regularization in the emphasis function, without and with regularization:
Revised: Then, under the apparent distribution υ induced by use of regularization in the emphasis function, without and with regularization:

Original: [Equation]
Revised: [Equation]

Original: It is immediately obvious that the use of regularization in the emphasis function causes the learnedvalue function to be incorrect.
Revised: It is immediately obvious that the use of regularization in the emphasis function causes the learnedvalue function to be incorrect.

Original: Including a regularizing term in the value estimate is not sufficient tofix the value function.
Revised: Including a regularizing term in the value estimate is not sufficient tofix the value function.

Original: This completes the example.
Revised: This completes the example.

Original: From Figure 0 .0 .0 .0 .
Revised: From Figure 0 .0 .0 .0 .

Original: From Figure 7(a) 0 .0 .0 .0 .
Revised: From Figure 7(a) 0 .0 .0 .0 .

Original: From Figure 7(b)  and our counterexample.
Revised: From Figure 7(b) Kolter’s non expansion condition and our counterexample.

Original: In the construction of COF-PAC, a key assumption made is that both the emphasis and value modelsare not subject to runaway TD [17, asm. 4].
Revised: In the construction of COF-PAC, a key assumption made is that both the emphasis and value modelsare not subject to runaway TD [20, asm. 4].

Original: Specifically, they make the strong assumption that
Revised: Specifically, they make the strong assumption that

Original: Kolter’s relaxed-contraction condition [5, eqn.
Revised: Kolter’s relaxed-contraction condition [6, eqn.

Original: 10] holds in the emphasis model at µ and value modelat υ .
Revised: 10] holds in the emphasis model at µ and value modelat υ .

Original: Kolter’s condition selects a convex subset of distributions under which one-step transitionfollowed by projection onto Φ is non-expansive.
Revised: Kolter’s condition selects a convex subset of distributions under which one-step transitionfollowed by projection onto Φ is non-expansive.

Original: We illustrate these regions in Figure 8.
Revised: We illustrate these regions in Figure 8.

Original: Even in theone-dimensional parameterization shown, this condition only holds in a small sub-region of the spaceand therefore appears to be a very strong condition.
Revised: Even in theone-dimensional parameterization shown, this condition only holds in a small sub-region of the spaceand therefore appears to be a very strong condition.

Original: Empirically determining if such a condition holds(or training models to enforce it) may be possible with TD-DO [5, sec 4.1], but it is not clear howthat method interacts with regularization.
Revised: Empirically determining if such a condition holds(or training models to enforce it) may be possible with TD-DO [6, sec 4.1], but it is not clear howthat method interacts with regularization.

Original: C Applied to multi-layer networks
Revised: C Applied to multi-layer networks

Original: We also use a variant of our example to study how the deadly triad appears in multi-layer networks.
Revised: We also use a variant of our example to study how the deadly triad appears in multi-layer networks.

Original: As illustrated in Figure 9, we replace each self-loop with two additional states, forming a clique withthe original state.
Revised: As illustrated in Figure 11, we replace each self-loop with two additional states, forming a clique withthe original state.

Original: The resultant MP has n = 9 states; we define a deterministic obervation functiono : S → B 6 .
Revised: The resultant MP has n = 9 states; we define a deterministic obervation functiono : S → B 6 .

Original: where each state is encoded as the concatenation of the one-hot vector of its subscripts.
Revised: where each state is encoded as the concatenation of the one-hot vector of its subscripts.

Original: The value function is assigned pseudo-randomly in range [ − 1 , 1] , and a consistent reward functionis assigned.
Revised: The value function is assigned pseudo-randomly in range [ − 1 , 1] , and a consistent reward functionis assigned.

Original: We select the family of sampling distributions µ ∝ [4 p, 1 p, 1 p, 4 p, 1 p, 1 p, 8(1 −p ) , 4(1 − p ) , 4(1 − p )] , where the on-policy distribution is at p = 0 .
Revised: We select the family of sampling distributions µ ∝ [4 p, 1 p, 1 p, 4 p, 1 p, 1 p, 8(1 −p ) , 4(1 − p ) , 4(1 − p )] , where the on-policy distribution is at p = 0 .

Original: We wish to learn the model with a two-layer network with k < n nodes in the inner layer.
Revised: We wish to learn the model with a two-layer network with k < n nodes in the inner layer.

Original: We definethe network as f ( o ( s i,j ))
Revised: We definethe network as f ( o ( s i,j ))

Original: = tan − 1 ( o ( s i,j ) ∗ ω 1 ) ∗ ω 2 .
Revised: = tan − 1 ( o ( s i,j ) ∗ ω 1 ) ∗ ω 2 .

Original: The parameters ω 1 ∈ R 6 × k , ω 2 ∈ R k × 1 aretrained to convergence using simple TD updates with semi-gradient updates, a fixed learning rate,and without a target network.
Revised: The parameters ω 1 ∈ R 6 × k , ω 2 ∈ R k × 1 aretrained to convergence using simple TD updates with semi-gradient updates, a fixed learning rate,and without a target network.

Revised: In addition to the example in Figure 5b, we present an additional example in Figure 9.

Original: D Markov Processes
Revised: The same Markov process, at a different off-policy distribution, attains a curve where the non-vacuous regionlies before the divergent region, similar to the second row in Figure 3a. An added observation is thatthese two graphs are mutually incompatible – there is no fixed η that can simultaneously do betterthan vacuity in both, which promotes the idea of testing multiple regularization parameters or usingan adaptive regularization scheme. C.1 Overparameterization does not solve this problem

Revised: Baird’s counterexample [18] shows how, in the linear case, that off-policy divergence can also happenunder overparameterization, as long as some amount of function approximation occurs.

Revised: It is notobvious that this conclusion persists in the neural network case, so we include an additional exampleshowing that the parameterization doesn’t resolve small- η divergence.

Revised: In Figure 10 we plot models with 3 to 13 nodes in the hidden layer.

Revised: For reference, the MDP has 9states, so some models under-parameterize and some models over-parameterize.

Revised: We observe that,in the low-regularization regime, increasing the number of parameters improves the error slightly.

Revised: However, increasing the number of parameters in the hidden layer does not change the behavior inthe the small- η divergence region.

Revised: C.2 Relationship to modern RL algorithms

Revised: It is still not obvious how strongly this instability affects modern RL algorithms, which are alsosensitive to a variety of other failure modes.

Revised: Unlike our analysis, the sampling distribution changesduring training, and regularization mechanisms are more complex than simple ℓ 2 penalities.

Revised: Theexact relationship between the instabilities we study and RL algorithms is an open problem, but weoffer two pieces of indirect evidence suggesting there is a link.

Revised: First, in the offline/batch RL literature, it is well-known that online RL algorithms naively applied cancatastrophically fail if the learned policy is not consistent with the data distribution.

Revised: This is knownas the distribution shift problem, [9, p. 26] and offline RL algorithms are generally constructed toexplicitly address this.

Revised: Second, when using experience replay buffers in online RL algorithms, policyquality generally improves when older transitions are more quickly evicted [3].

Revised: However, there aremultiple factors at work here, and it is not possible to separate out the instability from off-policysampling from the remaining factors.

Original: We use a three-, five- and nine-state Markov Processes to generate examples for this paper.
Revised: We use a three-, five- and nine-state Markov Processes to generate examples for this paper.

Original: Here wegive details of the construction of each example.
Revised: Here wegive details of the construction of each example.

Original: Mathematica code for all examples is included inthe supplementary material.
Revised: Mathematica code for all examples is included inthe supplementary material.

Original: D.1 Three-state
Revised: D.1 Three-state

Original: The construction of the three-state MDP is described in Section 3 and illustrated in Figure 1a.
Revised: The construction of the three-state MDP is described in Section 3 and illustrated in Figure 1a.

Original: Thisexample is used in Examples 1 and 3 For completeness, the transition matrix is:
Revised: Thisexample is used in Examples 1 and 3 For completeness, the transition matrix is:

Original: [Equation]
Revised: [Equation]

Original: s 3 ,s 1 ,s 2 , 1 1 ⁄1 ⁄1 ⁄1 ⁄1 ⁄ 4 1 ⁄s 2 ,s 2 ,s 2 ,e ⁄e ⁄e ⁄
Revised: [Figure]

Original: D
Revised: D.2 Nine-state

Original: -state

Original: This example is used to train neural networks.
Revised: This example is used to train neural networks.

Original: The construction is based on the three-state exampleand the construction is illustrated in Figures 1b and 9.
Revised: The construction is based on the three-state exampleand the construction is illustrated in Figures 1b and 11.

Original: The transition matrix (with omitted zeros) is:  1 1 1 4 44 4 1 1 14 44 4  2 24 44 4  (36) and the observation function that forces the neural network to approximate is:o : S → R 6 =   1 0 0 1 00 1 0 1 00 0 1 1 01 0 0 0 10 1 0 0 10 0 1 0 11 0 0 0 00 1 0 0 00 0 1 0 0  (37)
Revised: The transition matrix (with omitted zeros) is:  1 1 1 4 44 4 1 1 14 44 4  2 24 44 4  (36) and the observation function that forces the neural network to approximate is: [Equation]  [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  D.3 Five-state
Revised: [Equation] [Equation] 0 0 10 0 1 0 11 0 0 0 00 1 0 0 00 0 1 0 0     (37) D.3 Five-state

Original: We use this to generate Figure 3a.
Revised: We use this to generate Figure 3a.

Original: The transition matrix is:  .
Revised: The transition matrix is:  .

Original: 2 . 4 .  (38)
Revised: 2 . 4 .  (38)

Original: We set value function V =
Revised: We set value function V =

Original: [1 , 1 , 1 . 05 , 1 , 1] ⊤ , γ = 0 .
Revised: [1 , 1 , 1 . 05 , 1 , 1] ⊤ , γ = 0 .

Original: 99 , ϵ = 0 .
Revised: 99 , ϵ = 0 .

Original: and set basis:Φ = 1  3 00 31 10 01 1  ∗ (cid:34)  0 0 .   
Revised: and set basis:Φ = 1   3 00 31 10 01 1   ∗ (cid:34)  0 0 .

Original: 0 .(cid:35) (39)
Revised: 0 .(cid:35) (39)

Revised: [Figure]

Revised: The relationship between η and error with different amount of model parameterization (with 3, 5, 7, 9, 11, 13, and 64 nodes in the hidden layer, corresponding to darkening colors.)

Revised: s 3 ,s 1 ,s 2 , 1 1 ⁄1 ⁄1 ⁄1 ⁄1 ⁄ 4 1 ⁄s 2 ,s 2 ,s 2 ,e ⁄e ⁄e ⁄

Original: We also parameterize the off-policy distribution as:
Revised: We also parameterize the off-policy distribution as:

Original: [Equation]
Revised: [Equation]

Original:  where p, q ∈ (0 , 1) .
Revised: (40) where p, q ∈ (0 , 1) .

Original: We verify that (cid:80) D
Revised: We verify that (cid:80) D

Original: = 1 over this domain.
Revised: = 1 over this domain.

Original: The on-policy distribution isπ = 1 / 12 [2 , 3 , 2 , 3 , 2] .
Revised: The on-policy distribution isπ = 1 / 12 [2 , 3 , 2 , 3 , 2] .

Original: The plots correspond to the off-policy distributions:
Revised: The plots correspond to the off-policy distributions:

Original: p → 0 . 77 , q → 0 .
Revised: p → 0 . 77 , q → 0 .

Original: [Equation]
Revised: [Figure]

Original: p → 0 . 4 , q → 0 .
Revised: p → 0 . 4 , q → 0 .

Original: [Equation]
Revised: [Figure]

Original: p → 0 . 02 , q → 0 .
Revised: p → 0 . 02 , q → 0 .

Original: [Equation]
Revised: [Figure]