Original: The Phenomenon of Policy Churn
Revised: The Phenomenon of Policy Churn

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We identify and study the phenomenon of policy churn , that is, the rapid change ofthe greedy policy in value-based reinforcement learning.
Revised: We identify and study the phenomenon of policy churn , that is, the rapid change ofthe greedy policy in value-based reinforcement learning.

Original: Policy churn operates ata surprisingly rapid pace, changing the greedy action in a large fraction of stateswithin a handful of learning updates (in a typical deep RL set-up such as DQNon Atari).
Revised: Policy churn operates ata surprisingly rapid pace, changing the greedy action in a large fraction of stateswithin a handful of learning updates (in a typical deep RL set-up such as DQNon Atari).

Original: We characterise the phenomenon empirically, verifying that it is notlimited to specific algorithm or environment properties.
Revised: We characterise the phenomenon empirically, verifying that it is notlimited to specific algorithm or environment properties.

Original: A number of ablationshelp whittle down the plausible explanations on why churn occurs to just a handful,all related to deep learning.
Revised: A number of ablationshelp whittle down the plausible explanations on why churn occurs to just a handful,all related to deep learning.

Original: Finally, we hypothesise that policy churn is a beneficialbut overlooked form of implicit exploration that casts ϵ -greedy exploration in afresh light, namely that ϵ -noise plays a much smaller role than expected.
Revised: Finally, we hypothesise that policy churn is a beneficialbut overlooked form of implicit exploration that casts ϵ -greedy exploration in afresh light, namely that ϵ -noise plays a much smaller role than expected

Original:  The Phenomenon
Revised:  The Phenomenon

Original: Reinforcement learning (RL) involves agents that incrementally update their policy.
Revised: Reinforcement learning (RL) involves agents that incrementally update their policy.

Original: This process isdriven by the objective of maximising reward, and based on experience that the agent generates viaexploration.
Revised: This process isdriven by the objective of maximising reward, and based on experience that the agent generates viaexploration.

Original: The sequence of policies π 0 , . . .
Revised: The sequence of policies π 0 , . . .

Original: , π k , . .
Revised: , π k , . .

Original: , π T usually starts from a randomly initialisedpolicy π 0 and aims to end at a near-optimal policy π T ≈ π ∗ .
Revised: , π T usually starts from a randomly initialisedpolicy π 0 and aims to end at a near-optimal policy π T ≈ π ∗ .

Original: Ideally, steps in that sequence( π k → π k +1 ) are policy improvements that increase expected reward.
Revised: Ideally, steps in that sequence( π k → π k +1 ) are policy improvements that increase expected reward.

Original: This paper studies the amount of policy change that goes along with such a policy update process(for a definition, see Section 1.1).
Revised: This paper studies the amount of policy change that goes along with such a policy update process(for a definition, see Section 1.1).

Original: In particular, it makes the core observation that policy change inpractice (as illustrated in some typical deep RL settings) is orders of magnitude larger than couldhave been expected, and stands in contrast to various reference algorithms (Sections 1.2 and 3.3).
Revised: In particular, it makes the core observation that policy change inpractice (as illustrated in some typical deep RL settings) is orders of magnitude larger than couldhave been expected, and stands in contrast to various reference algorithms (Sections 1.2 and 3.3).

Original: Key observation 1: The greedy policy changes much more rapidly than you probably think.
Revised: Key observation 1: The greedy policy changes much more rapidly than you probably think.

Original: a a As a coarse magnitude for the impatient reader: in a typical run of DQN on Atari, the greedy policy changes in ≈ 10% of all states after a single gradient update (Figure 1 and Section 1.2).
Revised: a a As a coarse magnitude for the impatient reader: in a typical run of DQN on Atari, the greedy policy changes in ≈ 10% of all states after a single gradient update (Figure 1 and Section 1.2).

Original: We dub this phenomenon “ policy churn ”, to highlight that most of this policy change may beunnecessary.
Revised: We dub this phenomenon “ policy churn ”, to highlight that most of this policy change may beunnecessary.

Original: We study the phenomenon in depth, determining the range of deep RL scenarios itappears in, fleshing out its properties, and in the process narrowing the space of potential causes andmechanisms involved using a set of ablations (Section 3).
Revised: We study the phenomenon in depth, determining the range of deep RL scenarios itappears in, fleshing out its properties, and in the process narrowing the space of potential causes andmechanisms involved using a set of ablations (Section 3).

Original: Our second key message relates the phenomenon of churn to exploration, specifically in the contextof ϵ -greedy exploration (Section 2), with some more speculative ramifications in Section 4.
Revised: Our second key message relates the phenomenon of churn to exploration, specifically in the contextof ϵ -greedy exploration (Section 2), with some more speculative ramifications in Section 4.

Original: Key observation 2: Policy churn is a significant driver of exploration.
Revised: Key observation 2: Policy churn is a significant driver of exploration.

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: 1.1 Defining policy change
Revised: 1.1 Defining policy change

Original: A policy is a function from states s ∈ S to a distribution over actions a ∈ A , where for the purposesof this paper
Revised: A policy is a function from states s ∈ S to a distribution over actions a ∈ A , where for the purposesof this paper

Original: A is discrete.
Revised: A is discrete.

Original: We quantify the local, per-state policy change between policies π and π ′using the moved probability mass (i.e., the total variation distance):
Revised: We quantify the local, per-state policy change between policies π and π ′using the moved probability mass (i.e., the total variation distance):

Original: [Equation]
Revised: [Equation]

Original: which satisfies 0 ≤ W ( π, π ′ | s ) ≤ 1 .
Revised: which satisfies 0 ≤ W ( π, π ′ | s ) ≤ 1 .

Original: When π and π ′ are greedy policies derived from a state-actionvalue function—that is, π ∈ arg max a q ( s, a ) —then W ( π, π ′ | s )
Revised: When π and π ′ are greedy policies derived from a state-actionvalue function—that is, π ∈ arg max a q ( s, a ) —then W ( π, π ′ | s )

Original: = 1 if the arg max action in state schanges upon replacing q by the function q ′ underlying π ′ , and W ( π, π ′ | s )
Revised: = 1 if the arg max action in state schanges upon replacing q by the function q ′ underlying π ′ , and W ( π, π ′ | s )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: Similarreasoning applies more generally when both π and π ′ are deterministic policies.
Revised: Similarreasoning applies more generally when both π and π ′ are deterministic policies.

Original: We can aggregatepolicy change across all states, weighted by a state-distribution µ :
Revised: We can aggregatepolicy change across all states, weighted by a state-distribution µ :

Original: [Equation]
Revised: [Equation]

Original: where a reasonable choice for µ is the empirical state distribution encountered during training (whichis non-stationary, but it could also be the stationary distribution of a fixed policy, or the uniformdistribution across all states, as discussed for some of the toy scenarios below).
Revised: where a reasonable choice for µ is the empirical state distribution encountered during training (whichis non-stationary, but it could also be the stationary distribution of a fixed policy, or the uniformdistribution across all states, as discussed for some of the toy scenarios below).

Original: For greedy policies,
Revised: For greedy policies,

Original: W ( π, π ′ ) is simply the fraction of states where an arg max switch occurred.
Revised: W ( π, π ′ ) is simply the fraction of states where an arg max switch occurred.

Original: For settings where policy performance stabilises at some point t = P of training (e.g., hitting aperformance plateau, or converging to optimal behaviour), two additional metrics may be of interest,namely the cumulative policy change W 0: P until that point 1 , and the average policy change after thatpoint W + (which could be zero, e.g., if the process converges):
Revised: For settings where policy performance stabilises at some point t = P of training (e.g., hitting aperformance plateau, or converging to optimal behaviour), two additional metrics may be of interest,namely the cumulative policy change W 0: P until that point 1 , and the average policy change after thatpoint W + (which could be zero, e.g., if the process converges):

Original: [Equation]
Revised: [Equation]

Original: 1.2 Quantifying the effect
Revised: 1.2 Quantifying the phenomenon

Original: Given an initial and a final policy, the process with the minimum amount of policy change is an oraclethat jumps from π 0 to π P in a single step ( P = 1 ).
Revised: Given an initial and a final policy, the process with the minimum amount of policy change is an oraclethat jumps from π 0 to π P in a single step ( P = 1 ).

Original: By construction, this can incur a policy change ofat most one unit, W 0: P ≤ 1 , which is a form of lower bound.
Revised: By construction, this can incur a policy change ofat most one unit, W 0: P ≤ 1 , which is a form of lower bound.

Original: In value-based deep RL agents, a naturaldefinition for the sequence of policies is to use the induced greedy policies π k ∈ arg max a q θ k ( s, a )where θ k are the parameters of the Q-function at iteration k .
Revised: In value-based deep RL agents, a naturaldefinition for the sequence of policies is to use the induced greedy policies π k ∈ arg max a q θ k ( s, a )where θ k are the parameters of the Q-function at iteration k .

Original: In agents that use a target network(inducing π ) that is an older copy of the online network (inducing π ′ ), it is easy to measure W ( π, π ′ )
Revised: In agents that use a target network (inducing π ) that is an older copy of the online network (inducing π ′ ), it is easy to measure W ( π, π ′ )

Original: by comparing their arg max actions at the points in training where the target network lags behind byjust one update.
Revised: by comparing their arg max actions at the points in training where the target network lags behind byjust one update.

Original: Figure 1 shows typical values for W on a few Atari games, estimated by comparing [Figure] online and target network policies on batches of experience sampled from the agent’s replay buffer.
Revised: Figure 1 shows typical values for W on a few Atari games, estimated by comparingonline and target network policies on batches of experience sampled from the agent’s replay buffer.

Original: Itis worth emphasising that with such rates of average policy change, of W ≈ 10% per update, themagnitude of whole-lifetime change becomes enormous: across training, an agent like DQN, whichperforms 10 7 updates, changes its greedy action a million times in each state (on average).
Revised: Itis worth emphasising that with such rates of average policy change, of W ≈ 10% per update, themagnitude of whole-lifetime change becomes enormous: across training, an agent like DQN, whichperforms 10 7 updates, changes its greedy action a million times in each state (on average).

Original: A second striking result is the amount of policy change in late training, when the performance ofthe policy no longer changes (in the case of P ONG it is arguably optimal): there is still a change of
Revised: A second striking result is the amount of policy change in late training, when the performance ofthe policy no longer changes (in the case of P ONG it is arguably optimal): there is still a change of

Original: W + ≈ 5% per update (Figures 1 and 7).
Revised: W + ≈ 5% per update (Figures 1 and 7).

Original: This highlights that a lot (maybe most) of policy churn isnot directed at a policy improvement; we revisit this in Section 4.2.
Revised: This highlights that a lot (maybe most) of policy churn isnot directed at a policy improvement; we revisit this in Section 4.2.

Original: Is this unexpected?
Revised: Is this unexpected?

Original: We conducted an informal survey of over 25 deep RL practitioners for theirestimate on how rapidly the policy changes in a typical Q-learning based setup.
Revised: We conducted an informal survey of over 50 deep RL practitioners, includingsome of the inventors of DQN [27], for their estimate on how rapidly the policy changes in a typical Qlearning based setup.

Original: The questionwas: For the greedy policy to change in 10% of all states, how many learningupdates does it take? (or equivalent).
Revised: The question was: For the greedy policy to change in 10% of allstates, how many learning updates does it take? (or equivalent).

Original: The median response was 1 000 updates, with answers varying between 1 and 1 000 000 updates.
Revised: The median responsewas 1 000 updates, with answers varying between 1 and 1 000 000 updates.

Original: This deviates by three orders of magnitude from theempirical value of 1 update (or W ≈ 0 .
Revised: This deviates by threeorders of magnitude from the empirical value of 1 update (or W ≈ 0 .

Original: 1 , see Figure 1).
Revised: 1 , see Figure 1).

Original: Policy change in other settings.
Revised: Policy change in other settings.

Original: To appreciate the large empirical magnitudes of (cumulative)policy change we observe in deep RL, we can contrast them with a few alternative settings.
Revised: To appreciate the large empirical magnitudes of (cumulative)policy change we observe in deep RL, we can contrast them with a few alternative settings.

Original: Forexample, classic dynamic programming techniques such as value iteration or policy iteration [33],when applied to toy RL domains such as FourRooms, Catch, or DeepSea [23], accumulate W 0: P ≈ 1 ,not much more than the single-step oracle (see Appendix B.4).
Revised: Forexample, classic dynamic programming techniques such as value iteration or policy iteration [44],when applied to toy RL domains such as FourRooms, Catch, or DeepSea [33], accumulate W 0: P ≈ 1 ,not much more than the single-step oracle (see Appendix B.4).

Original: The two main differences fromdeep RL are their tabular nature (no function approximation, FA) and non-incremental updates.
Revised: The two main differences fromdeep RL are their tabular nature (no function approximation, FA) and non-incremental updates.

Original: It ispossible to construct tabular settings with much larger policy change, with either incremental updates(Appendix A.3) or bootstrapping (Appendix A.4).
Revised: It ispossible to construct tabular settings with much larger policy change, with either incremental updates(Appendix A.3) or bootstrapping (Appendix A.4).

Original: A minimalist example with non-linear FA andincremental updates is supervised learning on the MNIST dataset (the “policy” in this case are thepredicted label probabilities).
Revised: A minimalist example with non-linear FA andincremental updates is supervised learning on the MNIST dataset (the “policy” in this case are thepredicted label probabilities).

Original: Training a digit classifier to convergence accumulates W 0: P ≈ 10 , thatis, the average input goes through 10 label switches (see Appendix B.5).
Revised: Training a digit classifier to convergence accumulates W 0: P ≈ 10 , thatis, the average input goes through 10 label switches (see Appendix B.5).

Original: None of these examplesare fully satisfying, as they are not apples-to-apples comparisons; so in Section 3.3 we construct aspectrum of algorithmic variants that spans from tabular policy iteration (without churn), via tabular
Revised: None of these examplesare fully satisfying, as they are not apples-to-apples comparisons; so in Section 3.3 we construct aspectrum of algorithmic variants that spans from tabular policy iteration (without churn), via tabular

Original: Q-learning to an approximation of DQN (with realistic magnitudes of churn).
Revised: Q-learning to an approximation of DQN (with realistic magnitudes of churn).

Original:  The Exploration Effect
Revised:  The Exploration Effect

Original: The next question is what happens when acting with the policies produced by a learning process withhigh churn: it is likely that the rapid change of greedy policies impacts the exploratory behaviour of agents.
Revised: The next question is what happens when acting with the policies produced by a learning process withhigh churn: it is likely that the rapid change of greedy policies impacts the exploratory behaviourof agents.

Original: While each individual greedy policy would lead to a very narrow set of experience in a(nearly) deterministic environment like Atari, the fact that the greedy policy changes so rapidly (in10% of states per update, with one update every 16 environment frames in DoubleDQN) makes thedata distribution broad enough that in many circumstances, no other form of exploration (such asstochasticity introduced via an ϵ -greedy policy) is needed for good performance.
Revised: While each individual greedy policy would lead to a very narrow set of experience in a(nearly) deterministic environment like Atari, the fact that the greedy policy changes so rapidly (in10% of states per update, with one update every 16 environment frames in DoubleDQN) makes thedata distribution broad enough that in many circumstances, no other form of exploration (such asstochasticity introduced via an ϵ -greedy policy) is needed for good performance.

Original: To see this effect inpractice across a range of Atari games, compare green (baseline) and gold ( ϵ = 0 ) curves in Figure 2. {+
+}
Revised: To see this effectin practice across a range of Atari games, compare green (baseline) and gold ( ϵ = 0 ) curves in Figure 2. {+
+}

Original: Conversely, removing (some of the) churn in the behaviour during training, which can be done byacting with the target network (updated only every 120 000 frames in the DoubleDQN agent) insteadof the online network,  reduces performance even in the presence of ϵ -greedy exploration(blue in
Revised: Conversely, removing (some of the) churn in the behaviour during training, which can be done byacting with the target network (updated only every 120 000 frames in the DoubleDQN agent) insteadof the online network, sometimes reduces performance even in the presence of ϵ -greedy exploration

Original: Figure 2).
Revised: [Figure]

Original: Additionally, we show that performance often collapses completely when both forms ofexploration are removed (no churn, ϵ = 0 , in red).

Original: Figure 2 compares all four variants of exploration,indicating that the two sources of exploration have different contributions in different games.

Original: Sufficient exploration with ϵ = 0 .
Revised: Sufficient exploration with ϵ = 0 .

Original: The perhaps unintuitive observation of successful explorationwith purely greedy policies has been made before, albeit implicitly.
Revised: The perhaps unintuitive observation of successful exploration with purely greedy policies has been made before, albeit implicitly.

Original: In particular, in the presence ofcertain alternative exploration methods such as noisy nets [8], no significant additional advantageis obtained from using ϵ > 0 [10].
Revised: In particular, in the presence of certain alternative exploration methods such as noisy nets [15], no significant additional advantage is obtained from using ϵ > 0 [20].

Original: Other works containing experimental variants with ϵ = 0demonstrated successful training in this setting [25, 28] without highlighting the result.
Revised: Other works containing experimental variants with ϵ = 0demonstrated successful training in this setting [35, 39] without highlighting the result.

Original: Consistent behaviour and Thompson sampling.
Revised: Consistent behaviour and Thompson sampling.

Original: In considering the potential exploration benefitsof a rapidly changing policy, it is worth qualitatively contrasting the resulting behaviour to that of anϵ -greedy policy with ϵ > 0 .
Revised: In considering the potential exploration benefits of a rapidly changing policy, it is worth qualitatively contrasting the resulting behaviour to that of anϵ -greedy policy with ϵ > 0 .

Original: The latter generates high-frequency dithering, with uncorrelated randomaction decisions at consecutive states and the effect of most exploratory actions likely undone bythe following greedy action [5].
Revised: The latter generates high-frequency dithering, with uncorrelated random action decisions at consecutive states and the effect of most exploratory actions likely undone by the following greedy action [11].

Original: By contrast, policy churn induced exploration can be expected togenerate temporally correlated, consistent exploration (necessary, though perhaps not sufficient, toperform “deep” exploration, as in ensemble and Thompson sampling methods [34, 22]).
Revised: By contrast, policy churn induced exploration can be expected to generate temporally correlated, consistent exploration (necessary, though perhaps not sufficient, to perform “deep” exploration, as in ensemble and Thompson sampling methods [45, 32]).

Original: On the otherhand, while ϵ -greedy exploration is explicitly unbiased in action space, policy churn likely prefersexploration across near-optimal actions (with respect to the current value function).
Revised: On the other hand, while ϵ -greedy exploration is explicitly unbiased in action space, policy churn likely prefers exploration across near-optimal actions (with respect to the current value function).

Original: This may bebeneficial in some settings, for example when some actions are deadly, and high ϵ prevents longepisodes.
Revised: This may be beneficial in some settings, for example when some actions are deadly, and high ϵ prevents long episodes.

Original: It may also be detrimental in others, where ϵ > 0 helps the agent to avoid getting stuck.
Revised: It may also be detrimental in others, where ϵ > 0 helps the agent to avoid getting stuck.

Original:  Potential Causes
Revised:  Potential Causes

Original: With the presence and impact of the churn phenomenon established, this section aims to provideadditional depth.
Revised: With the presence and impact of the churn phenomenon established, this section aims to provide additional depth.

Original: First, we look at the generality of the effect in Section 3.1.
Revised: First, we look at the generality of the effect in Section 3.1.

Original: Second, we conductinvestigations into the sensitivity of the phenomenon, with Section 3.2 showing ablations to the largescale deep RL agents (with more material in Appendix A), and Section 3.3 taking the complementaryapproach of interpolating between dynamic programming and a DQN approximation on a toy domain.
Revised: Second, we conduct investigations into the sensitivity of the phenomenon, with Section 3.2 showing ablations to the large- scale deep RL agents (with more material in Appendix A), and Section 3.3 taking the complementary [Table] approach of interpolating between dynamic programming and a DQN approximation on a toy domain.

Original: Finally, Section 3.4 synthesises the findings and postulates some compatible underlying mechanisms.
Revised: Finally, Section 3.4 synthesises the findings and postulates some compatible underlying mechanisms.

Original: 3.1 Breadth of prevalence and non-causes
Revised: 3.1 Breadth of prevalence and non-causes

Original: To judge the importance of the phenomenon, we need to establish whether it is specific to a narrowrange of settings, or prevalent in a variety of domains, algorithmic variants, and hyper-parameters.
Revised: To judge the importance of the phenomenon, we need to establish whether it is specific to a narrowrange of settings, or prevalent in a variety of domains, algorithmic variants, and hyper-parameters.

Original: It turns out that this is easy to do, because the effect is very much not a subtle one.
Revised: It turns out that this is easy to do, because the effect is very much not a subtle one.

Original: In fact, policychurn is present in two very different deep RL agents, namely DoubleDQN [36] and (a variantof) R2D2 [11], both widely used for training on Atari (see Appendix B.1 and B.2 for agent andenvironment specifications).
Revised: In fact, policychurn is present in two very different deep RL agents, namely DoubleDQN [47] and (a variantof) R2D2 [21], both widely used for training on Atari (see Appendix B.1 and B.2 for agent andenvironment specifications).

Original: Despite the large differences between the algorithms summarised in
Revised: Despite the large differences between the algorithms summarised in

Original: Table 1, the magnitude of policy change is surprisingly similar, indicating that policy churn is unlikelystrongly dependent on any of these specific choices.
Revised: Table 1, the magnitude of policy change is surprisingly similar, indicating that policy churn is unlikelystrongly dependent on any of these specific choices.

Original: [Table]

Original: The effect also appears not specific to any environment: we measure similar magnitudes of policychange across a range of Atari games that vary in many dimensions, such as action space, rewardscale and sparsity, deadliness, etc.
Revised: The effect also appears not specific to any environment: we measure similar magnitudes of policychange across a range of Atari games that vary in many dimensions, such as action space, rewardscale and sparsity, deadliness, etc.

Original: Furthermore, it is present in all stages of training (see Figures 1and 5).
Revised: Furthermore, it is present in all stages of training (see Figures 1and 5).

Original: Unsurprisingly it it highest in early learning, but it remains high during training and evenafter evaluation performance converges or stabilises (possibly at the optimal level as in P ONG ).
Revised: Unsurprisingly it it highest in early learning, but it remains high during training and evenafter evaluation performance converges or stabilises (possibly at the optimal level as in P ONG ).

Original: 3.2 Ablations
Revised: 3.2 Ablations

Original: Redundant actions.
Revised: Redundant actions.

Original: A simple factor that could explain policy churn are redundant actions, i.e.,when nominally different actions have the same outcome (generally, or in a large fraction of states).
Revised: A simple factor that could explain policy churn are redundant actions, i.e.,when nominally different actions have the same outcome (generally, or in a large fraction of states).

Original: This property varies widely by environment (see also [21]), but we observe similar levels of policychange across different ones (Figures 1 and 5).
Revised: This property varies widely by environment (see also [31]), but we observe similar levels of policychange across different ones (Figures 1 and 5).

Original: Also, when exposing the full Atari action set in allgames, creating explicit global redundancy, the churn magnitudes are not affected much (Figure 6).
Revised: Also, when exposing the full Atari action set in allgames, creating explicit global redundancy, the churn magnitudes are not affected much (Figure 6).

Original: Appendix A.2 looks at the fine-grained aspect of which actions tend to be swapped for which others,and finds no structure easily related to the (known) equivalence relations (Figure 8).
Revised: Appendix A.2 looks at the fine-grained aspect of which actions tend to be swapped for which others,and finds no structure easily related to the (known) equivalence relations (Figure 8).

Original: In other words,most arg max changes are not happening between equivalent actions.
Revised: In other words,most arg max changes are not happening between equivalent actions.

Original: Small action gaps.
Revised: Small action gaps.

Original: Another potential factor for large policy churn with greedy policies can be theinterplay between FA and small action gaps (difference between largest and second-largest actionvalues): small approximation error can suffice for sub-optimal actions to overtake optimal ones.
Revised: Another potential factor for large policy churn with greedy policies can be theinterplay between FA and small action gaps (difference between largest and second-largest actionvalues): small approximation error can suffice for sub-optimal actions to overtake optimal ones.

Original: Thishypothesis predicts that value learning methods inducing larger action gaps (e.g., Advantage Learning(AL, [1]) which artificially lowers the values of sub-optimal actions) could reduce policy churn.
Revised: Thishypothesis predicts that value learning methods inducing larger action gaps (e.g., Advantage Learning(AL, [3]) which artificially lowers the values of sub-optimal actions) could reduce policy churn.

Original: Figure 3 (right) shows that indeed policy churn is diminished substantially by AL, correlating withan increase of action gaps (which consistently grows under AL, see Figure 9).
Revised: Figure 3 (right) shows that indeed policy churn is diminished substantially by AL, correlating with an increase of action gaps (which consistently grows under AL, see Figure 9).

Original: Curiously, this doesnot seem to severely diminish the remaining policy churn’s effectiveness for exploration: Figure 10shows successful training of an AL-DQN with ϵ = 0 .
Revised: Curiously, this doesnot seem to severely diminish the remaining policy churn’s effectiveness for exploration: Figure 10shows successful training of an AL-DQN with ϵ = 0 .

Original: Non-stationary state distribution.
Revised: Non-stationary state distribution.

Original: Another explanatory hypothesis for the observed large magnitude of policy churn relates it to the non-stationary data distribution caused by the evolving datagenerating policy.
Revised: Another explanatory hypothesis for the observed large magnitude of policy churn relates it to the non-stationary data distribution caused by the evolving datagenerating policy.

Original: Even when the policy seems to have converged with near-optimal performance,like in P ONG , policy churn may concentrate on states where action decisions are inconsequential(where multiple actions have near-equal value), causing non-stationarity in the data distribution andthereby driving further policy churn.
Revised: Even when the policy seems to have converged with near-optimal performance,like in P ONG , policy churn may concentrate on states where action decisions are inconsequential(where multiple actions have near-equal value), causing non-stationarity in the data distribution andthereby driving further policy churn.

Original: To test this, we utilize the “forked tandem” setting from [24], inwhich a high-performing policy is trained on the stationary data distribution generated by its initialsnapshot.
Revised: To test this, we utilize the “forked tandem” setting from [34], inwhich a high-performing policy is trained on the stationary data distribution generated by its initialsnapshot.

Original: As can be seen in Figures 3 (right) and 11, the high level of policy churn is still preservedin this stationary-data regime, ruling out data non-stationarity as the main driver of the phenomenon.
Revised: As can be seen in Figures 3 (right) and 11, the high level of policy churn is still preservedin this stationary-data regime, ruling out data non-stationarity as the main driver of the phenomenon.

Original: Non-stationary targets.
Revised: Non-stationary targets.

Original: Temporal difference (TD) learning can give rise to another form of nonstationarity, as the bootstrap targets change at the pace of the target value function, which is preservedeven in the forked tandem setting.
Revised: Temporal difference (TD) learning can give rise to another form of nonstationarity, as the bootstrap targets change at the pace of the target value function, which is preservedeven in the forked tandem setting.

Original: To sidestep this, a simple control experiment uses the same setupbut with Monte-Carlo returns as learning targets, turning it into pure policy evaluation via supervised [Figure] d v a n t a g e g [Figure]
Revised: To sidestep this, a simple control experiment uses the same setupbut with Monte-Carlo returns as learning targets, turning it into pure policy evaluation via supervisedregression (with noisy targets).

Original: p (

Original: [Figure]

Original: t

Original: [Figure]

Original: t o p

Original: [Figure]

Original: a c t i o n s ) [Figure] regression (with noisy targets). Figures 3, right and  show  tothe Q-learning updates,  phenomenon  algorithms.
Revised: The results (Figures 3, right, and 12) show a similar level of churn tothe Q-learning updates, indicating that the phenomenon is not specific to TD-based algorithms.

Original: Decoupled acting and target networks.
Revised: Decoupled acting and target networks.

Original: Using a purely greedy policy based on the agent’s targetnetwork and varying the frequency with which it is synchronized to the online network allows usto assess how much policy churn is beneficial for exploration.
Revised: Using a purely greedy policy based on the agent’s targetnetwork and varying the frequency with which it is synchronized to the online network allows usto assess how much policy churn is beneficial for exploration.

Original: To avoid conflation with an adverseeffect on learning stability, we keep the regular target network update frequency constant, while usingan additional acting network , periodically copied from the online network, for behaviour generationalone.
Revised: To avoid conflation with an adverseeffect on learning stability, we keep the regular target network update frequency constant, while usingan additional acting network , periodically copied from the online network, for behaviour generationalone.

Original: Figure 13 shows that an acting network updated more than every ≈ 1 000 gradient updatesachieves most of the exploration benefits of acting with the online network, though in some gameshigher frequencies yield further benefit.
Revised: Figure 13 shows that an acting network updated more than every ≈ 1 000 gradient updatesachieves most of the exploration benefits of acting with the online network, though in some gameshigher frequencies yield further benefit.

Original: This implies that the amount of policy change neededfor exploration is much smaller than what is generated by DQN’s learning process (otherwise theobserved collapse would happen at much smaller update intervals).
Revised: This implies that the amount of policy change neededfor exploration is much smaller than what is generated by DQN’s learning process (otherwise theobserved collapse would happen at much smaller update intervals).

Original: Self-correction and the tandem effect.
Revised: Self-correction and the tandem effect.

Original: In [24] a phenomenon dubbed the “tandem effect” wasobserved: the failure of a deep RL agent to adequately learn from the training data generated by adifferent instance of the same agent, highlighting the importance of self-correction by interactivelygenerated data from the policy being trained.
Revised: In [34] a phenomenon dubbed the “tandem effect” wasobserved: the failure of a deep RL agent to adequately learn from the training data generated by adifferent instance of the same agent, highlighting the importance of self-correction by interactivelygenerated data from the policy being trained.

Original: One of their settings, the “forked tandem”, starts with acopy of a high-performing policy and uses data sampled from its stationary distribution to continuetraining; even this apparently benign scenario leads to instability and potential collapse of the trainedpolicy.
Revised: One of their settings, the “forked tandem”, starts with acopy of a high-performing policy and uses data sampled from its stationary distribution to continuetraining; even this apparently benign scenario leads to instability and potential collapse of the trainedpolicy.

Original: Policy churn may provide a partial mechanistic explanation for the origin of the instability,showing that rapid policy change can be expected at all stages of training.
Revised: Policy churn may provide a partial mechanistic explanation for the origin of the instability,showing that rapid policy change can be expected at all stages of training.

Original: The observation that thetrained policy changes on a significant proportion of states at every update (performed on a negligiblysmall sample, a single minibatch of 32 state transitions) supports the hypothesis from [24] thaterroneous extrapolation or over-generalization may play a key role in causing deviation and instabilityand producing the tandem effect in the absence of corrective training signal from self-generated data.
Revised: The observation that thetrained policy changes on a significant proportion of states at every update (performed on a negligiblysmall sample, a single minibatch of 32 state transitions) supports the hypothesis from [34] thaterroneous extrapolation or over-generalization may play a key role in causing deviation and instabilityand producing the tandem effect in the absence of corrective training signal from self-generated data.

Original: Analogously to results in [24] we observe that the magnitude of policy churn is highly correlatedwith the depth of the trained function approximator, further supporting this hypothesis; see Figures 3(right) and 14, which show specifically how policy change drops dramatically in a linear FA regime.
Revised: Analogously to results in [34] we observe that the magnitude of policy churn is highly correlatedwith the depth of the trained function approximator, further supporting this hypothesis; see Figures 3(right) and 14, which show specifically how policy change drops dramatically in a linear FA regime.

Original: 3.3 Detailed case study: Catch
Revised: 3.3 Detailed case study: Catch

Original: Catch is a toy environment where the total number of states is small enough to be amenable toground-truth dynamic programming approaches using the explicit matrix of transition probabilities,while providing an observation space that requires non-linear FA.
Revised: Catch is a toy environment where the total number of states is small enough to be amenable toground-truth dynamic programming approaches using the explicit matrix of transition probabilities,while providing an observation space that requires non-linear FA.

Original: We construct a spectrum of settingsthat all learn the optimal policy after some P iterations.
Revised: We construct a spectrum of settingsthat all learn the optimal policy after some P iterations.

Original: For each we measure cumulative policychange W 0: P and average change after convergence W + .
Revised: For each we measure cumulative policy [Figure] [Figure] change W 0: P and average change after convergence W + .

Original: Main results are shown in Figure 4.
Revised: Main results are shown in Figure 4.

Original: Sitting

Original: [Figure] at one end of the , exact  converges  P = 10 steps with W 0: P = 
Revised: Sittingat one end of the spectrum, exact value iteration converges after P = 10 steps with W 0: P = 0 .

Original: (because the initial policy is random, and the optimal policy has ties in most states).
Revised: 09(because the initial policy is random, and the optimal policy has ties in most states).

Original: The next simplestsetting is tabular Q-learning with incremental updates (here and elsewhere, hyperparameters like thelearning rate are tuned for fast convergence to optimal performance, see Appendix B.3 for details).
Revised: The next simplestsetting is tabular Q-learning with incremental updates (here and elsewhere, hyperparameters like thelearning rate are tuned for fast convergence to optimal performance, see Appendix B.3 for details).

Original: The next steps add in non-linear FA (shallow, then deep).
Revised: The next steps add in non-linear FA (shallow, then deep).

Original: At the other end of the spectrum is anapproximation to DQN which includes experience replay, mini-batches, a deep neural network, and anadvanced optimiser (RMSProp [35]).
Revised: At the other end of the spectrum is anapproximation to DQN which includes experience replay, mini-batches, a deep neural network, and anadvanced optimiser (RMSProp [46]).

Original: Figure 4 also shows intermediate variations such as supervisedregression to q ∗ , and different optimisers (e.g., Adam [13]); additional results in Appendix A.1.
Revised: Figure 4 also shows intermediate variations such as supervisedregression to q ∗ , and different optimisers (e.g., Adam [23]); additional results in Appendix A.1.

Original: 3.4 Mechanistic hypothesis
Revised: 3.4 Mechanistic hypothesis

Original: In an attempt to synthesise all the evidence presented so far, we propose that the observed policychurn is primarily a result of two components that need to be present jointly:
Revised: In an attempt to synthesise all the evidence presented so far, we propose that the observed policychurn is primarily a result of two components that need to be present jointly:

Original: Non-linear, global function approximation (such as deep neural networks), where each update
Revised: Non-linear, global function approximation (such as deep neural networks), where each update

Original: can affect all states and all actions.
Revised: can affect all states and all actions.

Original: A learning process with a high amount of noise.
Revised: A learning process with a high amount of noise.

Original: This could have multiple sources: stochasticoptimisation (e.g. small batch sizes, large learning rates)
Revised: This could have multiple sources: stochastic optimisation (e g. small batch sizes, large learning rates), noisy learning targets, non-stationary

Original: , noisy learning targets, non-stationarydata (or targets, as with bootstrapping), an imbalanced data distribution (e.g., seeing someactions far more often than others).
Revised: data (or targets, as with bootstrapping), an imbalanced data distribution (e g., seeing some actions far more often than others).

Original: In other words, the rapid policy change dubbed churn is the symptom of high-variance updates to aglobal function approximator.
Revised: In other words, the rapid policy change dubbed churn is the symptom of high-variance updates to aglobal function approximator.

Original: A compounding effect could be a mismatch between the regressionloss driving the learning process and policy space, which is what matters for performance and forexploration (and is measured here).
Revised: A compounding effect could be a mismatch between the regressionloss driving the learning process and policy space, which is what matters for performance and forexploration (and is measured here).

Original: While supervised learning does not have this mismatch, it couldconceivably have large “policy change” too.
Revised: While supervised learning does not have this mismatch, it couldconceivably have large “policy change” too.

Original: An in-depth treatment exceeds the scope for this paper,but preliminary results on MNIST (Appendix B.5) indicate no large churn in the supervised setting.
Revised: An in-depth treatment exceeds the scope for this paper,but preliminary results on MNIST (Appendix B.5) indicate no large churn in the supervised setting.

Original:  Where do we go from here?
Revised:  Where do we go from here?

Original: 4.1 Learning at the edge of chaos
Revised: 4.1 Learning at the edge of chaos

Original: Deep learning has a well-known trade-off between speed of learning and stability that incentivisestuning the learning dynamics to be near the edge of chaos 4 [4].
Revised: Deep learning has a well-known trade-off between speed of learning and stability that incentivisestuning the learning dynamics to be near the edge of chaos 5 [10].

Original: The presence of policy churncould enrich this picture in two ways for the case of deep RL.
Revised: The presence of policy churncould enrich this picture in two ways for the case of deep RL.

Original: First, to the extent that rapid policychange helps drive exploration (Section 2), this is an additional incentive to keep learning dynamicssufficiently noisy.
Revised: First, to the extent that rapid policychange helps drive exploration (Section 2), this is an additional incentive to keep learning dynamicssufficiently noisy.

Original: Second, to the extent that self-correction is central to value-based learning [24],which relies on the actions to be corrected being picked by a greedy policy, this also encourages alarge amount of policy change.
Revised: Second, to the extent that self-correction is central to value-based learning [34],which relies on the actions to be corrected being picked by a greedy policy, this also encourages alarge amount of policy change.

Original: Circumstantial evidence for these is that value-based RL tends torequire additional stabilisation mechanisms (e.g., target networks), less common in policy-based RL.
Revised: Circumstantial evidence for these is that value-based RL tends torequire additional stabilisation mechanisms (e.g., target networks), less common in policy-based RL.

Original: 4.2 Policy null-spaces
Revised: 4.2 Policy null-spaces

Original: Policy change is necessary for policy improvement, but does not necessarily imply change in performance.
Revised: Policy change is necessary for policy improvement, but does not necessarily imply change inperformance.

Original: We define the space of policies that have the same value (in all states) as a reference policy πas its null space Null( π ) .
Revised: We define the space of policies that have the same value (in all states) as a referencepolicy π as its null space Null( π ) .

Original: We can quantify its diameter | Null( π ) | := max π ′ ,π ′′ ∈ Null( π ) W ( π ′ , π ′′ ) asthe largest policy change possible between any pair of policies within it.
Revised: We can quantify its diameter [Equation] as the largest policy change possible between any pair of policies within it.

Original: Typical scenarios with largenull spaces have states in which actions have no effect (e.g., move actions while falling), multipleactions with the same effect (globally, or locally in part of state space), or multiple paths leadingto the same outcome (e.g., up-then-left vs. left-then-up).
Revised: Typical scenarios withlarge null spaces have states in which actions have no effect (e.g., move actions while falling),multiple actions with the same effect (globally, or locally in part of state space), or multiple pathsleading to the same outcome (e.g., up-then-left vs. left-then-up).

Original: These scenarios are common among theenvironments typically studied in deep RL.
Revised: These scenarios are common amongthe environments typically studied in deep RL.

Original: Appreciating that policy null spaces can be large makesthe policy churn phenomenon more palatable, helping us reconcile that agents changing their minda million times about the best action (on average in each state) can have excellent performance.
Revised: Appreciating that policy null spaces can be largemakes the policy churn phenomenon more palatable, helping us reconcile that agents changing theirmind a million times about the best action (on average in each state) can have excellent performance.

Original: Aparticularly interesting null space is the one around the optimal policy Null( π ∗ ) : when it is large,there are many “safe” ways to keep changing the policy (possibly by a lot) after converging tomaximal performance, which is what we observe on, e.g.
Revised: A particularly interesting null space is the one around the optimal policy Null( π ∗ ) : when it is large,there are many “safe” ways to keep changing the policy (possibly by a lot) after converging tomaximal performance, which is what we observe on, e.g.

Original: , P ONG (see Figure 7), at least when ϵ > 0 .
Revised: , P ONG (see Figure 7), at least when ϵ > 0 .

Original: 4.3 Off-policy corrections in the presence of churn
Revised: 4.3 Off-policy corrections in the presence of churn

Original: Most types of off-policy correction are based on the gap between the data-generating behaviour policyµ , and a target policy π .
Revised: Most types of off-policy correction are based on the gap between the data-generating behaviour policyµ , and a target policy π .

Original: When doing multi-step updates, conservative methods truncate trajectoriesto bootstrap early compared to on-policy experience [15].
Revised: When doing multi-step updates, conservative methods truncate trajectoriesto bootstrap early compared to on-policy experience [25].

Original: To obtain the benefit of multi-step valuepropagation, the average truncation length cannot be too short, and thus a silent assumption is that µand π do not differ too frequently.
Revised: To obtain the benefit of multi-step valuepropagation, the average truncation length cannot be too short, and thus a silent assumption is that µand π do not differ too frequently.

Original: Empirically however, multi-step back-ups without any off-policycorrection can be surprisingly effective [10].
Revised: Empirically however, multi-step back-ups without any off-policycorrection can be surprisingly effective [20].

Original: We can now re-interpret these findings through the lensof policy churn: as the greedy policy changes much faster than expected, even a slight latency (a fewupdates) between the parameters of the data-generating policy and the current target policy leads tomassive truncation effect.
Revised: We can now re-interpret these findings through the lensof policy churn: as the greedy policy changes much faster than expected, even a slight latency (a fewupdates) between the parameters of the data-generating policy and the current target policy leads tomassive truncation effect.

Original: If consecutive greedy policies are (approximately) within the null-space ofeach other, the benefits of an uncorrected multi-step update may outweigh its cost.
Revised: If consecutive greedy policies are (approximately) within the null-space ofeach other, the benefits of an uncorrected multi-step update may outweigh its cost.

Original: It also suggeststhe possibility of new off-policy algorithms that exploit the knowledge of the churn phenomenon, bytruncating in a less aggressive fashion, motivated by the intuition that most rapidly changing policieslie within an (approximate) null space of each other (see Appendix A.5).
Revised: It also suggeststhe possibility of new off-policy algorithms that exploit the knowledge of the churn phenomenon, bytruncating in a less aggressive fashion, motivated by the intuition that most rapidly changing policieslie within an (approximate) null space of each other (see Appendix A.5).

Original: 4.4 The social dynamics of research
Revised: 4.4 The social dynamics of research

Original: If policy churn is indeed a hidden form of exploration, how did it come about?
Revised: If policy churn is indeed a hidden form of exploration, how did it come about?

Original: It seems unlikely thata useful mechanism emerges completely by chance.
Revised: It seems unlikely thata useful mechanism emerges completely by chance.

Original: An intriguing possibility is that policy churnis the effect of a gradual process of natural selection [6].
Revised: An intriguing possibility is that policy churnis the effect of a gradual process of natural selection [12].

Original: The hypothesis is that, if policy churnprovides benefits, algorithms that display some level of it would be favoured over their counterpartsin the inevitable engineering work surrounding the design of large-scale agents.
Revised: The hypothesis is that, if policy churnprovides benefits, algorithms that display some level of it would be favoured over their counterpartsin the inevitable engineering work surrounding the design of large-scale agents.

Original: In this view, RLpractitioners play the role of “nature” exerting a selective pressure that shapes algorithms across time.
Revised: In this view, RLpractitioners play the role of “nature” exerting a selective pressure that shapes algorithms across time.

Original: This process could be completely unconscious to the researchers involved: agents under-performingdue to weak exploration would be discarded in favour of agents that explored better using someform of (hidden) policy churn.
Revised: This process could be completely unconscious to the researchers involved: agents under-performingdue to weak exploration would be discarded in favour of agents that explored better using someform of (hidden) policy churn.

Original: Over time, the multiple degrees of freedom of large-scale agents(hyper-parameters, network architecture, etc.) would be tuned to reflect just the right amount of churn:enough to help in exploration, but not too much to make the overall learning process unstable.
Revised: Over time, the multiple degrees of freedom of large-scale agents(hyper-parameters, network architecture, etc.) would be tuned to reflect just the right amount of churn:enough to help in exploration, but not too much to make the overall learning process unstable.

Original: As appealing as the above hypothesis may be, as of now we do not have any evidence to supportit.
Revised: As appealing as the above hypothesis may be, as of now we do not have any evidence to supportit.

Original: Still, it is worth considering as it raises many interesting questions.
Revised: Still, it is worth considering as it raises many interesting questions.

Original: Are there other hiddeneffects that have been selected for over the years?
Revised: Are there other hiddeneffects that have been selected for over the years?

Original: How many good design choices got discardedbecause they happened not to promote policy churn (or other similarly hidden effects)?
Revised: How many good design choices got discardedbecause they happened not to promote policy churn (or other similarly hidden effects)?

Original: Is the AIresearch community narrowly focused on a handful of design templates that happen to induce someill-understood dynamics?
Revised: Is the AIresearch community narrowly focused on a handful of design templates that happen to induce someill-understood dynamics?

Original: This sort of question did not arise in the past, when agents were simpleenough that we could keep track of their functioning at the finest level of detail.
Revised: This sort of question did not arise in the past, when agents were simpleenough that we could keep track of their functioning at the finest level of detail.

Original: Now that deep RLagents have reached a certain level of complexity, any design choice may have a cascade effect whoseconsequences we do not anticipate.
Revised: Now that deep RLagents have reached a certain level of complexity, any design choice may have a cascade effect whoseconsequences we do not anticipate.

Original: This creates the perfect environment for the sort of selective process described above.
Revised: This creates the perfect environment for the sort of selectiveprocess described above.

Original: Acknowledging this possibility and being aware of it may be an importantstep to unveil hidden effects and perhaps turn them into more purposeful design.
Revised: Acknowledging this possibility and being aware of it may be an importantstep to unveil hidden effects and perhaps turn them into more purposeful design.

Original:  Related work
Revised:  Related work

Original: A phenomenon in the literature that is related, but not equivalent, to policy churn is that of policyoscillation or chattering [9, 2, 37].
Revised: A phenomenon in the literature that is related, but not equivalent, to policy churn is that of policyoscillation or chattering [16, 4, 48].

Original: Bertsekas and Tsitsiklis [2] define the greedy region of a functionq ∈ Q as the set of functions in Q that induce the same greedy policy as q .
Revised: Bertsekas and Tsitsiklis [4] define the greedy region of a functionq ∈ Q as the set of functions in Q that induce the same greedy policy as q .

Original: Each greedy regionhas an associated “fixed point”: the value function of the greedy policy induced by the functionsin it.
Revised: Each greedy regionhas an associated “fixed point”: the value function of the greedy policy induced by the functionsin it.

Original: Conversely, every value function is the fixed point of a greedy region.
Revised: Conversely, every value function is the fixed point of a greedy region.

Original: It is well known thatthe only value function that belongs to its own greedy region is the optimal value function q ∗ [2].
Revised: It is well known thatthe only value function that belongs to its own greedy region is the optimal value function q ∗ [4].

Original: However, when function approximation is used, projecting value functions onto the FA space maycreate cycles that repeat indefinitely, a phenomenon known as policy oscillation.
Revised: However, when function approximation is used, projecting value functions onto the FA space maycreate cycles that repeat indefinitely, a phenomenon known as policy oscillation.

Original: We do not thinkpolicy oscillation is a likely explanation for policy churn, because the approximators used in ourexperiments should have the capacity to approximate any value function to a reasonable level ofaccuracy, as indicated by the ablations in Figures 3 and 4 and Appendix A.1 that show neural networkswith more or wider layers exhibit more churn.
Revised: We do not thinkpolicy oscillation is a likely explanation for policy churn, because the approximators used in ourexperiments should have the capacity to approximate any value function to a reasonable level ofaccuracy, as indicated by the ablations in Figures 3 and 4 and Appendix A.1 that show neural networkswith more or wider layers exhibit more churn.

Original: We believe it is more instructive to think of eachupdate of the approximator as moving the value function across the boundaries of greedy regions.
Revised: We believe it is more instructive to think of eachupdate of the approximator as moving the value function across the boundaries of greedy regions.

Original: Asdiscussed, in many cases this has no effect on the agent’s performance, since the policies associatedwith neighbouring greedy regions may belong to each other’s null space (Section 4.2).
Revised: Asdiscussed, in many cases this has no effect on the agent’s performance, since the policies associatedwith neighbouring greedy regions may belong to each other’s null space (Section 4.2).

Original: Another body of related work is the literature on stochastic gradient descent, and how a relatively highamount of stochasticity can be beneficial to optimisation by overcoming local optima and convergingto flatter optima with better generalisation properties [12, 26, 14]; to the extent that it may even bebeneficial to inject additional noise with Langevin dynamics [39].
Revised: Another body of related work is the literature on stochastic gradient descent, and how a relatively highamount of stochasticity can be beneficial to optimisation by overcoming local optima and convergingto flatter optima with better generalisation properties [22, 37, 24]; to the extent that it may even bebeneficial to inject additional noise with Langevin dynamics [50].

Original: More loosely related are studiesof learning in animals, exhibiting large drift in synaptic or representation space [20, 27, 16, 7, 31],as well as evidence for highly variable behaviour policies that can get consolidated through salientdopamine events [3].
Revised: More loosely related are studiesof learning in animals, exhibiting large drift in synaptic or representation space [30, 38, 26, 13, 42],as well as evidence for highly variable behaviour policies that can get consolidated through salientdopamine events [9].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: Revisiting interpretations.
Revised: Revisiting interpretations.

Original: Nine years after the introduction of DQN [17], there are still phenomenain value-based deep RL that remain to be understood, with this paper putting the spotlight on one ofthem, which lies at the intersection of learning and exploration dynamics.
Revised: Nine years after the introduction of DQN [27], there are still phenomenain value-based deep RL that remain to be understood, with this paper putting the spotlight on one ofthem, which lies at the intersection of learning and exploration dynamics.

Original: In particular, we hope thatan awareness of the churn phenomenon will make researchers revisit some good ideas that may havebeen prematurely disregarded or under-valued, either because their promised exploration effect wastoo entangled with learning dynamics and the resulting churn, or because they improved the stabilityof learning dynamics at the expense of reduced exploration that undid the overall gains.
Revised: In particular, we hope thatan awareness of the churn phenomenon will make researchers revisit some good ideas that may havebeen prematurely disregarded or under-valued, either because their promised exploration effect wastoo entangled with learning dynamics and the resulting churn, or because they improved the stabilityof learning dynamics at the expense of reduced exploration that undid the overall gains.

Original: Churn beyond value-based RL.
Revised: Churn beyond value-based RL.

Original: An obvious follow-up question that we have not yet addressed iswhether policy churn is also an important effect beyond value-based algorithms.
Revised: An obvious follow-up question that we have not yet addressed iswhether policy churn is also an important effect beyond value-based algorithms.

Original: We could imaginethat actor-critic algorithms incur much less policy change, because stochastic policies change moresmoothly, or because various penalty terms keep the updated policy from deviating too much from itsprecursor.
Revised: We could imaginethat actor-critic algorithms incur much less policy change, because stochastic policies change moresmoothly, or because various penalty terms keep the updated policy from deviating too much from itsprecursor.

Original:  If this is indeed the case, that would indicate that exploration in these methods is differentas well, possibly with complementary advantages and disadvantages.
Revised: 6 If this is indeed the case, that would indicate that exploration in these methods is differentas well, possibly with complementary advantages and disadvantages.

Original: Similarly, various instances ofmodel-based RL [19, 32] may have less policy change, because the planning process could mitigatesome of the function approximation effects.
Revised: Similarly, various instances ofmodel-based RL [29, 43] may have less policy change, because the planning process could mitigatesome of the function approximation effects.

Original: Explicit and controllable churn.
Revised: Explicit and controllable churn.

Original: If policy churn is indeed a valuable and non-trivial explorationmechanism, then it may be costly to deliberately abandon, especially if its effect is complementary tosimpler noise-based exploration mechanisms.
Revised: If policy churn is indeed a valuable and non-trivial explorationmechanism, then it may be costly to deliberately abandon, especially if its effect turns out to becomplementary to simpler noise-based exploration mechanisms.

Original: Ideally we would want an explicit and controllablemechanism that produces the same kind of consistent, non-harmful exploration behaviour, but withoutsuch a tight entanglement with the learning dynamics.
Revised: Ideally we would want an explicitand controllable mechanism that produces the same kind of consistent, non-harmful explorationbehaviour, but without such a tight entanglement with the learning dynamics.

Original: The core benefit of such a division of labourwould be that practitioners can study, change or tune the learning process for what is valuable toit, namely stability or representation learning; and separately adapt or tune the exploration processbased on its differing criteria such as diversity, without having to trade one off against the other.
Revised: The core benefit ofsuch a division of labour would be that practitioners could study, change or tune the learning processfor what is valuable to it, namely stability or representation learning; and separately adapt or tune theexploration process based on its differing criteria such as diversity, without having to trade one offagainst the other.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] Section 3 discusses the generalityand applicability, Section 4 the ramifications.
Revised: [Yes] Section 3 discusses the generality and applicability, Section 4 the ramifications.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Section 4.4is specifically about the impact on the social dynamics of research.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [Yes] Section 4. is specifically about the impact on the social dynamics of research. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No] Our implementation relies on some proprietary code.
Revised: [No] Our implemen- tation relies on some proprietary code.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] Many of our experiments have 3 seeds across 15 gameswhich is enough to draw meaningful conclusions.
Revised: [Yes] Many of our experiments have 3 seeds across 15 games which is enough to draw meaningful conclusions.

Original: We report either error bars, showbox-plots, or show all seeds individually.
Revised: We report either error bars, show box-plots, or show all seeds individually.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix B.
Revised: [Yes] See Appendix B.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]