Original: The Phenomenon of Policy Churn
Revised: The Phenomenon of Policy Churn

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We identify and study the phenomenon of policy churn , that is, the rapid change ofthe greedy policy in value-based reinforcement learning.
Revised: We identify and study the phenomenon of policy churn , that is, the rapid change of the greedy policy in value-based reinforcement learning.

Original: Policy churn operates ata surprisingly rapid pace, changing the greedy action in a large fraction of stateswithin a handful of learning updates (in a typical deep RL set-up such as DQNon Atari).
Revised: Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari).

Original: We characterise the phenomenon empirically, verifying that it is notlimited to specific algorithm or environment properties.
Revised: We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties.

Original: A number of ablationshelp whittle down the plausible explanations on why churn occurs to just a handful,all related to deep learning.
Revised: A number of ablations help whittle down the plausible explanations on why churn occurs, the most likely one being deep learning with high-variance updates.

Original: Finally, we hypothesise that policy churn is a beneficialbut overlooked form of implicit exploration that casts ϵ -greedy exploration in afresh light, namely that ϵ -noise plays a much smaller role than expected
Revised: Finally, we hypothesise that policy churn is a potentially beneficial but overlooked form of implicit exploration , which casts ϵ -greedy exploration in a fresh light, namely that ϵ -noise plays a much smaller role than expected.

Original:  The Phenomenon
Revised:  The Phenomenon

Original: Reinforcement learning (RL) involves agents that incrementally update their policy.
Revised: Reinforcement learning (RL) involves agents that incrementally update their policy.

Original: This process isdriven by the objective of maximising reward, and based on experience that the agent generates viaexploration.
Revised: This process is driven by the objective of maximising reward, and based on experience that the agent generates via exploration.

Original: The sequence of policies π 0 , . . .
Revised: The sequence of policies π 0 , . . .

Original: , π k , . .
Revised: , π k , . .

Original: , π T usually starts from a randomly initialisedpolicy π 0 and aims to end at a near-optimal policy π T ≈ π ∗ .
Revised: , π T usually starts from a randomly initialised policy π 0 and aims to end at a near-optimal policy π T ≈ π ∗ .

Original: Ideally, steps in that sequence( π k → π k +1 ) are policy improvements that increase expected reward.
Revised: Ideally, steps in that sequence ( π k → π k +1 ) are policy improvements that increase expected reward.

Original: This paper studies the amount of policy change that goes along with such a policy update process(for a definition, see Section 1.1).
Revised: This paper studies the amount of policy change that goes along with such a policy update process (for a definition, see Section 1.1).

Original: In particular, it makes the core observation that policy change inpractice (as illustrated in some typical deep RL settings) is orders of magnitude larger than couldhave been expected, and stands in contrast to various reference algorithms (Sections 1.2 and 3.3).
Revised: In particular, it makes the core observation that policy change in practice (as illustrated in some typical deep RL settings) is orders of magnitude larger than could have been expected, and stands in contrast to various reference algorithms (Sections 1.2 and 3.3).

Original: Key observation 1: The greedy policy changes much more rapidly than you probably think.
Revised: Key observation 1: The greedy policy changes much more rapidly than you probably think.

Original: aa As a coarse magnitude for the impatient reader: in a typical run of DQN on Atari, the greedy policy changes in ≈ 10% of all states after a single gradient update (Figure 1 and Section 1.2).
Revised: a

Original: We dub this phenomenon “ policy churn ”, to highlight that most of this policy change may beunnecessary.
Revised: We dub this phenomenon “ policy churn ” to highlight that most of this policy change may be unnecessary.

Original: We study the phenomenon in depth, determining the range of deep RL scenarios itappears in, fleshing out its properties, and in the process narrowing the space of potential causes andmechanisms involved using a set of ablations (Section 3).
Revised: We study the phenomenon in depth, determining the range of deep RL scenarios it appears in, fleshing out its properties, and in the process narrowing the space of potential causes and mechanisms involved using a set of ablations (Section 3).

Original: Our second key message relates the phenomenon of churn to exploration, specifically in the contextof ϵ -greedy exploration (Section 2), with some more speculative ramifications in Section 4.
Revised: Our second key message relates the phenomenon of churn to exploration, specifically in the context of ϵ -greedy exploration (Section 2), with some more speculative ramifications in Section 4.

Original: Key observation 2: Policy churn is a significant driver of exploration.
Revised: Key observation 2: Policy churn is a significant driver of exploration.

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: 1.1 Defining policy change
Revised: 1.1 Defining policy change

Original: A policy is a function from states s ∈ S to a distribution over actions a ∈ A , where for the purposesof this paper A is discrete.
Revised: A policy is a function from states s ∈ S to a distribution over actions a ∈ A , where for the purposes of this paper A is discrete.

Original: We quantify the local, per-state policy change between policies π and π ′using the moved probability mass (i.e., the total variation distance):
Revised: We quantify the local, per-state policy change between policies π and π ′ using the moved probability mass (i.e., the total variation distance):

Original: [Equation]
Revised: [Equation]

Original: which satisfies 0 ≤ W ( π, π ′ | s ) ≤ 1 .
Revised: which satisfies 0 ≤ W ( π, π ′ | s ) ≤ 1 .

Original: When π and π ′ are greedy policies derived from a state-actionvalue function—that is, π  ∈ arg max a q ( s, a ) —then W ( π, π ′ | s )
Revised: When π and π ′ are greedy policies derived from a state-actionvalue function—that is, π ( s ) ∈ arg max a q ( s, a ) —then W ( π, π ′ | s )

Original: = 1 if the arg max action in state schanges upon replacing q by the function q ′ underlying π ′ , and W ( π, π ′ | s )
Revised: = 1 if the arg max action in state s changes upon replacing q by the function q ′ underlying π ′ , and W ( π, π ′ | s )

Original: = 

Revised: =

Original: otherwise.
Revised: otherwise.

Original: Similarreasoning applies more generally when both π and π ′ are deterministic policies.
Revised: Similar reasoning applies more generally when both π and π ′ are deterministic policies.

Original: We can aggregatepolicy change across all states, weighted by a state-distribution µ :
Revised: We can aggregate policy change across all states, weighted by a state-distribution d :

Original: [Equation]
Revised: [Equation]

Original: where a reasonable choice for µ is the empirical state distribution encountered during training (whichis non-stationary, but it could also be the stationary distribution of a fixed policy, or the uniformdistribution across all states, as discussed for some of the toy scenarios below).
Revised: where a reasonable choice for d is the empirical state distribution encountered during training (which is non-stationary, but it could also be the stationary distribution of a fixed policy, or the uniform distribution across all states, as discussed for some of the toy scenarios below).

Original: For greedy policies, W ( π, π ′ ) is simply the fraction of states where an arg max switch occurred.
Revised: For greedy policies (and uniform d ), W ( π, π ′ ) is simply the fraction of states where an arg max switch occurred.

Original: For settings where policy performance stabilises at some point t = P of training (e.g., hitting aperformance plateau, or converging to optimal behaviour), two additional metrics may be of interest,namely the cumulative policy change W 0: P until that point 1 , and the average policy change after thatpoint W + (which could be zero, e.g., if the process converges):
Revised: For settings where policy performance stabilises at some point t = P of training (e.g., hitting a performance plateau, or converging to optimal behaviour), two additional metrics may be of interest, namely the cumulative policy change W 0: P until that point 1 , and the average policy change after that point W + (which could be zero, e.g., if the process converges):

Original: [Equation]
Revised: [Equation]

Original: 1.2 Quantifying the phenomenon
Revised: 1.2 Quantifying the phenomenon

Original: Given an initial and a final policy, the process with the minimum amount of policy change is an oraclethat jumps from π 0 to π P in a single step ( P = 1 ).
Revised: Given an initial and a final policy, the process with the minimum amount of policy change is an oracle that jumps from π 0 to π P in a single step ( P = 1 ).

Original: By construction, this can incur a policy change ofat most one unit, W 0: P ≤ 1 , which is a form of lower bound.
Revised: By construction, this can incur a policy change of at most one unit, W 0: P ≤ 1 .

Original: In value-based deep RL agents, a naturaldefinition for the sequence of policies is to use the induced greedy policies π k  ∈ arg max a q θ k ( s, a )where θ k are the parameters of the Q-function at iteration k .
Revised: In value-based deep RL agents, a natural definition for the sequence of policies is to use the induced greedy policies π k ( s ) ∈ arg max a q θ k ( s, a ) where θ k are the parameters of the Q-function at iteration k .

Original: In agents that use a target network (inducing π ) that is an older copy of the online network (inducing π ′ ), it is easy to measure W ( π, π ′ )
Revised: In agents that use a target network (inducing π ) that is an older copy of the online network (inducing π ′ ), it is easy to measure W ( π, π ′ )

Original: by comparing their  arg max actions at the points in training where the target network lags behind byjust one update.
Revised: by comparing their [Figure] arg max actions at the points in training where the target network lags behind by just one update.

Original: Figure 1 shows typical values for W on a few Atari games, estimated by comparingonline and  target network policies on batches of experience sampled from the agent’s replay buffer.
Revised: Figure 1 shows typical values for W on a few Atari games, estimated by comparing the policies induced by online and (one update old) target networks, on batches of experience sampled from the agent’s replay buffer.

Original: Itis worth emphasising that with such rates of average policy change, of W ≈ 10% per update, themagnitude of whole-lifetime change becomes enormous: across training, an agent like DQN, whichperforms 10 7 updates, changes its greedy action a million times in each state (on average).
Revised: It is worth emphasising that with such rates of average policy change, of W ≈ 10% per update, the magnitude of whole-lifetime change becomes enormous: across training, an agent like DQN, which performs 10 7 updates, changes its greedy action a million times in each state (on average).

Original: A second striking result is the amount of policy change in late training, when the performance ofthe policy no longer changes (in the case of P ONG it is arguably optimal): there is still a change of
Revised: A second striking result is the amount of policy change in late training, when the performance of the policy no longer changes (in the case of P ONG it is arguably optimal): there is still a change of

Original: W + ≈ 5% per update (Figures 1 and 7).
Revised: W + ≈ 5% per update (Figures 1 and 14).

Original: This highlights that a lot (maybe most) of policy churn isnot directed at a policy improvement; we revisit this in Section 4.2.
Revised: This highlights that a lot (maybe most) of policy churn is not directed at a policy improvement; we revisit this in Section 4.2.

Original: Is this unexpected?
Revised: Is this unexpected?

Original: We conducted an informal survey of over 50 deep RL practitioners, includingsome of the inventors of DQN [27], for their estimate on how rapidly the policy changes in a typical Qlearning based setup.
Revised: We conducted an informal survey of over 50 deep RL practitioners, including three of the inventors of DQN [27], for their estimate on how rapidly the policy changes in a typical Qlearning based setup.

Original: The question was: For the greedy policy to change in 10% of allstates, how many learning updates does it take? (or equivalent).
Revised: The question was: For the greedy policy to change in 10% of all states, how many learning updates does it take? (or equivalent).

Original: The median responsewas 1 000 updates, with answers varying between 1 and 1 000 000 updates.
Revised: The median response was 1 000 updates, with answers varying between 1 and 1 000 000 updates.

Original: This deviates by threeorders of magnitude from the empirical value of 1 update (or W ≈ 0 .
Revised: This deviates by three orders of magnitude from the empirical value of 1 update (or W ≈ 0 .

Original: 1 , see Figure 1).
Revised: 1 , see Figure 1).

Original: Policy change in other settings.
Revised: Policy change in other settings.

Original: To appreciate the large empirical magnitudes of (cumulative)policy change we observe in deep RL, we can contrast them with a few alternative settings.
Revised: To appreciate the large empirical magnitudes of (cumulative) policy change we observe in deep RL, we can contrast them with a few alternative settings.

Original: Forexample, classic dynamic programming techniques such as value iteration or policy iteration [44],when applied to toy RL domains such as FourRooms, Catch, or DeepSea [33], accumulate W 0: P ≈ 1 ,not much more than the single-step oracle (see Appendix B.4).
Revised: For example, classic dynamic programming techniques such as value iteration or policy iteration [44], when applied to toy RL domains such as FourRooms, C ATCH , or DeepSea [33], accumulate W 0: P ≈ 1 , not much more than the single-step oracle (see Appendix B.4).

Original: The two main differences fromdeep RL are their tabular nature (no function approximation, FA) and non-incremental updates.
Revised: The two main differences from deep RL are their tabular nature (no function approximation, FA) and non-incremental updates.

Original: It ispossible to construct tabular settings with much larger policy change, with either incremental updates(Appendix A.3) or bootstrapping (Appendix A.4).
Revised: It is possible to construct tabular settings with much larger policy change, with either incremental updates (Appendix A.3) or bootstrapping (Appendix A.4).

Original: A minimalist example with non-linear FA andincremental updates is supervised learning on the MNIST dataset (the “policy” in this case are thepredicted label probabilities).
Revised: A minimalist example with non-linear FA and incremental updates is supervised learning on the MNIST dataset (the “policy” in this case are the predicted label probabilities).

Original: Training a digit classifier to convergence accumulates W 0: P ≈ 10 , thatis, the average input goes through 10 label switches (see Appendix B.5).
Revised: Training a digit classifier to convergence accumulates W 0: P ≈ 10 , that is, the average input goes through 10 label switches (see Appendix B.5).

Original: None of these examplesare fully satisfying, as they are not apples-to-apples comparisons; so in Section 3.3 we construct aspectrum of algorithmic variants that spans from tabular policy iteration (without churn), via tabular
Revised: None of these examples are fully satisfying, as they are not apples-to-apples comparisons; so in Section 3.3 we construct a spectrum of algorithmic variants that spans from tabular policy iteration (without churn), via tabular

Original: Q-learning to an approximation of DQN (with realistic magnitudes of churn).
Revised: Q-learning to an approximation of DQN (with realistic magnitudes of churn).

Original:  The Exploration Effect
Revised:  The Exploration Effect

Original: The next question is what happens when acting with the policies produced by a learning processwithhigh churn: it is likely that  the rapid change of greedy policies impacts the exploratory behaviourof agents.
Revised: We now turn to the impact of policy churn on agent behaviour: What happens when acting according to policies produced by a learning process that induces rapidly changing policies? In other words, what is the effect of policy churn on exploration?

Original: While each individual greedy policy would lead to a very narrow set of experience in a(nearly) deterministic environment like Atari, the fact that the greedy policy changes so rapidly (in10% of states per update, with one update every 16 environment frames in DoubleDQN) makes thedata distribution broad enough that in many circumstances, no other form of exploration (such asstochasticity introduced via an ϵ -greedy policy) is needed for good performance.
Revised: While each individual greedy policy would lead to a very narrow set of experience in a (nearly) deterministic environment like Atari, the fact that the greedy policy changes so rapidly (in 10% of states per update, with one update every 16 environment frames in DoubleDQN) makes for a broad data distribution. And in many circumstances this is sufficient for good performance, even in the absence of any other form of exploration, such as stochasticity introduced via an ϵ -greedy policy.

Original: To see thiseffectin practice across a range of Atari games, compare green (baseline) and gold ( ϵ = 0 ) [-curves in Figure 2. 
-]
Revised: Figure 2 shows this across a range of Atari games: compare green (baseline) and gold ( ϵ = 0 ) [-curves in Figure 2. 
-]curves.

Original: Conversely, removing (some of the) churn in the behaviour during training, which can be done byacting with the target network (updated only every 120 000 frames in the DoubleDQN agent) insteadof the online network, sometimes reduces performance even in the presence of ϵ -greedy exploration 
Revised: Conversely, removing (some of the) churn in the behaviour during training, which can be done by acting with the target network (updated only every 120 000 frames in the DoubleDQN agent) instead of the online network, sometimes reduces performance even in the presence of ϵ -greedy exploration ( blue in Figure 2). Additionally, we show that performance often collapses completely when both forms of exploration are removed ( ϵ = 0 and no churn, in red ).

Original: [Figure]
Revised: Figure 2 compares all four variants of exploration, indicating that the two sources of exploration have different contributions in different games.

Original: Sufficient exploration with ϵ = 0 .
Revised: Sufficient exploration with ϵ = 0 .

Original: The perhaps unintuitive observation of successful exploration with purely greedy policies has been made before, albeit implicitly.
Revised: The perhaps unintuitive observation of successful exploration with purely greedy policies has been made before, albeit implicitly.

Original: In particular, in the presence of certain alternative exploration methods such as noisy nets [15], no significant additional advantage is obtained from using ϵ > 0 [20].
Revised: In particular, in the presence of certain alternative exploration methods such as noisy nets [15], no significant additional advantage is obtained from using ϵ > 0 [20].

Original: Other works containing experimental variants with ϵ = 0demonstrated successful training in this setting [35, 39] without highlighting the result.
Revised: Other works containing experimental variants with ϵ =demonstrated successful training in this setting [35, 39], but did not highlight the result.

Original: Consistent behaviour and Thompson sampling.
Revised: Consistent behaviour and Thompson sampling.

Original: In considering the potential exploration benefits of a rapidly changing policy, it is worth qualitatively contrasting the resulting behaviour to that of anϵ -greedy policy with ϵ > 0 .
Revised: In considering the potential exploration benefits of a rapidly changing policy, it is worth qualitatively contrasting the resulting behaviour with that of an ϵ -greedy policy with ϵ > 0 .

Original: The latter generates high-frequency dithering, with uncorrelated random action decisions at consecutive states and the effect of most exploratory actions likely undone by the following greedy action [11].
Revised: The latter generates high-frequency dithering, with uncorrelated random action decisions at consecutive states and the effect of most exploratory actions likely undone by the following greedy action [11].

Original: By contrast, policy churn induced exploration can be expected to generate temporally correlated, consistent exploration (necessary, though perhaps not sufficient, to perform “deep” exploration, as in ensemble and Thompson sampling methods [45, 32]).
Revised: By contrast, policy-churn-induced exploration can be expected to generate temporally correlated, consistent exploration (necessary, though perhaps not sufficient, to perform “deep” exploration, as in ensemble and Thompson sampling methods [45, 32]).

Original: On the other hand, while ϵ -greedy exploration is explicitly unbiased in action space, policy churn likely prefers exploration across near-optimal actions (with respect to the current value function).
Revised: On the other hand, while ϵ -greedy exploration is explicitly unbiased in action space, policy churn likely prefers exploration across near-optimal actions (with respect to the current value function).

Original: This may be beneficial in some settings, for example when some actions are deadly, and high ϵ prevents long episodes.
Revised: This may be beneficial in some settings, for example when some actions are deadly, and high ϵ prevents long episodes.

Original: It may also be detrimental in others, where ϵ > 0 helps the agent to avoid getting stuck.
Revised: It may also be detrimental in others, where ϵ > 0 helps the agent get unstuck.

Original:  Potential Causes
Revised:  Potential Causes

Original: With the presence and impact of the churn phenomenon established, this section aims to provide additional depth.
Revised: With the presence and impact of the churn phenomenon established, this section aims to provide additional depth.

Original: First, we look at the generality of the effect in Section 3.1.
Revised: First, we look at the generality of the effect in Section 3.1.

Original: Second, we conduct investigations into the sensitivity of the phenomenon, with Section 3.2 showing ablations to the large- scale deep RL agents (with more material in Appendix A), and Section 3.3 taking the complementary[Table] approach of interpolating between dynamic programming and a DQN approximation on a toy domain.
Revised: Second, we conduct investigations into the sensitivity of the phenomenon, with Section 3.2 showing ablations to the large-scale deep RL agents (more material in Appendix A), and Section 3.3 taking the complementary approach of interpolating between dynamic programming and a DQN approximation on a toy domain.

Original: Finally, Section 3.4 synthesises the findings and postulates some compatible underlying mechanisms.
Revised: Finally, Section 3.4 synthesises the findings and postulates some compatible underlying mechanisms.

Original: 3.1 Breadth of prevalence and non-causes
Revised: 3.1 Breadth of prevalence and non-causes

Original: To judge the importance of the phenomenon, we need to establish whether it is specific to a narrowrange of settings, or prevalent in a variety of domains, algorithmic variants, and hyper-parameters.
Revised: To judge the importance of the phenomenon, we need to establish whether it is specific to a narrow range of settings, or prevalent in a variety of domains, algorithmic variants, and hyper-parameters.

Original: It turns out that this is easy to do, because the effect is very much not a subtle one.
Revised: It turns out that this is easy to do, because the effect is very much not a subtle one.

Original: In fact, policychurn is present in two very different deep RL agents, namely DoubleDQN [47] and (a variantof) R2D2 [21], both widely used for training on Atari (see Appendix B.1 and B.2 for agent andenvironment specifications).
Revised: In fact, policy churn is present in two very different deep RL agents, namely DoubleDQN [47] and (a variant [Table] of) R2D2 [21], both widely used for training on Atari (see Appendix B.1 and B.2 for agent and environment specifications).

Original: Despite the large differences between the algorithms summarised in
Revised: Despite the large differences between the algorithms summarised in

Original: Table 1, the magnitude of policy change is surprisingly similar, indicating that  policy churn is unlikelystrongly dependent on any of these specific choices.
Revised: Table 1, the magnitude of policy change is surprisingly similar, indicating that it is unlikely that policy churn strongly depends on any of these specific choices.

Original: The effect  alsoappears not specific to any environment: we measure similar magnitudes of policychange across a range of Atari games that vary in many dimensions, such as action space, rewardscale and sparsity, deadliness, etc.
Revised: The effect is also not specific to any environment: we measure similar magnitudes of policy change across a range of Atari games that vary in many dimensions, such as action space, reward scale and sparsity, deadliness, etc.

Original: Furthermore, it is present in all stages of training (see Figures 1and 5).
Revised: Furthermore, it is present in all stages of training (see Figures 1 and 15).

Original: Unsurprisingly it it highest in early learning, but it remains high during training and evenafter evaluation performance converges or stabilises (possibly at the optimal level as in P ONG ).
Revised: Unsurprisingly it is highest in early learning, but it remains high during training and even after evaluation performance converges or stabilises.

Original: 3.2 Ablations
Revised: 3.2 Ablations

Original: Redundant actions.
Revised: Redundant actions.

Original: A simple factor that could explain policy churn are redundant actions, i.e.,when nominally different actions have the same outcome (generally, orin a large fraction of states).
Revised: A simple factor that could explain policy churn are redundant actions, i.e., when nominally different actions have the same outcome (in all, or a large fraction of state space).

Original: This property varies widely by environment (see also [31]), but we observe similar levels of policychange across different ones (Figures 1 and 5).
Revised: This property varies widely by environment (see also [31]), but we observe similar levels of policy change across them (Figures 1 and 15).

Original: Also, when exposing the full Atari action set in allgames, creating explicit global redundancy, the churn magnitudes are not affected much (Figure 6).
Revised: Also, when exposing the full Atari action set in all games, creating explicit global redundancy, the churn magnitudes are not affected much (Figure 16).

Original: Appendix A.2 looks at the fine-grained aspect of which actions tend to be swapped for which others,and finds no structure easily related to the (known) equivalence relations (Figure 8).
Revised: Appendix A. looks at the fine-grained aspect of which actions tend to be swapped for which others, and finds no structure easily related to the (known) equivalence relations (Figure 10).

Original: In other words,most arg max changes are not happening between equivalent actions.
Revised: In other words, most arg max changes are not happening between equivalent actions.

Original: Small action gaps.
Revised: Small action gaps.

Original: Another potential factor for large policy churn with greedy policies can be theinterplay between FA and small action gaps (difference between largest and second-largest actionvalues): small approximation error can suffice for sub-optimal actions to overtake optimal ones.
Revised: Another potential factor for large policy churn with greedy policies can be the interplay between FA and small action gaps (difference between largest and second-largest action values): small approximation error can suffice for sub-optimal actions to overtake optimal ones.

Original: Thishypothesis predicts that value learning methods inducing larger action gaps (e.g., Advantage Learning(AL, [3]) which artificially lowers the values of sub-optimal actions) could reduce policy churn.
Revised: This hypothesis predicts that value learning methods inducing larger action gaps (e.g., Advantage Learning (AL, [3]) which artificially lowers the values of sub-optimal actions) could reduce policy churn.

Original: Figure 3 (right) shows that indeed policy churn is diminished substantially by AL, correlating with an increase of action gaps (which consistently grows under AL, see Figure 9).
Revised: Figure 3 (right) shows that indeed policy churn is decreased substantially by AL, correlating with an increase of action gaps (which consistently grows under AL, see Figure 17).

Original: Curiously, this doesnot seem to severely diminish the remaining policy churn’s effectiveness for exploration: Figure 10shows successful training of an AL-DQN with ϵ = 0 .
Revised: Curiously, this does not seem to severely diminish the remaining policy churn’s effectiveness for exploration: Figureshows successful training of an AL-DQN with ϵ = 0 .

Original: Non-stationary state distribution.
Revised: Non-stationary state distribution.

Original: Another explanatory hypothesis for the observed large magnitude of policy churn relates it to the non-stationary data distribution caused by the evolving datagenerating policy.
Revised: Another explanatory hypothesis for the observed large magnitude of policy churn relates it to the non-stationary data distribution caused by the policy generating the data, which is evolving.

Original: Even when the policy seems tohave converged with near-optimal performance,like in P ONG , policy churn may concentrate on states where action decisions are inconsequential(where multiple actions have near-equal value), causing non-stationarity in the data distribution andthereby driving further policy churn.
Revised: Even when the agent converges to near-optimal performance, like in P ONG , it may happen that the policy keeps changing on states where actions are inconsequential (where multiple actions have near-equal value), causing non-stationarity in the data distribution and thereby driving further policy churn.

Original: To test this, we utilize the “forked tandem” setting from [34], inwhich a high-performing policy is trained on the stationary data distribution generated by its initialsnapshot.
Revised: To test this, we utilize the “forked tandem” setting from [34], in which a high-performing policy is trained on the stationary data distribution generated by its initial [Figure] a [Figure]

Revised: t a

Original: As can be seen in Figures 3 (right) and 11, the high level of policy churn is still preservedin this stationary-data regime, ruling out data non-stationarity as the main driver of the phenomenon.
Revised: [Figure] p ( [Figure] t o p

Revised: [Figure]

Revised: a c t i o n s )

Revised: Advantage

Revised: [Figure]

Revised: snapshot (see below for more details).

Revised: As can be seen in Figures 3 (left) and 19, the high level of policy churn is still preserved in this stationary-data regime, ruling out data non-stationarity as the main driver of the phenomenon.

Original: Non-stationary targets.
Revised: Non-stationary targets.

Original: Temporal difference (TD) learning can give rise to another form of nonstationarity, as the bootstrap targets change at the pace of the target value function, which is preservedeven in the forked tandem setting.
Revised: Temporal difference (TD) learning can give rise to another form of nonstationarity, as the bootstrap targets change at the pace of the target value function, an aspect that is preserved even in the forked tandem setting.

Original: To sidestep this, a simple control experiment uses the same setupbut with Monte-Carlo returns as learning targets, turning it into pure policy evaluation via supervisedregression (with noisy targets).
Revised: To sidestep this, a simple control experiment uses the same setup but with Monte-Carlo returns as learning targets, turning it into pure policy evaluation via supervised regression (with noisy targets).

Original: The results (Figures 3, right, and 12) show a similar level of churn tothe Q-learning updates, indicating that the phenomenon is not specific to TD-based algorithms.
Revised: The results (Figures 3, left, and 20) show a similar level of churn to the Q-learning updates, indicating that the phenomenon is not specific to TD-based algorithms.

Original: Decoupled acting and target networks.
Revised: Decoupled acting and target networks.

Original: Using a purely greedy policy based on the agent’s targetnetwork and varying the frequency with which it is synchronized to the online network allows usto assess how much policy churn is beneficial for exploration.
Revised: To assess how much policy churn is beneficial for exploration, we devise the following setup: A greedy policy based on the agent’s target network drives behaviour, and we vary the frequency with which it is synchronized to the online network.

Original: To avoid conflation with an adverseeffect on learning stability, we keep the regular target network update frequency constant, while usingan additional acting network , periodically copied from the online network, for behaviour generationalone.
Revised: To avoid conflation with an adverse effect on learning stability, we keep the regular target network update frequency constant, while using an additional acting network , periodically copied from the online network, for behaviour generation alone.

Original: Figure 13 shows that an acting network updated more than every ≈ 1 000 gradient updatesachieves most of the exploration benefits of acting with the online network, though in some gameshigher frequencies yield further benefit.
Revised: Figure 21 shows that an acting network updated more than every ≈ 1 000 gradient updates achieves most of the exploration benefits of acting with the online network, though in some games higher frequencies yield further benefit.

Original: This implies that the amount of policy change neededfor exploration is much smaller than what is generated by DQN’s learning process (otherwise theobserved collapse would happen at much smaller update intervals).
Revised: This implies that the amount of policy change needed for exploration is much smaller than what is generated by DQN’s learning process (otherwise the observed collapse would happen at much smaller update intervals).

Original: Self-correction and the tandem effect.
Revised: Self-correction and the tandem effect.

Original: In [34] a phenomenon dubbed the “tandem effect” wasobserved: the failure of a deep RL agent to adequately learn from the training data generated by adifferent instance of the same agent, highlighting the importance of self-correction by interactivelygenerated data from the policy being trained.
Revised: In [34] a phenomenon dubbed the “tandem effect” was observed: the failure of a deep RL agent to adequately learn from the training data generated by a different instance of the same agent, highlighting the importance of self-correction by interactively generated data from the policy being trained.

Original: One of their settings, the “forked tandem”, starts with acopy of a high-performing policy and uses data sampled from its stationary distribution to continuetraining; even this apparently benign scenario leads to instability and potential collapse of the trainedpolicy.
Revised: One of their settings, the “forked tandem”, starts with a copy of a high-performing policy and uses data sampled from its stationary distribution to continue training; even this apparently benign scenario leads to instability and potential collapse of the trained policy.

Original: Policy churn may provide a partial mechanistic explanation for the origin of the instability,showing that rapid policy change can be expected at all stages of training.
Revised: Policy churn may provide a partial mechanistic explanation for the origin of the instability, showing that rapid policy change can be expected at all stages of training.

Original: The observation that thetrained policy changes on a significant proportion of states at every update (performed on a negligiblysmall sample, a single minibatch of 32 state transitions) supports the hypothesis from [34] thaterroneous extrapolation or over-generalization may play a key role in causing deviation and instabilityand producing the tandem effect in the absence of corrective training signal from self-generated data.
Revised: The observation that the trained policy changes on a significant proportion of states at every update (performed on a negligibly small sample, a single minibatch of 32 state transitions) supports the hypothesis from [34] that erroneous extrapolation or over-generalization may play a key role in causing deviation and instability and producing the tandem effect in the absence of corrective training signal from self-generated data.

Original: Analogously to results in [34] we observe that the magnitude of policy churn is highly correlatedwith the depth of the trained function approximator, further supporting this hypothesis; see Figures 3(right) and 14, which show specifically how policy change drops dramatically in a linear FA regime. 
Revised: Analogously to results in [34] we observe that the magnitude of policy churn is highly correlated with the depth of the trained function approximator, further supporting this hypothesis; see Figures(left) and 22, which show specifically how policy change drops dramatically in a linear FA regime. [Figure]

Original: 3.3 Detailed case study: Catch
Revised: 3.3 Detailed case study: C ATCH

Original: Catch is a toy environment where the total number of states is small enough to be amenable toground-truth dynamic programming approaches using the explicit matrix of transition probabilities,while providing an observation space that requires non-linear FA.
Revised: C ATCH is a toy environment where the total number of states is small enough to be amenable to ground-truth dynamic programming approaches using the explicit matrix of transition probabilities, while providing an observation space that requires non-linear FA to represent the optimal value function.

Original: We construct a spectrum of settingsthat all learn the optimal policy after some P iterations.For each we measure cumulative policy [Figure] [Figure]
Revised: We construct a spectrum of settings that all learn the optimal policy after some P iterations.

Original:  change W 0: P and average change after convergence W+ .
Revised: For each we measure cumulative policy change W 0: P and average change after convergence W

Original:  Main results are shown in Figure 4.
Revised: + (Eq. 3). Main results are shown in Figure 4.

Original: Sittingat one end of the spectrum, exact value iteration converges after P = 10 steps with W 0: P = 0 . {+
+}
Revised: Sitting at one end of the spectrum, exact value iteration converges after P = 10 steps with W 0: P = 0 . {+
+}

Original: 09(because the initial policy is random, and the optimal policy has ties in most states).
Revised: (because the initial policy is random, and the optimal policy has ties in most states).

Original: The next simplestsetting is tabular Q-learning with incremental updates (here and elsewhere, hyperparameters like thelearning rate are tuned for fast convergence to optimal performance, see Appendix B.3 for details).
Revised: The next simplest setting is tabular Q-learning with incremental updates (here and elsewhere, hyperparameters like the learning rate are tuned for fast convergence to optimal performance, see Appendix B.3 for details).

Original: The next steps add in non-linear FA (shallow, then deep).
Revised: The next steps add in non-linear FA (shallow withhidden layers, then deep, with 3 ).

Original: At the other end of the spectrum is anapproximation to DQN which includes experience replay, mini-batches, a deep neural network, and anadvanced optimiser (RMSProp [46]).
Revised: At the other end of the spectrum is an approximation to DQN which includes experience replay, mini-batches, a deep neural network, and an advanced optimiser (RMSProp [46]).

Original: Figure 4 also shows intermediate variations such as supervisedregression to q ∗ , and different optimisers (e.g., Adam [23]); additional results in Appendix A.1.
Revised: Figure 4 also shows intermediate variations such as supervised regression to q ∗ , and different optimisers (e.g., Adam [23]); additional results in Appendix A.1.

Revised: Overall, the experiments in this section show that, among the factors we considered, the presence of function approximation is by far the aspect that correlates the most with the occurrence of the phenomenon of policy churn.

Original: 3.4 Mechanistic hypothesis
Revised: 3.4 Mechanistic hypothesis

Original: In an attempt to synthesise all the evidence presented so far, we propose that the observed policychurn is primarily a result of two components that need to be present jointly:
Revised: In an attempt to synthesise all the evidence presented so far, we propose that the observed policy churn is primarily a result of two components that need to be present jointly:

Original: Non-linear, global function approximation (such as deep neural networks), where each update can affect all states and all actions.
Revised: Non-linear, global function approximation (such as deep neural networks), where each update can affect all states and all actions.

Original: A learning process with a high amount of noise.
Revised: A learning process with a high amount of noise.

Original: This could have multiple sources: stochastic optimisation (e
Revised: This could have multiple sources: stochastic optimisation (e

Original: g. small batch sizes, large learning rates), noisy learning targets, non-stationary data (or targets, as with bootstrapping), an imbalanced data distribution (e g., seeing some actionsfar more often than others).
Revised: g. small batch sizes, large learning rates), noisy learning targets, non-stationary data (or targets, as with bootstrapping), an imbalanced data distribution (e.g., seeing some actions more often than others).

Original: In other words, the rapid policy change dubbed churn is the symptom of high-variance updates to aglobal function approximator.
Revised: In other words, the rapid policy change dubbed churn is the symptom of high-variance updates to a global function approximator, and neither aspect in isolation is enough to promote it.

Original: A compounding effect could be a mismatch between the regressionloss driving the learning process and policy space, which is what matters for performance and forexploration (and is measured here).

Original: While supervised learning does not have this mismatch, it couldconceivably have large “policy change” too.

Original: An in-depth treatment exceeds the scope for this paper,but preliminary results on MNIST (Appendix B.5) indicate no large churn in the supervised setting.

Original: Where do we go from here?
Revised: Where do we go from here?

Original: 4.1 Learning at the edge of chaos
Revised: 4.1 Learning at the edge of chaos

Original: Deep learning has a well-known trade-off between speed of learning and stability that incentivisestuning the learning dynamics to be near the edge of chaos 5 [10].
Revised: Deep learning has a well-known trade-off between speed of learning and stability that incentivises tuning the learning dynamics to be near the edge of chaos 6 [10].

Original: The presence of policy churncould enrich this picture in two ways for the case of deep RL.
Revised: The presence of policy churn could enrich this picture in two ways for the case of deep RL.

Original: First, to the extent that rapid policychange helps drive exploration (Section 2), this is an additional incentive to keep learning dynamicssufficiently noisy.
Revised: First, to the extent that rapid policy change helps drive exploration (Section 2), this is an additional incentive to keep learning dynamics sufficiently noisy.

Original: Second, to the extent thatself-correction is central to value-based learning[34],which relies on  the actions to be corrected being picked by a greedy policy, this also encourages alarge amount of policy change.
Revised: Second, to the extent that value-based learning relies on self-correction [34], that is, the actions to be corrected need to be picked by a greedy policy, this also encourages a large amount of policy change.

Original: Circumstantial evidence for these is that value-based RL tends torequire additional stabilisation mechanisms (e.g., target networks), less common in policy-based RL.
Revised: Circumstantial evidence for these is that value-based RL tends to require additional stabilisation mechanisms (e.g., target networks), less common in policy-based RL.

Original: 4.2 Policy null-spaces
Revised: 4.2 Policy null-spaces

Original: Policy change is necessary for policy improvement, but does not necessarily imply change inperformance.
Revised: Policy change is necessary for policy improvement, but does not necessarily imply change in performance.

Original: We define the space of policies that have the same value(in all states) as a referencepolicy π  as its null space Null( π ) .
Revised: We define the space of policies that have the same value as a reference policy π (in all states) as its null space Null( π ) .

Original: We can quantify its diameter
Revised: One way to quantify it is the diameter

Original: [Equation]
Revised: [Equation]

Original: as the largest policy change possible between any pair of policies within it.
Revised: as the largest policy change possible between any pair of policies within it.

Original: Typical scenarios withlarge null spaces have states in which actions have no effect (e.g., move actions while falling),multiple actions with the same effect (globally, or locally in part of state space), or multiple pathsleading to the same outcome (e.g., up-then-left vs. left-then-up).
Revised: Typical scenarios with large null spaces have states in which actions have no effect (e.g., move actions while falling), multiple actions with the same effect (globally, or locally in part of state space), or multiple paths leading to the same outcome (e.g., up-then-left vs. left-then-up).

Original: These scenarios are common amongthe environments typically studied in deep RL.
Revised: These scenarios are common among the environments typically studied in deep RL.

Original: Appreciating that policy null spaces can be largemakes the policy churn phenomenon more palatable, helping us reconcile that agents changing theirmind a million times about the best action (on average in each state) can have excellent performance.
Revised: Appreciating that policy null spaces can be large makes the policy churn phenomenon more palatable, helping us reconcile that agents changing their mind a million times about the best action (on average in each state) can have excellent performance.

Original: A particularly interesting null space is the one around the optimal policy Null( π ∗ ) : when it is large,there are many “safe” ways to keep changing the policy (possibly by a lot) after converging tomaximal performance, which is what we observe on, e.g. , P ONG (see Figure 7), at least when ϵ > 0 .
Revised: A particularly interesting null space is the one around the optimal policy Null( π ∗ ) : when it is large, there are many “safe” ways to keep changing the policy (possibly by a lot) after converging to maximal performance, which is what we observe on, e.g., P ONG (see Figure 14), at least when ϵ > 0 .

Original: 4.3 Off-policy corrections in the presence of churn
Revised: 4.3 Off-policy corrections in the presence of churn

Original: Most types of off-policy correction are based on the gap between the data-generating behaviour policyµ , and a target policy π .
Revised: Most types of off-policy correction are based on the gap between the data-generating behaviour policy µ , and a target policy π .

Original: When doing multi-step updates, conservative methods truncate trajectoriesto bootstrap early compared to on-policy experience [25].
Revised: When doing multi-step updates, conservative methods truncate trajectories to bootstrap early compared to on-policy experience [25].

Original: To obtain the benefit of multi-step valuepropagation, the average truncation length cannot be too short, and thus a silent assumption is that µand π do not differ too frequently.
Revised: To obtain the benefit of multi-step value propagation, the average truncation length cannot be too short, and thus a silent assumption is that µ and π do not differ too frequently.

Original: Empirically however, multi-step back-ups without any off-policycorrection can be surprisingly effective [20].
Revised: Empirically however, multi-step back-ups without any off-policy correction can be surprisingly effective [20].

Original: We can now re-interpret these findings through the lensof policy churn: as the greedy policy changes much faster than expected, even a slight latency (a fewupdates) between the parameters of the data-generating policy and the current target policy leads tomassive truncation effect.
Revised: We can now re-interpret these findings through the lens of policy churn: as the greedy policy changes much faster than expected, even a slight latency (a few updates) between the parameters of the data-generating policy and the current target policy leads to massive truncation effect.

Original: If consecutive greedy policies are (approximately) within the null-space ofeach other, the benefits of an uncorrected multi-step update may outweigh its cost.
Revised: If consecutive greedy policies are (approximately) within the null-space of each other, the benefits of an uncorrected multi-step update may outweigh its cost.

Original: It also suggeststhe possibility of new off-policy algorithms that exploit the knowledge of the churn phenomenon, bytruncating in a less aggressive fashion, motivated by the intuition that most rapidly changing policieslie within an (approximate) null space of each other (see Appendix A.5).
Revised: It also suggests the possibility of new off-policy algorithms that exploit the knowledge of the churn phenomenon, by truncating in a less aggressive fashion, motivated by the intuition that majority of rapidly changing policies lie within an (approximate) null space of each other (see Appendix A.5).

Original: 4.4 The social dynamics of research
Revised: 4.4 The social dynamics of research

Original: If policy churn is indeed a hidden form of exploration, how did it come about?
Revised: If policy churn is indeed a hidden form of exploration, how did it come about?

Original: It seems unlikely thata useful mechanism emerges completely by chance.
Revised: It seems unlikely that a useful mechanism emerges completely by chance.

Original: An intriguing possibility is that policy churnis the effect of a gradual process of natural selection [12].
Revised: An intriguing possibility is that policy churn is the effect of a gradual process of natural selection [12].

Original: The hypothesis is that, if policy churnprovides benefits, algorithms that display some level of it would be favoured over their counterpartsin the inevitable engineering work surrounding the design of large-scale agents.
Revised: The hypothesis is that, if policy churn provides benefits, algorithms that display some level of it would be favoured over their counterparts in the inevitable engineering work surrounding the design of large-scale agents.

Original: In this view, RLpractitioners play the role of “nature” exerting a selective pressure that shapes algorithms across time.
Revised: In this view, RL practitioners play the role of “nature” exerting a selective pressure that shapes algorithms across time.

Original: This process could be completely unconscious to the researchers involved: agents under-performingdue to weak exploration would be discarded in favour of agents that explored better using someform of (hidden) policy churn.
Revised: This process could be completely unconscious to the researchers involved: agents under-performing due to weak exploration would be discarded in favour of agents that explored better using some form of (hidden) policy churn.

Original: Over time, the multiple degrees of freedom of large-scale agents(hyper-parameters, network architecture, etc.) would be tuned to reflect just the right amount of churn:enough to help in exploration, but not too much to make the overall learning process unstable.
Revised: Over time, the multiple degrees of freedom of large-scale agents (hyper-parameters, network architecture, etc.) would be tuned to reflect just the right amount of churn: enough to help in exploration, but not too much to make the overall learning process unstable.

Original: As appealing as the above hypothesis may be, as of now we do not have any evidence to supportit.
Revised: As appealing as the above hypothesis may be, as of now we do not have any evidence to support it.

Original: Still, it is worth considering as it raises many interesting questions.
Revised: Still, it is worth considering, as it raises many interesting questions.

Original: Are there other hiddeneffects that have been selected for over the years?
Revised: Are there other hidden effects that have been selected for over the years?

Original: How many good design choices got discardedbecause they happened not to promote policy churn (or other similarly hidden effects)?
Revised: How many good design choices got discarded because they happened not to promote policy churn (or other similarly hidden effects)?

Original: Is the AIresearch community narrowly focused on a handful of design templates that happen to induce someill-understood dynamics?
Revised: Is the AI research community narrowly focused on a handful of design templates that happen to induce some ill-understood dynamics?

Original: This sort of question did not arise in the past, when agents were simpleenough that we could keep track of their functioning at the finest level of detail.
Revised: This sort of question did not arise in the past, when agents were simple enough that we could keep track of their functioning at the finest level of detail.

Original: Now that deep RLagents have reached a certain level of complexity, any design choice may have a cascade effect whoseconsequences we do not anticipate.
Revised: Now that deep RL agents have reached a certain level of complexity, any design choice may have a cascade effect whose consequences we do not anticipate.

Original: This creates the perfect environment for the sort of selectiveprocess described above.
Revised: This creates the perfect environment for the sort of selective process described above.

Original: Acknowledging this possibility and being aware of it may be an importantstep to unveil hidden effects and perhaps turn them into more purposeful design.
Revised: Acknowledging this possibility and being aware of it may be an important step toward unveiling hidden effects, and perhaps turning them into more purposeful design.

Original:  Related work
Revised:  Related work

Original: A phenomenon in the literature that is related, but not equivalent, to policy churn is that of policyoscillation or chattering [16, 4, 48].
Revised: A phenomenon in the literature that is related to policy churn, but not equivalent to it, is that of policy oscillation or chattering [16, 4, 48].

Original: Bertsekas and Tsitsiklis [4] define the greedy region of a functionq ∈ Q as the set of functions in Q that induce the same greedy policy as q .
Revised: Bertsekas and Tsitsiklis [4] define the greedy region of a function q ∈ Q as the set of functions in Q that induce the same greedy policy as q .

Original: Each greedy regionhas an associated “fixed point”: the value function of the greedy policy induced by the functionsin it.
Revised: Each greedy region has an associated “fixed point”: the value function of the greedy policy induced by the functions in it.

Original: Conversely, every value function is the fixed point of a greedy region.
Revised: Conversely, every value function is the fixed point of a greedy region.

Original: It is well known thatthe only value function that belongs to its own greedy region is the optimal value function q ∗ [4].
Revised: It is well known that the only value function that belongs to its own greedy region is the optimal value function q ∗ [4].

Original: However, when function approximation is used, projecting value functions onto the FA space maycreate cycles that repeat indefinitely, a phenomenon known as policy oscillation.
Revised: However, when function approximation is used, projecting value functions onto the FA space may create cycles that repeat indefinitely, a phenomenon known as policy oscillation.

Original: We do not thinkpolicy oscillation is a likely explanation for policy churn, because the approximators used in ourexperiments should have the capacity to approximate any value function to a reasonable level ofaccuracy, as indicated bythe ablations in Figures 3 and 4 and Appendix A.1 that show neural networkswith more or wider layers exhibit more churn.
Revised: We do not think policy oscillation is a likely explanation for policy churn, because the approximators used in our experiments should have the capacity to approximate any value function to a reasonable level of accuracy. This is corroborated by Figures 3 and 4 and Appendix A.1, which show neural networks with more or wider layers exhibit more churn.

Original: We believe it is more instructive to think of eachupdate of the approximator as moving the value function across the boundaries of greedy regions.
Revised: We believe it is more instructive to think of each update of the approximator as moving the value function across the boundaries of greedy regions.

Original: Asdiscussed, in many cases this has no effect on the agent’s performance, since the policies associatedwith neighbouring greedy regions may belong to each other’s null space (Section 4.2).
Revised: As discussed, in many cases this has no effect on the agent’s performance, since the policies associated with neighbouring greedy regions may belong to each other’s null space (Section 4.2).

Original: Another body of related work is the literature on stochastic gradient descent, and how a relatively highamount of stochasticity can be beneficial to optimisation by overcoming local optima and convergingto flatter optima with better generalisation properties [22, 37, 24]; to the extent that  it may even bebeneficial to inject additional noise with Langevin dynamics [50].
Revised: Another body of related work is the literature on stochastic gradient descent, and how a relatively high amount of stochasticity can be beneficial to optimisation by overcoming local optima and converging to flatter optima with better generalisation properties [22, 37, 24]; this is such a prominent effect that in some cases it is even beneficial to inject additional noise with Langevin dynamics [50].

Original: More loosely related are studiesof learning in animals, exhibiting large drift in synaptic or representation space [30, 38, 26, 13, 42],as well as evidence for highly variable behaviour policies that can get consolidated through salientdopamine events [9].
Revised: More loosely related are studies of learning in animals, exhibiting large drift in synaptic or representation space [30, 38, 26, 13, 42], as well as evidence for highly variable behaviour policies that can get consolidated through salient dopamine events [9].

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: Revisiting interpretations.
Revised: Revisiting interpretations.

Original: Nine years after the introduction of DQN [27], there are still phenomenain value-based deep RL that remain to be understood, with this paper putting the spotlight on one ofthem, which lies at the intersection of learning and exploration dynamics.
Revised: Nine years after the introduction of DQN [27], there are still phenomena in value-based deep RL that remain to be understood, with this paper putting the spotlight on one of them that lies at the intersection of learning and exploration dynamics.

Original: In particular, we hope thatan awareness of the churn phenomenon will make researchers revisit some good ideas that may havebeen prematurely disregarded or under-valued, either because their promised exploration effect wastoo entangled with learning dynamics and the resulting churn, or because they improved the stabilityof learning dynamics at the expense of reduced exploration that undid the overall gains.
Revised: In particular, we hope that an awareness of the churn phenomenon will make researchers revisit some good ideas that may have been prematurely disregarded or under-valued, either because their promised exploration effect was too entangled with learning dynamics and the resulting churn, or because they improved the stability of learning dynamics at the expense of reduced exploration that undid the overall gains.

Original: Churn beyond value-based RL.
Revised: Churn beyond value-based RL.

Original: An obvious follow-up question that we have not yet addressed iswhether policy churn is also an important effect beyond value-based algorithms.
Revised: An obvious follow-up question is whether policy churn is also an important effect beyond value-based algorithms.

Original: We could imaginethat actor-critic algorithms incur much less policy change, because stochastic policies change moresmoothly, or because various penalty terms keep the updated policy from deviating too much from itsprecursor.
Revised: We could imagine that actor-critic algorithms incur much less policy change, because stochastic policies change more smoothly, or because various penalty terms keep the updated policy from deviating too much from its precursor.

Original: 6 If this is indeed the case, that would indicate that exploration in these methods is differentas well, possibly with complementary advantages and disadvantages.
Revised: If that were the case, it would indicate that exploration in these agents differs as well, possibly with complementary advantages and disadvantages.

Original: Similarly, various instances ofmodel-based RL [29, 43] may have less policy change, because the planning process could mitigatesome of the function approximation effects. 
Revised: Similarly, various instances of model-based RL [29, 43] may have less policy change, because the planning process could mitigate some of the function approximation effects. We leave these investigations to future work, but include one preliminary result (Figure 26) that seems to hint at churn being high in some actor-critic agents as well.

Original: Explicit and controllable churn.
Revised: Explicit and controllable churn.

Original: If policy churn is indeed a valuable and non-trivial explorationmechanism, then it may be costly to deliberately abandon, especially if its effect turns out to becomplementary to simpler noise-based exploration mechanisms.
Revised: If policy churn is indeed a valuable and non-trivial exploration mechanism, then it may be costly to deliberately abandon it, especially if its effect turns out to be complementary to simpler noise-based exploration mechanisms.

Original: Ideally we would want an explicitand controllable mechanism that produces the same kind of consistent, non-harmful explorationbehaviour, but without such a tight entanglement with the learning dynamics.
Revised: Ideally we would want an explicit and controllable mechanism that produces the same kind of consistent, non-harmful exploration behaviour, but without such a tight entanglement with the learning dynamics.

Original: The core benefit ofsuch a division of labour would be that practitioners could study, change or tune the learningprocessfor what is valuable to it, namely stability or representation learning; and separately adapt or tune theexploration process based on its differing criteria such as diversity, without having to trade one offagainst the other.
Revised: The core benefit of such a division of labour would be that practitioners could study, change or tune the learning and exploration processes separately, without having to make arbitrary trade-offs (e.g., around stability, diversity, representations) that inevitably arise when they are considered in combination.

Original: Checklist
Revised: Acknowledgments

Original: For all authors...
Revised: The ideas presented here were refined in discussion with numerous of our DeepMind colleagues. Will Dabney, Joseph Modayil and Matteo Hessel helped improve the paper with detailed feedback, and we thank David Silver, Diana Borsa, Miruna Pîslar, Claudia Clopath, Vlad Mnih, Iurii Kemaev, Junhyuk Oh, Bilal Piot, Greg Farquhar, Dan Calian, Hado van Hasselt and Alex Pritzel for their input. We also thank the anonymous NeurIPS reviewers for their suggestions, as well as the many RLDM and ICML attendees whom we put our guesstimate survey question to.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: This appendix contains a number of figures that are already referenced from within the main paper. A.1 Additional results on C ATCH

Original: contributions and scope?
Revised: Complementing Figure 4 in Section 3.3 are Figures 5, 6, and 7 which show policy change for additional variants, in particular wider networks, behavioural cloning of π ∗ , and learning rate annealing.

Original: [Yes]
Revised: Policy change per state. Figure 8 shows the policy change per state averaged over different periods of training and 1 000 seeds for the “DQN-like” agent with RMSProp optimiser.

Original: (b) Did you describe the limitations of your work?
Revised: See Appendix B. for exact hyper-parameters. Note that episodes in C ATCH always start with the paddle in the centre.

Original: [Yes] Section 3 discusses the generality
Revised: This means some states shown in the plots in Figure 8 are not actually possible, in particular states corresponding to the dark top row of cells in all but the central column of plots. Another consequence is that starting states (corresponding to the top row of cells in the middle column of plots) have disproportionately more policy change. Indeed, in a version of the environment where the paddle is initialised randomly, this large relative difference in policy change disappears. After convergence, in states where the action gap is high (states where the ball is diagonal from the paddle) there is little policy change as expected and most of the policy change happens in states corresponding to the ball higher up where the exact actions taken matter less; see also Figure 9.

Original:  and applicability, Section 4 the ramifications.
Revised: There is some conflation from the state distribution induced by the policy; everything else being equal, policy change is higher in states that are updated more often. At the start of training and even a little while after convergence, for states where the paddle is on one of the sides (the first and last column of plots) and the ball is directly just above the paddle the relative amount of policy change is low. But well after convergence this flips and policy change is relatively high in these states. Presumably this is because early in training the agent has yet to learn that values for no-op action and the action that would move the paddle into the wall have the same effect.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: The DoubleDQN and R2D2 settings differ in the actions spaces used to act in the set of Atari games.

Original: [Yes] Section 4.
Revised: As indicated in Table 1, DoubleDQN always employs the minimal action set 3 ≤ |A| ≤ 18 (see subplot titles in Figure 15), while R2D2 always uses the full action set |A| = 18 . Adding to that, the experiments in Figure 10 also include a “redundant × 3 ” setting where the full action set is artificially replicated 3 times ( |A| = 54 ). A.3 Unlimited policy change in a two-armed bandit One minimalist setting in which it is possible to obtain large (cumulative) policy change is incremental learning of similar Q-values using small step-sizes.

Original: is specifically about the impact on the social dynamics of research.
Revised: For example, consider learning the two (tabular) Q-values of a two-armed bandit.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: Q-values are initialised near each other ( q 0 ( a 1 ) ≈ q 0 ( a 2 ) , and their true targets are also nearly identical ( q P ( a 1 )

Original: them?
Revised: ≈ q P ( a 2 ) , but far from initialisation, q 0 ( · ) ≪ q P ( · ) . With that set-up, a learning process that alternates between the actions to update can produce an arg max switch on each update, because the last-updated Q-value will always be the larger one of the [Figure] [Figure]

Original: [Yes]
Revised: [Figure]

Revised: two.

Revised: And with the appropriate setting of step-sizes and initialisation, P and thus W 1: P can be made arbitrarily large.

Original: If you are including theoretical results...
Revised: Throughout the paper we treated policy change as an unexpected phenomenon. However, some amount of policy change is inherent to all RL algorithms. Value-based methods, in particular, are based on dynamic programming, which has at its core two operations: policy evaluation and policy improvement.

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: Since by definition policy improvement involves change, it is fair to ask: how much change is in fact expected? In other words: if we could isolate all other effects, like approximation and noise, how much policy change would still remain? In Section 3.3 we already touched on this subject with the experiments on C ATCH using value iteration. In this section we revisit the question and try to provide a more definite answer to it.

Original: [N/A]
Revised: As it turns out, and perhaps not surprisingly, the answer to this question seems to be very domain dependent. The expected amount of policy change that is inherent to dynamic programming can vary significantly from one environment to the other. To illustrate this point, we now describe a simple policy evaluation setting that does not involve any approximation, incremental learning, or noise; and yet we see a large amount of policy change happening. Given the value function q π of a policy π , we compute the greedy policy π ′ with respect to q π , and monitor the changes in the greedy policy induced by the intermediate functions as we move from q π to q π ′ .

Original: (b) Did you include complete proofs of all theoretical results?
Revised: To describe our example precisely, we will need two concepts. First, we define the greedy operator g : Q → Π as g ( q ) = π such that π ( s ) = argmax a q ( s, a ) , for all s ∈ S , where q is an arbitrary function in Q and ties are broken in an arbitrary, but consistent, way. It will also be convenient to introduce the Bellman operator of a policy π as

Original: [N/A]
Revised: [Equation]

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in  the supplemental material or as a URL)?
Revised: where q ∈ Q , r ( s, a ) is the expected reward following the execution of a in s , p ( s ′ | s, a ) is the probability of transitioning to state s ′ given that action a was executed in state s , and E [ · ] is the expectation operator.

Original: [No] Our implemen-
Revised: It is well known that lim k →∞ T kπ q = q π for any q ∈ Q . Equipped with the concepts above, we can now present our example. Figure 11 shows an MDP composed of an arbitrary number of states structured as two chains. We are interested in monitoring how the policy will change in state s as we do policy evaluation.

Original: tation relies on some proprietary code.
Revised: Suppose that we start with a policy [Figure] li c [Figure]

Revised: c

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: [Figure] States with action gap States with action gap π that selects action red everywhere. Clearly, v π ( s ) = 0 .

Revised: The greedy policy π ′ =

Revised: g ( q π ) will select actions associated with nonzero rewards whenever they are available; when they are not available, we will assume that the greedy operator g will resolve the ties by always picking the green or the blue action over their red counterpart.

Original:  how they
Revised: Starting from q π , we will now monitor how much the greedy policy g ( T kπ ′ q π ) changes in s with the sequence k = 1 , 2 , ... , that is, as we move from q π to q π ′ as part of policy evaluation. For ease of exposition, we will use π k ≡ g ( T kπ ′ q π ) to refer to the greedy policies along the way.

Revised: Clearly, in the first step, when we change from π to π 1 = g ( T 1 π ′ q π ) , the policy changes in s from red to green .

Revised: Now, in the second step, an easy calculation shows that the policy changes again, now from green to blue .

Revised: If we keep doing this exercise, a simple pattern emerges: policies π k whose index k is odd will pick action green in s , while their counterparts with an even index will instead select blue on that state.

Revised: This means that W ( π k , π k +1 | s )

Revised: = 1 along the sequence of greedy policies

Revised: π 1 , π 2 , ... .

Revised: This deliberately simple example illustrates that the maximum possible amount of policy change can happen on a given state simply as an effect of policy evaluation.

Revised: It is not difficult to construct examples in which a similar effect is observed throughout state space.

Revised: In Section 5 we discussed how the well-known policy oscillation effect may be responsible for part of the policy change when function approximation is used.

Revised: The “dynamic-programming effect” discussed in this section happens in addition to that, regardless of function approximation.

Revised: In general, we expect that policy change could be a result of both effects, plus other causes like the ones discussed in Section 3.4 and Appendix A.3.

Revised: Given all the empirical evidence we have collected, we are reasonably confident that the causes discussed in Section 3.4—namely, global function approximation and noise—play a much more important role than the policy oscillation and dynamic programming effects in the setup studied.

Revised: A.5 Churn-aware off-policy correction

Revised: Following up on Section 4.3, this section spells out some concrete possibilities for forms of off-policy correction that take the churn phenomenon into account.

Revised: In a low-latency setting for example, it may be worth truncating traces when the noise of ϵ -greedy leads to a low-advantage action getting executed, but not when the action discrepancy is purely due to churn ( µ was acting greedily).

Original: were chosen)?
Revised: Alternatively, [Figure] n [Figure]

Original: [Yes] See Appendix B.
Revised: n - r [Figure] n t [Figure]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: c t i [Figure] we think it is plausible to make truncation decisions based on (relative) advantage gaps, effectively ignoring arg max switches between actions of similar value. A.6 Relating churn to other game-specific properties

Revised: Overall, we have not identified game-specific properties that are clearly predictive of the magnitude of policy change.

Original: ments multiple times)?
Revised: Figure 12 provides a number of scatter plots for game-specific properties that we had considered as possibly having an influence. B Experimental Details B.1 DQN experiments We chose to use double Q-learning with DQN (DoubleDQN, [47]) instead of vanilla DQN [27, 28] for all of our experiments, as it is generally the more robust and better tuned of the two algorithms. Apart

Original: [Yes] Many of our experiments have 3 seeds across 15 games{+Cumulative policy change W 0: T 0. A v e r a g e v a l u e value iteration policy iteration oracle (1 step) Cumulative policy change W 0: T 
 E rr+}
Revised: {+Cumulative policy change W 0: T 0. A v e r a g e v a l u e value iteration policy iteration oracle (1 step) Cumulative policy change W 0: T 
 E rr+}

Original: which is enough to draw meaningful conclusions.
Revised: o r r a t e 10000, 1e1000, 1e1000, 1e1000, 1e100, 1e- from overall improved performance, for the purposes of this investigation there is little difference between the two, notably in terms of policy change, see Figure 23. We use an identical setting as the original DoubleDQN paper, including all hyper-parameters (which differ slightly from those in vanilla DQN). The main ones are listed in Table 1, the remaining ones in Table 2. Our implementation is based on a slightly modified variant of the open-source DoubleDQN implementation in DQN

Original: We report either error bars, show
Revised: Zoo [36]. Our Atari investigations did not involve any hyper-parameter tuning. The modifications we did to existing settings for the exploration experiments (Section 2) are binary ablations: • Reducing ϵ = 0 .

Revised: 01 to ϵ = 0 in the ϵ -greedy behaviour policies • Using the target network instead of the online network

Original: box-plots, or show all  seeds individually.
Revised: for acting. The “forked tandem” setup used in several ablations in Section 3.2 follows [34] and is based on their accompanying open-source implementation. Our Atari experiments are run with the same ALE variant of the Atari 2600 benchmark [2] as in the original DQN and DoubleDQN works, using an action repeat of 4 , a zero discount on transitions involving a life loss, and the only source of stochasticity being a random number (uniformly between 0 and 30 ) of no-op actions applied at the beginning of each episode. Unless stated otherwise, all these experiments are run with 3 seeds for each configuration.

Revised: A lot of preliminary investigations used a small subset of Atari games (B REAKOUT , P ONG , M S .

Revised: P AC -M

Original: (d) Did you include the total amount ofcompute and the type of resources used (e.g., type
Revised: AN and S PACE I NVADERS ). For the final runs on 15 games, we picked a representative subset of the 57 Atari games, with a preference for games on which DoubleDQN can achieve a decent performance level. B.2 R2D2 experiments The agent denoted as “R2D2” throughout the paper is a variant of the Recurrent Replay Distributed

Original: of GPUs, internal cluster, or cloud provider)?
Revised: DQN architecture [21]. It comprises 192 CPU-based actors concurrently generating experience [Table] DoubleDQN

Original: [Yes] See Appendix B.
Revised: R2D [Table] [Table] [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: [Table] [Table] [Table] [Table]

Original: [N/A]
Revised: N/A [Table] N

Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: [Table] N [Table] [Table]

Revised: Dueling

Revised: [Table]

Revised: [Table]

Revised: ϵ for ϵ -greedy policy annealed from 1 to 0 .fixed

Revised: [Table]

Revised: [Table]

Revised: observations

Original: [N/A]
Revised: [Table] N

Revised: [Table]

Revised: sampling

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Table] N/A [Table] 400 updates

Original: [N/A]
Revised: [Table]

Revised: clipping

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: N/A Normalisation N/A Return-based [40]

Original: using/curating?
Revised: Optimiser [Table] RMSProp [Table]

Revised: Adam

Revised: [Table]

Original: [N/A]
Revised: [Table]

Revised: and feeding it to a distributed experience replay buffer, and a single GPU-based learner randomly sampling mini-batches of experience sequences from replay and performing updates of the recurrent value function by gradient descent.

Revised: The value function is represented by a convolutional torso feeding into a linear layer, followed by a recurrent LSTM core, whose output is processed by a further linear layer before finally being output via a Dueling value head [49].

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: The exact parameterization follows the slightly modified R2D2 presented in [11, 40], see Table 2 for a full list of hyper-parameters. It is trained using the Adam optimiser [23] on a 5 -step Q-learning loss, using a periodically updated target network for bootstrap target computation. Replay sampling is performed using prioritized experience replay [41] with priorities computed from sequences’ TD errors following the scheme introduced in [21]. The agent uses a fixed replay ratio of 1 , i.e. the learner or actors are throttled dynamically if the average number of times a sample gets replayed exceeds or falls below this value.

Revised: It also uses unclipped rewards and unclipped gradients, and an accompanying return-based normalisation, as in [40].

Revised: Differently from those Atari RL agents following DQN [28], our agent uses the raw 210 × 160 RGB frames as input to its value function (one at a time, without frame stacking), though it still applies a max-pool operation over the most recent 2 frames to mitigate flickering inherent to the Atari simulator.

Revised: As in most past work, an action-repeat of 4 is applied, episodes begin with a random number of no-op actions (up to 30 ) being applied, and time-out after 108 000 frames (i.e.

Revised: 30 minutes of real-time game play).

Original: information or offensive content?
Revised: The agent is implemented with JAX [5], uses the Haiku [17], Optax [7], Chex [6], and RLax [18] libraries for neural networks, optimisation, testing, and RL losses, respectively, and Reverb [8] for distributed experience replay. All our experiments ran for 40 000 learner updates. With a replay ratio of 1 , sequence length of(adjacent sequences overlapping by 40 observations), a batch size of 32 , and an action-repeat ofthis corresponds to a training budget of ≈ 200 M environment frames ( ≈ 100 times fewer than the original R2D2).

Revised: In wall-clock-time, one such experiment takes about 2 hours.

Revised: All experiments are conducted across 15 games, using 3 seeds per game, unless stated otherwise.

Revised: B. C ATCH experiments

Revised: For C ATCH [33] experiments, Table 3 lists the hyper-parameters for each of the variants specified in Figure 4.

Revised: For each variant, seeds that did not converge after 5 000 episodes of training were filtered out.

Revised: In practice, all seeds for all variants in the table converged.

Revised: For all C ATCH experiments

Revised: [Table]

Revised: [Table]

Revised: MLP

Revised: [Table]

Revised: Regression on q

Revised: 1 layer MLP

Revised: [Table]

Revised: Q-learning with 3 layer MLP

Revised: [Table]

Revised: DQN-like with RMSProp

Revised: [Table]

Revised: DQN-like with SGD

Revised: [Table]

Revised: DQN-like with Adam

Revised: [Table]

Revised: parameters

Revised: [Table]

Revised: convergence is defined as when the greedy policy achieves the maximum score for 100 evaluation episodes.

Revised: Convergence is periodically tested every 100 training episodes.

Revised: B.4 Dynamic programming

Revised: To measure policy change of dynamic programming in a tabular MDP, we exploit the knowledge of the exact transition dynamics, encoded via a matrix T to compute value or policy iteration updates that do not involve sampling or interactions.

Revised: Values are initialised at 0 , and for the purposes of measuring policy change, all arg max actions whose Q-values are exactly tied also share equal probability mass.

Revised: As example domain we use a 16 × 16 Gridworld with 4 -room structure, initial state in one corner, goal state in opposing corner and γ = 0 . 97 .

Revised: Figure 13 (left) shows the amounts of policy change accumulated in such a process.

Revised: B.5 MNIST experiments

Revised: For a simple initial supervised learning experiment, we used an off-the-shelf neural network training setup on MNIST.

Revised: Thus we used a 3 -layer MLP with 300 and 100 hidden units, ReLU non-linearities, a softmax output, cross-entropy loss and the Adam optimiser [23].

Revised: Policy change is measured on the softmax probability outputs of the classification network, with equal weight on all samples of the test

Revised: [Figure]

Revised: set.

Revised: It is accumulated across all P gradient updates.

Revised: Our experiments are stopped when reaching 2% training error, which happens after P ≈ 1 000 − 10 000 updates.

Revised: Figure 13 (right) shows the results.

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: k

Original: [N/A]
Revised: [Figure] [Figure] [Figure] [Figure]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: | | [Figure] [Figure] | |=

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: [Figure] | | [Figure] In contrast to Figure 15, these results are from the R2D2 agent (always using the full action set of |A| = 18 ).

Revised: Thick green lines show the average across training, while thin lines show snapshots from different points in training, with cooler and warmer colors denoting early and late stages of training respectively.

Original: applicable?
Revised: We can see how policy change quickly rises and then saturates, generally between 20% and 60% . This means that, compared to π t , the policy π t +100 generally does not differ much more than π t +20 . This is consistent with the hypothesis that policy churn only affects a subset of states. An outlier here are the S KIING results, where the observed fraction of arg max switches (in a minibatch of 32 × 80 states) is always either 0 or 1 : this seems to indicate that the Q-values have essentially no state-dependence (note that performance also does not take off in this game, see Figure 24)

Original: [N/A]
Revised: [Figure]

Revised: [Figure]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: [Figure] [Figure] [Figure] this network is a periodic copy of the online (learning) network, just like the target network, but updated at a different frequency.

Original: Board (IRB) approvals, if applicable?
Revised: “Interval = 16 ” corresponds to the DoubleDQN baseline, while “Interval = 120 000 ” corresponds to the “act with target network” of Section 2 and Figure 2 (denoted “no churn” there). We find again (cf. Figure 18) that the full empirical magnitude of policy change in DoubleDQN is not needed for exploration: reducing the number of different greedy policies used for acting by a factor 100 − 1 000 still retains a very similar exploration effect.

Original: [N/A]
Revised: [Figure]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: [Figure] [Figure] p i s o d e R e t u r n

Original: spent on participant compensation?
Revised: [Figure] [Figure] due to two aspects: first, DoubleDQN has a high amount of random exploration in early learning (it takes 4 M frames until ϵ has decayed to 0 ). Second, DoubleDQN traverses many more distinct policy networks over the course of its lifetime ( ≈ 10 7 ), compared to R2D2 ( ≈ 10 4 ), due to the latter’s much larger batch size, greater parallelism, and smaller replay ratio.

Revised: Note also that the maximal “policy age” (in gradient updates) and as a consequence policy diversity representend in the replay buffer data is very different in R2D2 and

Revised: DQN.

Revised: Because of the data generation parallelism (and the near-deterministic dynamics of the Atari environment), diversity of replay data in R2D2 may be driven more by ϵ -exploration than in DQN.

Revised: The case ϵ = 0 may therefore result in a very narrow data distribution and potentially collapse of the neural network representation in R2D2.

Original: [N/A]
Revised: [Figure]

Revised: [Figure]