Original: T RANSIENT N ON - STATIONARITY AND G ENERALISATION IN D EEP R EINFORCEMENT L EARNING
Revised: T RANSIENT N ON - STATIONARITY AND G ENERALISATION IN D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Non-stationarity can arise in Reinforcement Learning ( RL ) even in stationary environments.
Revised: Non-stationarity can arise in Reinforcement Learning ( RL ) even in stationary environments.

Original: For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy.
Revised: For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy.

Original: Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated.
Revised: Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated.

Original: However, we ﬁnd evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance.
Revised: However, we ﬁnd evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance.

Original: Consequently, to improve generalisation of deep RL agents, we propose Iterated
Revised: Consequently, to improve generalisation of deep RL agents, we propose Iterated

Original: Relearning ( ITER ).
Revised: Relearning ( ITER ).

Original: ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training.
Revised: ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training.

Original: Experimentally, we show that
Revised: Experimentally, we show that

Original: ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom .
Revised: ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In RL , as an agent explores more of its environment and updates its policy and value function, the data distribution it uses for training changes.
Revised: In RL , as an agent explores more of its environment and updates its policy and value function, the data distribution it uses for training changes.

Original: In deep RL , this non-stationarity is often not addressed explicitly.
Revised: In deep RL , this non-stationarity is often not addressed explicitly.

Original: Typically, a single neural network model is initialised and continually updated during training.
Revised: Typically, a single neural network model is initialised and continually updated during training.

Original: Conventional wisdom about catastrophic forgetting (Kemker et al., 2018) implies that old updates from a different data-distribution will simply be forgotten.
Revised: Conventional wisdom about catastrophic forgetting (Kemker et al., 2018) implies that old updates from a different data-distribution will simply be forgotten.

Original: However, we provide evidence for an alternative hypothesis: networks exhibit a memory effect in their learned representations which can harm generalisation permanently if the data-distribution changed over the course of training.
Revised: However, we provide evidence for an alternative hypothesis: networks exhibit a memory effect in their learned representations which can harm generalisation permanently if the data-distribution changed over the course of training.

Original: To build intuition, we ﬁrst study this phenomenon in a supervised setting on the CIFAR-10 dataset.
Revised: To build intuition we ﬁrst study this phenomenon in a supervised setting on the CIFAR-10 dataset.

Original: We artiﬁcially introduce transient non-stationarity into the training data and investigate how this affects the asymptotic performance under the ﬁnal, stationary data in the later epochs of training.
Revised: We artiﬁcially introduce transient non-stationarity into the training data and investigate how this affects the asymptotic performance under the ﬁnal, stationary data in the later epochs of training.

Original: Interestingly, we ﬁnd that while asymptotic training performance is nearly unaffected, test performance degrades considerably, even after the data-distribution has converged.
Revised: Interestingly, we ﬁnd that while asymptotic training performance is nearly unaffected, test performance degrades considerably, even after the data-distribution has converged.

Original: In other words, we ﬁnd that latent representations in deep networks learned under certain types of non-stationary data can be inadequate for good generalisation and might not be improved by later training on stationary data.
Revised: In other words, we ﬁnd that latent representations in deep networks learned under certain types of non-stationary data can be inadequate for good generalisation and might not be improved by later training on stationary data.

Original: Such transient non-stationarity is typical in RL .
Revised: Such transient non-stationarity is typical in RL .

Original: Consequently, we argue that this observed degradation of generalisation might contribute to the inferior generalisation properties recently attributed to many RL agents evaluated on held out test environments (Zhang et al., 2018a;b; Zhao et al., 2019).
Revised: Consequently, we argue that this observed degradation of generalisation might contribute to the inferior generalisation properties recently attributed to many RL agents evaluated on held out test environments (Zhang et al., 2018a;b; Zhao et al., 2019).

Original: Furthermore, in contrast to Supervised Learning ( SL ), simply re-training the agent from scratch once the data-distribution has changed is infeasible in RL as current state of the art algorithms require data close to the on-policy distribution, even for off-policy algorithms like Q-learning (Fedus et al., 2020).
Revised: Furthermore, in contrast to Supervised Learning ( SL ), simply re-training the agent from scratch once the data-distribution has changed is infeasible in RL as current state of the art algorithms require data close to the on-policy distribution, even for off-policy algorithms like Q-learning (Fedus et al., 2020).

Original: To improve generalisation of RL agents despite this restriction, we propose Iterated Relearning ( ITER ).
Revised: To improve generalisation of RL agents despite this restriction, we propose Iterated Relearning ( ITER ).

Original: In this paradigm for deep RL training, the agent’s policy and value are periodically distilled into a freshly initialised student, which subsequently replaces the teacher for further optimisation.
Revised: In this paradigm for deep RL training, the agent’s policy and value are periodically distilled into a freshly initialised student, which subsequently replaces the teacher for further optimisation.

Original: While this occasional distillation step simply aims to re-learn and replace the current policy and value outputs for the training data, it allows the student to learn a better latent representation with improved performance for unseen inputs because it eliminates non-stationarity during distillation.
Revised: While this occasional distillation step simply aims to re-learn and replace the current policy and value outputs for the training data, it allows the student to learn a better latent representation with improved performance for unseen inputs because it eliminates non-stationarity during distillation.

Original: We propose a practical implementation of ITER which performs the distillation in parallel to the training process without requiring additional training data.
Revised: We propose a practical implementation of ITER which performs the distillation in parallel to the training process without requiring additional training data.

Original: While this introduces a small amount of non-stationarity into the distillation step, it greatly improves sample efﬁciency without noticeably impacting performance.
Revised: While this introduces a small amount of non-stationarity into the distillation step, it greatly improves sample efﬁciency without noticeably impacting performance.

Original: Experimentally, we evaluate ITER on the Multiroom environment, as well as several environments from the recently proposed ProcGen benchmark and ﬁnd that it improves generalisation.
Revised: Experimentally, we evaluate ITER on the Multiroom environment, as well as several environments from the recently proposed ProcGen benchmark and ﬁnd that it improves generalisation.

Original: This provides further support to our hypothesis and indicates that the non-stationarity inherent to many
Revised: This provides further support to our hypothesis and indicates that the non-stationarity inherent to many

Original: RL algorithms, even when training on stationary environments, should not be ignored when aiming to learn robust agents.
Revised: RL algorithms, even when training on stationary environments, should not be ignored when aiming to learn robust agents.

Original: Lastly, to further support this claim and provide more insight into possible causes of the discovered effect, we perform additional ablation studies on the CIFAR-10 dataset.
Revised: Lastly, to further support this claim and provide more insight into possible causes of the discovered effect, we perform additional ablation studies on the CIFAR-10 dataset.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We describe an RL problem as a Markov decision process ( MDP ) ( S , A , T, r, p 0 , γ ) (Puterman, 2014) with actions a ∈A , states s ∈S , initial state s 0 ∼ p 0 , transition dynamics s (cid:48) ∼ T ( s, a ) , reward function r ( s, a ) ∈ R and discount factor γ .
Revised: We describe an RL problem as a Markov decision process ( MDP ) ( S , A , T, r, p 0 , γ ) (Puterman, 2014) with actions a ∈A , states s ∈S , initial state s 0 ∼ p 0 , transition dynamics s (cid:48) ∼ T ( s, a ) , reward function r ( s, a ) ∈ R and discount factor γ .

Original: The unnormalised discounted state distribution induced by a policy π is deﬁned as d π ( s ) =
Revised: The unnormalised discounted state distribution induced by a policy π is deﬁned as d π ( s ) =

Original: (cid:80) ∞ t =0 γ t Pr (cid:0) S t = s | S 0 ∼ p 0 , A t ∼ π ( ·| S t ) , S t +1 ∼ T ( S t , A t ) (cid:1) .
Revised: (cid:80) ∞ t =0 γ t Pr (cid:0) S t = s | S 0 ∼ p 0 , A t ∼ π ( ·| S t ) , S t +1 ∼ T ( S t , A t ) (cid:1) .

Original: In ITER , we learn a sequence of policies and value functions, which we denote with π ( k ) ( a | s ) and V ( k ) ( s ) at the k th iteration ( k ∈ { 0 , 1 , 2 , . . . } ), parameterized by θ k .
Revised: In ITER , we learn a sequence of policies and value functions, which we denote with π ( k ) ( a | s ) and V ( k ) ( s ) at the k th iteration ( k ∈ { 0 , 1 , 2 , . . . } ), parameterized by θ k .

Original: We brieﬂy discuss some forms of non-stationarity which can arise in RL , even when the environment is stationary.
Revised: We brieﬂy discuss some forms of non-stationarity which can arise in RL , even when the environment is stationary.

Original: For simplicity, we focus the exposition on actor-critic methods which use samples from interaction with the environment to estimate the policy gradient given by [Equation]
Revised: For simplicity, we focus the exposition on actor-critic methods which use samples from interaction with the environment to estimate the policy gradient given by g = E [ ∇ θ log π θ ( a | s )

Original:  | s ) T ( s (cid:48) | s, a )] .
Revised: A π ( s, a, s (cid:48) ) | s, a, s (cid:48) ∼ d π ( s ) π ( a | s ) T ( s (cid:48) | s, a )] .

Original: The advantage is often estimated as A π ( s, a, s (cid:48) )
Revised: The advantage is often estimated as A π ( s, a, s (cid:48) )

Original: =
Revised: =

Original: r ( s, a ) + γV π
Revised: r ( s, a ) + γV π

Original: ( s (cid:48) )
Revised: ( s (cid:48) )

Original: − V π ( s ) .
Revised: − V π ( s ) .

Original: Typically, we also use neural networks to approximate the baseline V πφ ( s ) and for bootstrapping from the future value V πφ ( s (cid:48) ) .
Revised: Typically, we also use neural networks to approximate the baseline V πφ ( s ) and for bootstrapping from the future value V πφ ( s (cid:48) ) .

Original: φ can be learned by minimising E
Revised: φ can be learned by minimising E

Original: [ A π ( s, a, s (cid:48) ) 2 ] by stochastic semi-gradient descent, treating V πφ ( s (cid:48) ) as a constant.
Revised: [ A π ( s, a, s (cid:48) ) 2 ] by stochastic semi-gradient descent, treating V πφ ( s (cid:48) ) as a constant.

Original: There are at least three main types of non-stationarity in deep RL .
Revised: There are at least three main types of non-stationarity in deep RL .

Original: First, we update the policy π θ , which leads to changes in the state distribution d π θ ( s ) .
Revised: First, we update the policy π θ , which leads to changes in the state distribution d π θ ( s ) .

Original: Early on in training, a random policy π θ only explores states close to initial states s 0 .
Revised: Early on in training, a random policy π θ only explores states close to initial states s 0 .

Original: As π θ improves, new states further from s 0 are encountered.
Revised: As π θ improves, new states further from s 0 are encountered.

Original: Second, changes to the policy also change the true value function V π ( s ) which V πφ ( s ) is approximating.
Revised: Second, changes to the policy also change the true value function V π ( s ) which V πφ ( s ) is approximating.

Original: Lastly, due to the use of bootstrap targets in temporal difference learning, the learned value V πφ ( s ) is not regressed directly towards V π ( s ) .
Revised: Lastly, due to the use of bootstrap targets in temporal difference learning, the learned value V πφ ( s ) is not regressed directly towards V π ( s ) .

Original: Instead V πφ ﬁts a gradually evolving target sequence even under a ﬁxed policy π , thereby also changing the policy gradient estimator g .
Revised: Instead V πφ ﬁts a gradually evolving target sequence even under a ﬁxed policy π , thereby also changing the policy gradient estimator g .

Original:  T HE I MPACT OF N ON - STATIONARITY ON G ENERALISATION
Revised:  T HE I MPACT OF N ON - STATIONARITY ON G ENERALISATION

Original: In this section we investigate how asymptotic performance is affected by changes to the datadistribution during training.
Revised: In this section we investigate how asymptotic performance is affected by changes to the datadistribution during training.

Original: In particular, we assume an initial, transient phase of non-stationarity, followed by an extended phase of training on a stationary data-distribution.
Revised: In particular, we assume an initial, transient phase of non-stationarity, followed by an extended phase of training on a stationary data-distribution.

Original: This is similar to the situation in RL where the data-distribution is affected by a policy which converges over time.
Revised: This is similar to the situation in RL where the data-distribution is affected by a policy which converges over time.

Original: We show that this transient non-stationarity has a permanent effect on the learned representation and negatively impacts generalisation.
Revised: We show that this transient non-stationarity has a permanent effect on the learned representation and negatively impacts generalisation.

Original: As interventions in RL training can lead to confounding factors due to off-policy data or changed exploration behaviour, we utilise Supervised Learning ( SL ) here to provide initial evidence in a more controlled setup.
Revised: As interventions in RL training can lead to confounding factors due to off-policy data or changed exploration behaviour, we utilise Supervised Learning ( SL ) here to provide initial evidence in a more controlled setup.

Original: We use the CIFAR-10 dataset for image classiﬁcation (Krizhevsky et al., 2009) and artiﬁcially inject non-stationarity.
Revised: We use the CIFAR-10 dataset for image classiﬁcation (Krizhevsky et al., 2009) and artiﬁcially inject non-stationarity.

Original: Our goal is to provide qualitative results on the impact of non-stationarity, not to obtain optimal performance.
Revised: Our goal is to provide qualitative results on the impact of non-stationarity, not to obtain optimal performance.

Original: We use a ResNet18 (He et al., 2016) architecture, similar to those used by Espeholt et al.
Revised: We use a ResNet18 (He et al., 2016) architecture, similar to those used by Espeholt et al.

Original: (2018) and Cobbe et al. (2019a).
Revised: (2018) and Cobbe et al. (2019a).

Original: Parameters are updated using Stochastic Gradient Descent ( SGD ) with momentum and, following standard practice in RL , we use a constant learning rate and do not use batch normalisation.
Revised: Parameters are updated using Stochastic Gradient Descent ( SGD ) with momentum and, following standard practice in RL , we use a constant learning rate and do not use batch normalisation.

Original: Weight decay is used for regularisation.
Revised: Weight decay is used for regularisation.

Original: Hyper-parameters and more details can be found in appendix B.
Revised: Hyper-parameters and more details can be found in appendix B.

Original: We train for a total of 2500 epochs.
Revised: We train for a total of 2500 epochs.

Original: While the last 1500 epochs are trained on the full, unaltered dataset, we modify the training data in three different ways during the ﬁrst 1000 epochs.
Revised: While the last 1500 epochs are trained on the full, unaltered dataset, we modify the training data in three different ways during the ﬁrst 1000 epochs.

Original: Test data is left unmodiﬁed throughout training.
Revised: Test data is left unmodiﬁed throughout training.

Original: While each modiﬁcation is loosely motivated by the RL setting, our goal is not to mimic it exactly (which would be infeasible), nor to disentangle the contributions of different types of non-stationarity.
Revised: While each modiﬁcation is loosely motivated by the RL setting, our goal is not to mimic it exactly (which would be infeasible), nor to disentangle the contributions of different types of non-stationarity.

Original: Instead, we aim to show that these effects reliably occur in the presence to various types of non-stationarity, and provide intuitions that can be brought into the RL setting in Section 4.
Revised: Instead, we aim to show that these effects reliably occur in the presence to various types of non-stationarity, and provide intuitions that can be brought into the RL setting in Section 4.

Original: For the ﬁrst modiﬁcation, called Dataset Size , we initially train only on a small fraction of the full dataset and gradually add more data points after each epoch, at a constant rate, until the full dataset is available after epoch 1000.
Revised: For the ﬁrst modiﬁcation, called Dataset Size , we initially train only on a small fraction of the full dataset and gradually add more data points after each epoch, at a constant rate, until the full dataset is available after epoch 1000.

Original: During the non-stationary phase, data points are reused multiple times to ensure the same number of network updates are made in each epoch.
Revised: During the non-stationary phase, data points are reused multiple times to ensure the same number of network updates are made in each epoch.

Original: For the modiﬁcation
Revised: For the modiﬁcation

Original: Wrong Labels we replace all training labels by randomly drawn incorrect ones.
Revised: Wrong Labels we replace all training labels by randomly drawn incorrect ones.

Original: After each epoch, a constant number of these labels are replaced by their correct values.
Revised: After each epoch, a constant number of these labels are replaced by their correct values.

Original: Lastly, Noisy Labels is similar to Wrong Labels , but the incorrect labels are sampled uniformly at the start of each epoch.
Revised: Lastly, Noisy Labels is similar to Wrong Labels , but the incorrect labels are sampled uniformly at the start of each epoch.

Original: For both, all training labels are correct after epoch 1000.
Revised: For both, all training labels are correct after epoch 1000.

Original: While Dataset Size is inspired by the changing state distribution seen by an evolving policy, Wrong Labels and Noisy Labels are motivated by the consistent bias or ﬂuctuating errors a learned critic can introduce in the policy gradient estimate.
Revised: While Dataset Size is inspired by the changing state distribution seen by an evolving policy, Wrong Labels and Noisy Labels are motivated by the consistent bias or ﬂuctuating errors a learned critic can introduce in the policy gradient estimate.

Original: [Figure]
Revised: [Figure]

Revised: reduced by initial, transient non-stationarity.

Original:  results are shown in ﬁg
Revised: The results are shown in ﬁg. 1.

Original: While the ﬁnal training accuracy (left) is almost unaffected (see table 1 in the appendix for exact results), all three nonstationary modiﬁcations significantly reduce the test accuracy (right).
Revised: While the ﬁnal training accuracy (left) is almost unaffected (see table 1 in the appendix for exact results), all three nonstationary modiﬁcations significantly reduce the test accuracy (right).

Original: The plateau in accuracy shows that this effect persists even after the models are further trained using the full dataset with correct labels: non-stationarity early in training has a permanent effect on the learned representations and quality of generalisation.
Revised: The plateau in accuracy shows that this effect persists even after the models are further trained using the full dataset with correct labels: non-stationarity early in training has a permanent effect on the learned representations and quality of generalisation.

Original: These results indicate that the non-stationarity introduced by the gradual convergence of the policy in RL might similarly deteriorate the generalisation of the agent.
Revised: These results indicate that the non-stationarity introduced by the gradual convergence of the policy in RL might similarly deteriorate the generalisation of the agent.

Original: To overcome this, we propose ITER in the next section.
Revised: To overcome this, we propose ITER in the next section.

Original: The key insight enabling ITER is that the observed negative effect is restricted to the test data, whereas the predictions on the training data are unaffected and of high quality.
Revised: The key insight enabling ITER is that the observed negative effect is restricted to the test data, whereas the predictions on the training data are unaffected and of high quality.

Original:  ITER
Revised:  ITER

Original: In section 3, we have seen evidence that the non-stationarity which is present in many deep RL algorithms might lead to impaired generalisation on held-out test environments.
Revised: In section 3, we have seen evidence that the non-stationarity which is present in many deep RL algorithms might lead to impaired generalisation on held-out test environments.

Original: To mitigate this and improve generalisation to previously unseen states, we propose Iterated Relearning ( ITER ): instead of updating a single agent model throughout the entire training process, ITER learns a sequence of models, each of which is exposed to less non-stationarity during its training.
Revised: To mitigate this and improve generalisation to previously unseen states, we propose Iterated Relearning ( ITER ): instead of updating a single agent model throughout the entire training process, ITER learns a sequence of models, each of which is exposed to less non-stationarity during its training.

Original: As we will show in section 5, this improves generalisation.
Revised: As we will show in section 5, this improves generalisation.

Original: ITER can be applied on top of a wide range of base RL training methods.
Revised: ITER can be applied on top of a wide range of base RL training methods.

Original: For simplicity, we focus in the following exposition on actor-critic methods and use Proximal Policy Optimization ( PPO ) (Schulman et al., 2017) for the experimental evaluation.
Revised: For simplicity, we focus in the following exposition on actor-critic methods and use Proximal Policy Optimization ( PPO ) (Schulman et al., 2017) for the experimental evaluation.

Original: The underlying insight behind ITER is that at any point during RL training the latent representation of our current agent network might be signiﬁcantly damaged by non-stationarity, but its outputs on the training data are comparatively unaffected (see ﬁg. 1).
Revised: The underlying insight behind ITER is that at any point during RL training the latent representation of our current agent network might be signiﬁcantly damaged by non-stationarity, but its outputs on the training data are comparatively unaffected (see ﬁg. 1).

Original: Consequently, ITER aims to periodically replace the current agent network, the ‘teacher’, by a ‘student’ network which was freshly initialised and trained to mimic the teacher on the current training data.
Revised: Consequently, ITER aims to periodically replace the current agent network, the ‘teacher’, by a ‘student’ network which was freshly initialised and trained to mimic the teacher on the current training data.

Original: Because this re-learning and replacement step can be performed on stationary data, it allows us to re-learn a policy that matches performance on the training data but generalises better to novel test environments.
Revised: Because this re-learning and replacement step can be performed on stationary data, it allows us to re-learn a policy that matches performance on the training data but generalises better to novel test environments.

Original: ITER begins with an initial policy π (0) and value function V (0) and then repeats the following steps, starting with iteration k = 0 .
Revised: ITER begins with an initial policy π (0) and value function V (0) and then repeats the following steps, starting with iteration k = 0 .

Original: Use the base RL algorithm to train π ( k ) and V ( k ) .
Revised: Use the base RL algorithm to train π ( k ) and V ( k ) .

Original: Initialise new student networks for π ( k +1) and V ( k +1) .
Revised: Initialise new student networks for π ( k +1) and V ( k +1) .

Original: We refer to the current policy π ( k ) and value function V ( k ) as the teacher .
Revised: We refer to the current policy π ( k ) and value function V ( k ) as the teacher .

Original: Distill the teacher into the student.
Revised: Distill the teacher into the student.

Original: This phase is discussed in more detail in section 4.1.
Revised: This phase is discussed in more detail in section 4.1.

Original: The student replaces the teacher: π ( k ) and V ( k ) can be discarded.
Revised: The student replaces the teacher: π ( k ) and V ( k ) can be discarded.

Original: Increment k and return to step 1.
Revised: Increment k and return to step 1.

Original: Repeat as many times as needed.
Revised: Repeat as many times as needed.

Original: This results in alternating RL training with distillation into a freshly initialised student.
Revised: This results in alternating RL training with distillation into a freshly initialised student.

Original: The RL training phases continue to introduce non-stationarity until the models converge, so we want to iterate the process, repeating steps 1-4.
Revised: The RL training phases continue to introduce non-stationarity until the models converge, so we want to iterate the process, repeating steps 1-4.

Original: How often we do so depends on the environment and can be chosen as a hyper-parameter.
Revised: How often we do so depends on the environment and can be chosen as a hyper-parameter.

Original: In practise we found the results to be quite robust to this choice and recommend, as general rule, to iterate as often as possible within the limits outlined in section 4.2.
Revised: In practise we found the results to be quite robust to this choice and recommend, as general rule, to iterate as often as possible within the limits outlined in section 4.2.

Original: There, we introduce a practical implementation of ITER which re-uses data between steps 1 and 3 in order to not require additional samples from the environment.
Revised: There, we introduce a practical implementation of ITER which re-uses data between steps 1 and 3 in order to not require additional samples from the environment.

Original: 4. D ISTILLATION L OSS
Revised: 4. D ISTILLATION L OSS

Original: Our goal during the distillation phase (step 3) is to learn a new student policy π ( k +1) and value function V ( k +1) that imitate the current teacher ( π ( k ) , V ( k ) ) .
Revised: Our goal during the distillation phase (step 3) is to learn a new student policy π ( k +1) and value function V ( k +1) that imitate the current teacher ( π ( k ) , V ( k ) ) .

Original: If the student and teacher share the same network architecture, the student could of course imitate the teacher exactly by copying its parameters.
Revised: If the student and teacher share the same network architecture, the student could of course imitate the teacher exactly by copying its parameters.

Original: However, since the teacher was trained under non-stationarity, its generalisation performance is likely degraded (see section 3).
Revised: However, since the teacher was trained under non-stationarity, its generalisation performance is likely degraded (see section 3).

Original: Consequently, we want to instead train a freshly initialised student network to mimic the teacher’s outputs for the available data, but learn a better internal representation by training on a stationary data distribution collected by the teacher π ( k ) , i.e., s, a ∼ d π ( k ) ( s ) π ( k ) ( a | s ) .
Revised: Consequently, we want to instead train a freshly initialised student network to mimic the teacher’s outputs for the available data, but learn a better internal representation by training on a stationary data distribution collected by the teacher π ( k ) , i.e., s, a ∼ d π ( k ) ( s ) π ( k ) ( a | s ) .

Original: The student, parameterised by θ k +1 , is trained using a linear combination of four loss terms
Revised: The student, parameterised by θ k +1 , is trained using a linear combination of four loss terms:

Original: [Equation]
Revised: [Equation]

Original: where λ TD is a ﬁxed hyper-parameter and we linearly anneal α π and α V from some ﬁxed initial value to zero over the course of each distillation phase
Revised: where λ TD is a ﬁxed hyper-parameter and we linearly anneal α π and α V from some ﬁxed initial value to zero over the course of each distillation phase

Original: [Equation]
Revised: [Equation]

Original: L π and L V are supervised losses minimising the disagreement between outputs of the student and the teacher:
Revised: L π and L V are supervised losses minimising the disagreement between outputs of the student and the teacher:

Original: [Equation]
Revised: [Equation]

Original: The additional terms L PG and L TD are off-policy RL objectives for updating the actor and critic:
Revised: The additional terms L PG and L TD are off-policy RL objectives for updating the actor and critic:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where A ( k +1) ( s, a, s (cid:48) 
Revised: where A ( k +1) ( s, a, s (cid:48) )

Original: [Equation]

Original: =
Revised: =

Original: [Equation] s, 
Revised: r ( s, a ) + γ ⊥ V ( k +1)

Original: [Equation]
Revised: ( s (cid:48) )

Original: ) − V ( k +1) ( s ) denotes the estimated advantage of choosing action a and ⊥ is a stop-gradient operator, its operand is treated as a constant when taking derivatives of the objective.
Revised: − V ( k +1) ( s ) denotes the estimated advantage of choosing action a and ⊥ is a stop-gradient operator, its operand is treated as a constant when taking derivatives of the objective.

Original: In practice, the losses in eq.
Revised: In practice, the losses in eq.

Original: (2) remain nonzero during distillation, potentially causing a drop in performance once the student replaces the teacher.
Revised: (2) remain nonzero during distillation, potentially causing a drop in performance once the student replaces the teacher.

Original: The off-policy RL losses in eq.
Revised: The off-policy RL losses in eq.

Original: (3) allow the student to already take performance on the RL task into account, reducing or eliminating this performance drop.
Revised: (3) allow the student to already take performance on the RL task into account, reducing or eliminating this performance drop.

Original: We use PPO losses to optimise eq.
Revised: We use PPO losses to optimise eq.

Original: (3) in our experiments.
Revised: (3) in our experiments.

Original: 4. C OMBINING T RAINING AND D ISTILLATION
Revised: 4. C OMBINING T RAINING AND D ISTILLATION

Original: To fully eliminate non-stationarity during the distillation step we need to collect additional data from the environment using a ﬁxed teacher policy.
Revised: To fully eliminate non-stationarity during the distillation step we need to collect additional data from the environment using a ﬁxed teacher policy.

Original: However, this would slow down training by increasing the total number of samples required.
Revised: However, this would slow down training by increasing the total number of samples required.

Original: Here, to improve sample efﬁciency, we propose two practical implementations of ITER which reuse data between teacher and student: 
Revised: Here, to improve sample efﬁciency, we propose two practical implementations of ITER which reuse data between teacher and student: [Figure]

Original: Sequential training: Store the last N transitions that were used to update the teacher in a dataset
Revised: Sequential training: Store the last N transitions that were used to update the teacher in a dataset

Original: D .
Revised: D .

Original: During the distillation phase, draw batches from D instead of collecting new data from the environment.
Revised: During the distillation phase, draw batches from D instead of collecting new data from the environment.

Original: While this does not introduce non-stationarity, it leads to evaluating the teacher on old,[Figure] off-policy data, for which the quality of its outputs may be degraded.
Revised: While this does not introduce non-stationarity, it leads to evaluating the teacher on old, off-policy data, for which the quality of its outputs may be degraded.

Original: Furthermore, some of the data might be obsolete under the current state-distribution and we require additional memory to store D .
Revised: Furthermore, some of the data might be obsolete under the current state-distribution and we require additional memory to store D .

Original: Parallel training: Whenever the teacher is updated on a batch of data B , also update the the student on the same batch.
Revised: Parallel training: Whenever the teacher is updated on a batch of data B , also update the the student on the same batch.

Original: This approach introduces some non-stationarity as distillation is performed over multiple batches B while the teacher is changing.
Revised: This approach introduces some non-stationarity as distillation is performed over multiple batches B while the teacher is changing.

Original: However, the teacher evolves much less over the course of the distillation phase than does a policy trained from scratch to achieve the same performance.
Revised: However, the teacher evolves much less over the course of the distillation phase than does a policy trained from scratch to achieve the same performance.

Original: In practise we found this to be a worthy trade-off.
Revised: In practise we found this to be a worthy trade-off.

Original: Advantages of this method are that no additional memory D is required, the teacher is only evaluated on data on which it is currently trained and updates to the student and teacher can be performed in parallel.
Revised: Advantages of this method are that no additional memory D is required, the teacher is only evaluated on data on which it is currently trained and updates to the student and teacher can be performed in parallel.

Original: Both approaches perform similarly in our experimental validation.
Revised: Both approaches perform similarly in our experimental validation.

Original: We use parallel training for the main experiments due to the smaller memory requirements, the ability to efﬁciently perform the student distillation in parallel and because tuning the hyper-parameter was signiﬁcantly easier:
Revised: We use parallel training for the main experiments due to the smaller memory requirements, the ability to efﬁciently perform the student distillation in parallel and because tuning the hyper-parameter was signiﬁcantly easier:

Original: While tuning the size of D for sequential ITER involves trading off overﬁtting (for small D ) against off-policy data (for large D ), in parallel ITER the results were robust to the choice of how many batches B were used in the distillation phase as long as some minimum number was surpassed.
Revised: While tuning the size of D for sequential ITER involves trading off overﬁtting (for small D ) against off-policy data (for large D ), in parallel ITER the results were robust to the choice of how many batches B were used in the distillation phase as long as some minimum number was surpassed.

Original: Consequently, hyper-parameter tuning for parallel ITER only involved increasing the length of the distillation phase until no drop in student performance was observed when replacing the teacher.
Revised: Consequently, hyper-parameter tuning for parallel ITER only involved increasing the length of the distillation phase until no drop in student performance was observed when replacing the teacher.

Original: We set α π = 1 and α V = 0 .
Revised: We set α π = 1 and α V = 0 .

Original: 5 as initial values without further tuning as preliminary experiments showed no impact within reasonable ranges.
Revised: 5 as initial values without further tuning as preliminary experiments showed no impact within reasonable ranges.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In the following, we evaluate ITER on Multiroom (Chevalier-Boisvert & Willems, 2018) and on several environments from the ProcGen (Cobbe et al., 2019a) benchmark which was speciﬁcally designed to measure generalisation by introducing separate test- and training levels.
Revised: In the following, we evaluate ITER on Multiroom (Chevalier-Boisvert & Willems, 2018) and on several environments from the ProcGen (Cobbe et al., 2019a) benchmark which was speciﬁcally designed to measure generalisation by introducing separate test- and training levels.

Original: We also provide ablation studies showing that parallel and sequential training perform comparably, and that the loss terms eq.
Revised: We also provide ablation studies showing that parallel and sequential training perform comparably, and that the loss terms eq.

Original: (3) in eq.
Revised: (3) in eq.

Original: (1) are beneﬁcial.
Revised: (1) are beneﬁcial.

Original: We ﬁnd that ITER improves generalisation, which also supports our hypothesis about the negative impact of transient non-stationarity in RL .
Revised: We ﬁnd that ITER improves generalisation, which also supports our hypothesis about the negative impact of transient non-stationarity in RL .

Original: Lastly, we re-visit the SL setting from section 3 and perform additional experiments leading us to the formulation of the legacy feature hypothesis to explain the observed effects.
Revised: Lastly, we re-visit the SL setting from section 3 and perform additional experiments leading us to the formulation of the legacy feature hypothesis to explain the observed effects.

Original: 5. E XPERIMENTAL R ESULTS ON M ULTIROOM
Revised: 5. E XPERIMENTAL R ESULTS ON M ULTIROOM

Original: First, we evaluate ITER on the Multiroom environment.
Revised: First, we evaluate ITER on the Multiroom environment.

Original: The agent’s task is to traverse a sequence of rooms to reach the goal (see ﬁg. 6 for example layout) as quickly as possible.
Revised: The agent’s task is to traverse a sequence of rooms to reach the goal (see ﬁg. 6 for example layout) as quickly as possible.

Original: It can take discrete actions to rotate 90° in either direction, move forward, and open or close the doors between rooms. 
Revised: It can take discrete actions to rotate 90° in either direction, move forward, and open or close the doors between rooms. [Figure]

Original: The observation contains the full grid, one pixel per square.
Revised: The observation contains the full grid, one pixel per square.

Original: Object type, including empty space and walls, as well as any object status, like direction, are encoded in the three ‘colour’ channels.
Revised: Object type, including empty space and walls, as well as any object status, like direction, are encoded in the three ‘colour’ channels.

Original: For each episode, a new layout is generated by randomly placing between one and four connected rooms on [Figure] d T [Figure]
Revised: For each episode, a new layout is generated by randomly placing between one and four connected rooms on the grid.

Original: a

Original: [Figure]

Original: the grid.

Original: The agent is always initialised in the room furthest from the goal.
Revised: The agent is always initialised in the room furthest from the goal.

Original: This randomness favours agents that are better at generalising between layouts as memorisation is impossible due to the high number of feasible layouts.
Revised: This randomness favours agents that are better at generalising between layouts as memorisation is impossible due to the high number of feasible layouts.

Original: The results are shown in ﬁg. 2: Using ITER on top of PPO increases performance.
Revised: The results are shown in ﬁg. 2: Using ITER on top of PPO increases performance.

Original: The performance difference is more pronounced for layouts with more rooms, possibly because such layouts are harder and likely only solved later in training, at which point negative effects due to prior non-stationarity in training are more pronounced.
Revised: The performance difference is more pronounced for layouts with more rooms, possibly because such layouts are harder and likely only solved later in training, at which point negative effects due to prior non-stationarity in training are more pronounced.

Original: 5. E XPERIMENTAL R ESULTS ON S OKOBAN
Revised: 5. E XPERIMENTAL R ESULTS ON B OXOBAN

Original: We also evaluate ITER on Sokoban (Schrader, 2018) 1 .
Revised: We also evaluate ITER on Boxoban (Guez et al., 2018; Schrader, 2018) 1 .

Original: See ﬁg. 6 for an example layout.
Revised: See ﬁg. 8 for an example.

Original: Similarly to Multiroom the observation contains the full grid, one pixel per square.
Revised: Similarly to Multiroom the observation contains the full grid, one pixel per square.

Original: Object types are encoded by colour.
Revised: Object types are encoded by colour.

Original: Again, a new layout is generated at the beginning of each new episode, favouring agents that can generalise well between states.
Revised: Again, a new layout is generated at the beginning of each new episode, favouring agents that can generalise well between states.

Original: Actions allow to push, pull or move in all four cardinal directions, or do nothing.
Revised: Actions allow to push, pull or move in all four cardinal directions, or do nothing.

Original: The goal of the agent is to position the two boxes, which can be pushed or pulled, on the two available targets.
Revised: The goal of the agent is to position the four boxes, which can be pushed or pulled, on the four available targets.

Original: Walls prevent movement for both the agent and the boxes.
Revised: Walls prevent movement for both the agent and the boxes.

Original: Positive rewards are provided for positioning a box on a target ( r b = 1 ) and successfully solving each level ( r l = 10 ).
Revised: Positive rewards are provided for positioning a box on a target ( r b = 1 ) and successfully solving each level ( r l = 10 ).

Original: A small negative reward per time-step ( r s = − 0 . 1 ) encourages fast solutions.
Revised: A small negative reward per time-step ( r s = − 0 . 1 ) encourages fast solutions.

Original: The results can be seen in ﬁg. 2: ITER learns much faster and has also a slightly higher asymptotic performance.
Revised: As shown in ﬁg. 2, ITER learns much faster. We provide additional results and examples for wrongly chosen distillation lengths in ﬁg. 8.

Original: Note that both for Multiroom and Sokoban we train and test on the same (very large) set of possible layout conﬁgurations, so the main expected advantage of ITER is a more sample efﬁcient training.
Revised: Note that both for Multiroom and Boxoban we train and test on the same (very large) set of possible layout conﬁgurations, so the main expected advantage of ITER is a more sample efﬁcient training due to better generalisation.

Original: In the next section, we will evaluate the agent on previously unseen environments, directly measuring its generalisation performance.
Revised: In the next section, we will evaluate the agent on previously unseen environments, directly measuring its generalisation performance.

Original: 5. E XPERIMENTAL R ESULTS ON P ROC G EN
Revised: 5. E XPERIMENTAL R ESULTS ON P ROC G EN

Original: Next, we evaluate ITER on several environments from the ProcGen benchmark.
Revised: Next, we evaluate ITER on several environments from the ProcGen benchmark.

Original: We follow the evaluation protocol proposed in (Cobbe et al., 2019a): for each environment, we train on 500 randomly generated level layouts and test on additional, previously unseen levels.
Revised: We follow the evaluation protocol proposed in (Cobbe et al., 2019a): for each environment, we train on 500 randomly generated level layouts and test on additional, previously unseen levels.

Original: Due to computational constraints, we consider a subset of ﬁve environments.
Revised: Due to computational constraints, we consider a subset of ﬁve environments.

Original: We chose StarPilot , Dodgeball , Climber , Ninja ,
Revised: We chose StarPilot , Dodgeball , Climber , Ninja ,

Original: Fruitbot and BigFish based on the results presented in (Cobbe et al., 2019a) as they showed baseline generalisation performance better than a random policy, but with a large enough generalisation gap.
Revised: Fruitbot and BigFish based on the results presented in (Cobbe et al., 2019a) as they showed baseline generalisation performance better than a random policy, but with a large enough generalisation gap.

Original: ITER improves performance for both PPO and Information Bottleneck Actor Critic ( IBAC ) with selective noise injection (Igl et al., 2019).
Revised: ITER improves performance for both PPO and Information Bottleneck Actor Critic ( IBAC ) with selective noise injection (Igl et al., 2019).

Original: Results are presented in ﬁg. 3 and more individual plots, including performance on training levels, can be found in the appendix.
Revised: Results are presented in ﬁg. 3 and more individual plots, including performance on training levels, can be found in the appendix.

Original: In ﬁg. 4 we show in ablations  that both the parallel and sequential implementations of ITER perform comparably, while not using the off-policy RL terms L PG and L TD in eq.
Revised: In ﬁg. 4 we show in ablations [Figure] More details are given in that both the parallel and sequential implementations of ITER perform comparably, while not using the off-policy RL terms L PG and L TD in eq.

Original: (1) decreases performance.
Revised: (1) decreases performance.

Original: [Figure] More Similarly to previous literature (Cobbe et al., 2019b; Igl et al., 2019) we found that weight decay improves performance and apply it to all algorithms evaluated on ProcGen .
Revised: Similarly to previous literature (Cobbe et al., 2019b; Igl et al., 2019), we found that weight decay improves performance and apply it to all algorithms evaluated on ProcGen .

Original: Our results show that the negative effects of non-stationarity cannot easily be avoided through standard network regularisation: we can improve test returns through ITER despite regularisation with weight decay and IBAC , both shown to be among the most effective regularisation methods on this benchmark (Igl et al., 2019).
Revised: Our results show that the negative effects of non-stationarity cannot easily be avoided through standard network regularisation: we can improve test returns through ITER despite regularisation with weight decay and IBAC , both shown to be among the most effective regularisation methods on this benchmark (Igl et al., 2019).

Original: In the previous two sections we have shown the effectiveness of ITER in improving generalisation of
Revised: In the previous two sections we have shown the effectiveness of ITER in improving generalisation of

Original: RL agents.
Revised: RL agents.

Original: Because the main mechanism of ITER is in reducing the non-stationarity in the training data which is used to train the agent, this result further supports that such transient non-stationarity is detrimental to generalisation in RL .
Revised: Because the main mechanism of ITER is in reducing the non-stationarity in the training data which is used to train the agent, this result further supports that such transient non-stationarity is detrimental to generalisation in RL .

Original: 5. S UPERVISED L EARNING ON CIFAR- 
Revised: 5. S UPERVISED L EARNING ON CIFAR-

Original: [Figure]
Revised: [Figure]

Original: In this section, we aim to further understand the mechanism by which non-stationarity impacts generalisation by revisiting the easily controlled SL setting presented in section 3.
Revised: In this section [Figure] we aim to further understand the mechanism by which non-stationarity impacts generalisation by revisiting the easily controlled SL setting presented in section 3.

Original: First, we conﬁrm that, like with ITER for RL , we can improve generalisation while only learning to mimic the outputs of a poorly generalising teacher for the training data.
Revised: First, we conﬁrm that, like with ITER for RL , we can improve generalisation while only learning to mimic the outputs of a poorly generalising teacher for the training data.

Original: We train the teacher using the same setup as in section 3.
Revised: We train the teacher using the same setup as in section 3.

Original: In a second step, we train a freshly initialised student for 1000 epochs to ﬁt the argmax predictions of this teacher on the training data, i.e., the true labels in the training data are unused.
Revised: In a second step, we train a freshly initialised student for 1000 epochs to ﬁt the argmax predictions of this teacher on the training data, i.e., the true labels in the training data are unused.

Original: Test accuracy is still measured using the true labels.
Revised: Test accuracy is still measured using the true labels.

Original: The results of this distillation phase are shown in ﬁg. 5 (left).
Revised: The results of this distillation phase are shown in ﬁg. 5 (left).

Original: The student (solid lines) recovers the test accuracy achieved by stationary training, compared to the poor asymptotic teacher performance (dashed lines) from ﬁg. 1.
Revised: The student (solid lines) recovers the test accuracy achieved by stationary training, compared to the poor asymptotic teacher performance (dashed lines) from ﬁg. 1.

Original: This conﬁrms that the teacher’s outputs on the training data are suitable targets for distillation.
Revised: This conﬁrms that the teacher’s outputs on the training data are suitable targets for distillation.

Original: Second, we aim to better understand why non-stationarity affects the generalisation performance.
Revised: Second, we aim to better understand why non-stationarity affects the generalisation performance.

Original: To do so, we investigate the latent representation of the network.
Revised: To do so, we investigate the latent representation of the network.

Original: We view all but the last layer as the encoder, producing the latent representation φ ( x ) ∈ R p for input x , on which the classiﬁcation is linear: y = softmax( Wφ ( x ) + b ) with W ∈ R | C |× p denoting a weight matrix and b ∈ R | C | denoting a bias vector.
Revised: We view all but the last layer as the encoder, producing the latent representation φ ( x ) ∈ R p for input x , on which the classiﬁcation is linear: y = softmax( Wφ ( x ) + b ) with W ∈ R | C |× p denoting a weight matrix and b ∈ R | C | denoting a bias vector.

Original: By features we refer not to the representation φ ( x ) as a whole, but to aspects of the input to which the encoder learned to pay attention and which therefore can impact the latent representation (Kriegeskorte & Kievit, 2013).
Revised: By features we refer not to the representation φ ( x ) as a whole, but to aspects of the input to which the encoder learned to pay attention and which therefore can impact the latent representation (Kriegeskorte & Kievit, 2013).

Original: More quantitatively, we can deﬁne the representation matrix Φ ∈ R N × p , consisting of the latent representations of all N = 10000 test data points.
Revised: More quantitatively, we can deﬁne the representation matrix Φ ∈ R N × p , consisting of the latent representations of all N = 10000 test data points.

Original: Performing Singular Value Decomposition ( SVD ) on Φ yields mutually orthogonal directions (the right-singular vectors) in which the latent representations of the various inputs are different from one another.
Revised: Performing Singular Value Decomposition ( SVD ) on Φ yields mutually orthogonal directions (the right-singular vectors) in which the latent representations of the various inputs are different from one another.

Original: We can see each such direction as corresponding to one feature, with the corresponding singular value expressing its strength, i.e., how strongly it impacts the latent representation.
Revised: We can see each such direction as corresponding to one feature, with the corresponding singular value expressing its strength, i.e., how strongly it impacts the latent representation.

Original: Our hypothesis is that under a non-stationary data distribution, the encoder is more likely to reuse previously learned features (as these are already available) instead of learning new features from scratch.
Revised: Our hypothesis is that under a non-stationary data distribution, the encoder is more likely to reuse previously learned features (as these are already available) instead of learning new features from scratch.

Original: If these old (or ‘legacy’) features generalise worse on the new data, for example because they are overﬁt to a smaller or less diverse dataset, this in turn deteriorates generalisation permanently if they are not replaced.
Revised: If these old (or ‘legacy’) features generalise worse on the new data, for example because they are overﬁt to a smaller or less diverse dataset, this in turn deteriorates generalisation permanently if they are not replaced.

Original: This leads to two predictions: First, the observed drop in generalisation should only occur if the previously learned features are relevant for the new task, but suboptimal.
Revised: This leads to two predictions: First, the observed drop in generalisation should only occur if the previously learned features are relevant for the new task, but suboptimal.

Original: If they are irrelevant, they will not be reused.
Revised: If they are irrelevant, they will not be reused.

Original: If they are optimal, they do not negatively impact generalisation.
Revised: If they are optimal, they do not negatively impact generalisation.

Original: Second, we expect the ﬁnal network to rely on more features in its latent representation if it is reusing suboptimal features: because these are not as general, more features are required to discriminate between all inputs.
Revised: Second, we expect the ﬁnal network to rely on more features in its latent representation if it is reusing suboptimal features: because these are not as general, more features are required to discriminate between all inputs.

Original: To experimentally evaluate both predictions, we simplify the experimental setting to two phases (see ﬁg. 4 for a schematic depiction of the training setup).
Revised: To experimentally evaluate both predictions, we simplify the experimental setting to two phases (see ﬁg. 4 for a schematic depiction of the training setup).

Original: The ﬁrst training phase uses a stationary, but modiﬁed, dataset D f,m , and the second phase uses the full, unmodiﬁed, training dataset D .
Revised: The ﬁrst training phase uses a stationary, but modiﬁed, dataset D f,m , and the second phase uses the full, unmodiﬁed, training dataset D .

Original: To generate D f,m , we use the same modiﬁcations as before, m ∈ { Noisy Labels , Wrong Labels , Dataset Size } , but instead of annealing the fraction of correct data points f from 0 to 1 as in section 3, it is ﬁxed at a certain value.
Revised: To generate D f,m , we use the same modiﬁcations as before, m ∈ { Noisy Labels , Wrong Labels , Dataset Size } , but instead of annealing the fraction of correct data points f from 0 to 1 as in section 3, it is ﬁxed at a certain value.

Original: Changing this value f allows us to tune the relevance of the features learned on D f,m (also see ﬁg. 6).
Revised: Changing this value f allows us to tune the relevance of the features learned on D f,m (also see ﬁg. 6).

Original: In this setup the only non-stationarity is the change in data from phase 1 to phase 2.
Revised: In this setup the only non-stationarity is the change in data from phase 1 to phase 2.

Original: We ﬁrst train the network forepochs on D f,m , which yields an intermediate representation φ f,minter ( x ) , followed by anotherepochs on D yielding the ﬁnal representation φ f,mfinal ( x ) (see ﬁg. 7 for training curves).
Revised: We ﬁrst train the network forepochs on D f,m , which yields an intermediate representation φ f,minter ( x ) , followed by anotherepochs on D yielding the ﬁnal representation φ f,mfinal ( x ) (see ﬁg. 7 for training curves).

Original: To test our ﬁrst hypothesis, we want to measure how relevant the representation φ f,minter ( x ) is for the ﬁnal data distribution and how well φ f,mfinal ( x ) generalises.
Revised: To test our ﬁrst hypothesis, we want to measure how relevant the representation φ f,minter ( x ) is for the ﬁnal data distribution and how well φ f,mfinal ( x ) generalises.

Original: We train a linear predictor for each ﬁxed representation on the full dataset D . 

Revised: We train a linear predictor for each ﬁxed representation on the full dataset D .

Original: The test accuracy of the classiﬁer based on φ f,minter ( x ) measures how well we can perform on D with features learned on D f,m , i.e., their relevance .
Revised: The test accuracy of the classiﬁer based on φ f,minter ( x ) measures how well we can perform on D with features learned on D f,m , i.e., their relevance .

Original: The test accuracy of the classiﬁer based on φ f,mfinal measures how well the ﬁnal network was able to recover from the initial bias and learn to generalise well despite non-stationarity.
Revised: The test accuracy of the classiﬁer based on φ f,mfinal measures how well the ﬁnal network was able to recover from the initial bias and learn to generalise well despite non-stationarity.

Original: In ﬁg. 5 (middle), we plot the accuracy of the intermediate predictor (i.e., the relevance) on the x -axis and of the ﬁnal generalisation performance on the y -axis.
Revised: In ﬁg. 5 (middle), we plot the accuracy of the intermediate predictor (i.e., the relevance) on the x -axis and of the ﬁnal generalisation performance on the y -axis.

Original: Each point corresponds to one value of f ∈ (0 , 1] , shown as colour, and one modiﬁcation type m as indicated by the marker shape.
Revised: Each point corresponds to one value of f ∈ (0 , 1] , shown as colour, and one modiﬁcation type m as indicated by the marker shape.

Original: By changing f from 0 to 1 (i.e., from blue to dark red) we can increase the relevance of the intermediate features from nearly irrelevant to optimal on D .
Revised: By changing f from 0 to 1 (i.e., from blue to dark red) we can increase the relevance of the intermediate features from nearly irrelevant to optimal on D .

Original: Interestingly, an almost useless intermediate representation (30% on the x -axis) does not impede the ﬁnal performance much, while relevant but suboptimal intermediate features (around 60% on the x -axis) lead to a marked drop in performance.
Revised: Interestingly, an almost useless intermediate representation (30% on the x -axis) does not impede the ﬁnal performance much, while relevant but suboptimal intermediate features (around 60% on the x -axis) lead to a marked drop in performance.

Original: This supports our ﬁrst hypothesis.
Revised: This supports our ﬁrst hypothesis.

Original: The strong ﬁnal performance for f → 0 (i.e., for low relevance) also rules out decreased network ﬂexibility, for example due to dead neurons (for ReLUs) or saturation (for tanh), as the main driver of reduced generalisation.
Revised: The strong ﬁnal performance for f → 0 (i.e., for low relevance) also rules out decreased network ﬂexibility, for example due to dead neurons (for ReLUs) or saturation (for tanh), as the main driver of reduced generalisation.

Original: Our second prediction is that relevant, but suboptimal features in φ f,minter should lead to the usage of more features in φ f,mfinal compared to irrelevant or optimal features.
Revised: Our second prediction is that relevant, but suboptimal features in φ f,minter should lead to the usage of more features in φ f,mfinal compared to irrelevant or optimal features.

Original: To test this prediction, we plot in Figure 5 (right) the singular values (SVs) of the representation matrix Φ ∈ R N × p as deﬁned above.
Revised: To test this prediction, we plot in Figure 5 (right) the singular values (SVs) of the representation matrix Φ ∈ R N × p as deﬁned above.

Original: We plot the values for the smallest values of f (dashed lines, “irrelevant features”) and for intermediate values of f (solid lines, ‘relevant but sub-optimal features’).
Revised: We plot the values for the smallest values of f (dashed lines, “irrelevant features”) and for intermediate values of f (solid lines, ‘relevant but sub-optimal features’).

Original: The blue line corresponds to optimal features.
Revised: The blue line corresponds to optimal features.

Original: The tails of the singular values are heavier for intermediate values of f , indicating that the network is relying on more features in those cases, supporting our second prediction.
Revised: The tails of the singular values are heavier for intermediate values of f , indicating that the network is relying on more features in those cases, supporting our second prediction.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Knowledge distillation (Hinton et al., 2015) with identical teacher and student architectures has been shown to improve test accuracy (Furlanello et al., 2018), even in the absence of non-stationarities in the data.
Revised: Knowledge distillation (Hinton et al., 2015) with identical teacher and student architectures has been shown to improve test accuracy (Furlanello et al., 2018), even in the absence of non-stationarities in the data.

Original: This improvement has been attributed to the ease of predicting the output distribution of the teacher compared to the original ‘hard’ labels (Mobahi et al., 2020; Gotmare et al., 2019).
Revised: This improvement has been attributed to the ease of predicting the output distribution of the teacher compared to the original ‘hard’ labels (Mobahi et al., 2020; Gotmare et al., 2019).

Original: While we apply such ‘soft’ distillation for ITER on RL , we use ‘hard’ labels in our SL experiments.
Revised: While we apply such ‘soft’ distillation for ITER on RL , we use ‘hard’ labels in our SL experiments.

Original: Policy distillation has been applied to RL (Czarnecki et al., 2019), for example for multi-task learning and compression (Teh et al., 2017;
Revised: Policy distillation has been applied to RL (Czarnecki et al., 2019), for example for multi-task learning and compression (Teh et al., 2017;

Original: Rusu et al., 2015; Parisotto et al., 2016), imitation learning (Ross et al., 2011), or faster training (Schmitt et al., 2018; Ghosh et al., 2018).
Revised: Rusu et al., 2015; Parisotto et al., 2016), imitation learning (Ross et al., 2011), or faster training (Schmitt et al., 2018; Ghosh et al., 2018).

Original: Closer to ITER , Czarnecki et al.
Revised: Closer to ITER , Czarnecki et al.

Original: (2018) use policy distillation to learn a sequence of agents.
Revised: (2018) use policy distillation to learn a sequence of agents.

Original: However, their Mix & Match algorithm solves tasks of growing complexity, for example, to grow the action space of the agent, not to tackle generalisation or non-stationarity.
Revised: However, their Mix & Match algorithm solves tasks of growing complexity, for example, to grow the action space of the agent, not to tackle generalisation or non-stationarity.

Original: While the topic of non-stationarity is central to the area of continual learning (see (Parisi et al., 2019) for a recent review), the ﬁeld is primarily concerned with preventing catastrophic forgetting (French, 1999) when the environment or task changes during training (Li & Hoiem, 2017; Schwarz et al., 2018).
Revised: While the topic of non-stationarity is central to the area of continual learning (see (Parisi et al., 2019) for a recent review), the ﬁeld is primarily concerned with preventing catastrophic forgetting (French, 1999) when the environment or task changes during training (Li & Hoiem, 2017; Schwarz et al., 2018).

Original: For non-stationary environments during agent deployment, the approach is typically to detect such changes and respond accordingly (Choi et al., 2000; Da Silva et al., 2006; Doya et al., 2002).
Revised: For non-stationary environments during agent deployment, the approach is typically to detect such changes and respond accordingly (Choi et al., 2000; Da Silva et al., 2006; Doya et al., 2002).

Original: By contrast, we assume a stationary environment and investigate the impact of transient non-stationarity, for example induced by an improving policy.
Revised: By contrast, we assume a stationary environment and investigate the impact of transient non-stationarity, for example induced by an improving policy.

Original: We also show that intentionally forgetting the representation, but not the learned outputs, can improve generalisation in this case.
Revised: We also show that intentionally forgetting the representation, but not the learned outputs, can improve generalisation in this case.

Original: Neural networks are used in deep RL to allow generalisation across similar states (Sutton & Barto, 2018).
Revised: Neural networks are used in deep RL to allow generalisation across similar states (Sutton & Barto, 2018).

Original: Several possibilities have been proposed to further improve generalisation, including to provide more diverse training environments (Tobin et al., 2017), inject noise into the environment (Stulp et al., 2011; Lee et al., 2020), incorporate inductive biases in the architecture (Kansky et al., 2017), or regularise the network (Cobbe et al., 2019b; Igl et al., 2019; Liu et al., 2019).
Revised: Several possibilities have been proposed to further improve generalisation, including to provide more diverse training environments (Tobin et al., 2017), inject noise into the environment (Stulp et al., 2011; Lee et al., 2020), incorporate inductive biases in the architecture (Kansky et al., 2017), or regularise the network (Cobbe et al., 2019b; Igl et al., 2019; Liu et al., 2019).

Original: While regularisation reduces overﬁtting, we show in our experiments that this is insufﬁcient to counter the negative effects of non-stationarity, and that ITER can be complementary to other types of regularisation.
Revised: While regularisation reduces overﬁtting, we show in our experiments that this is insufﬁcient to counter the negative effects of non-stationarity, and that ITER can be complementary to other types of regularisation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we investigate the impact of non-stationarity on the generalisation performance of trained
Revised: In this work, we investigate the impact of non-stationarity on the generalisation performance of trained

Original: RL agents.
Revised: RL agents.

Original: First, in several SL experiments on the CIFAR-10 dataset, we conﬁrm that non-stationarity can considerably degrade test performance while leaving training performance nearly unchanged.
Revised: First, in several SL experiments on the CIFAR-10 dataset, we conﬁrm that non-stationarity can considerably degrade test performance while leaving training performance nearly unchanged.

Original: To explain this effect, we propose and experimentally support the legacy feature hypothesis that networks exhibit a tendency to adopt, rather then forget, features learned earlier during training if they are sufﬁciently relevant, though not necessarily optimal, for the new data.
Revised: To explain this effect, we propose and experimentally support the legacy feature hypothesis that networks exhibit a tendency to adopt, rather then forget, features learned earlier during training if they are sufﬁciently relevant, though not necessarily optimal, for the new data.

Original: We also show that self-distillation, even without using the true training labels, improves performance on the test-data.
Revised: We also show that self-distillation, even without using the true training labels, improves performance on the test-data.

Original: Many deep RL algorithms induce similar transient non-stationarity, for example due to a gradually converging policy.
Revised: Many deep RL algorithms induce similar transient non-stationarity, for example due to a gradually converging policy.

Original: Consequently, to improve generalisation in deep RL , we propose ITER which reduces the non-stationarity the agent networks experience during training.
Revised: Consequently, to improve generalisation in deep RL , we propose ITER which reduces the non-stationarity the agent networks experience during training.

Original: Our experimental results on the Multiroom and ProcGen benchmarks empirically support the beneﬁts of ITER , indicating that transient non-stationarity has a negative impact in deep RL .
Revised: Our experimental results on the Multiroom and ProcGen benchmarks empirically support the beneﬁts of ITER , indicating that transient non-stationarity has a negative impact in deep RL .

Original: A P SEUDO CODE
Revised: A P SEUDO CODE

Original: 1 Input Length of initial RL training phase t init , length of distillation phase t distill 2 Initialise k ← 0 , policy π ( k ) , value function V ( k ) 3 // Normal RL training at the beginning 4 for t init steps do
Revised: 1 Input Length of initial RL training phase t init , length of distillation phase t distill 2 Initialise k ← 0 , policy π ( k ) , value function V ( k ) 3 // Normal RL training at the beginning 4 for t init steps do

Original: B ← collect trajectory data using π (0)
Revised: B ← collect trajectory data using π (0)

Original: Update π (0) and V (0) using standard RL method using B 7 // Combine further RL training of π ( k ) , V ( k ) with distillation of π ( k +1) , V ( k +1) 8 while not converged do
Revised: Update π (0) and V (0) using standard RL method using B 7 // Combine further RL training of π ( k ) , V ( k ) with distillation of π ( k +1) , V ( k +1) 8 while not converged do

Original: Initialise student policy π ( k +1) and value function V ( k +1) for t distill steps do α V , α π ← linear annealing to 0 over t distill steps
Revised: Initialise student policy π ( k +1) and value function V ( k +1) for t distill steps do α V , α π ← linear annealing to 0 over t distill steps

Original: B ← collect trajectory data using π ( k )
Revised: B ← collect trajectory data using π ( k )

Original: Update π ( k ) and V ( k ) with standard RL method using B
Revised: Update π ( k ) and V ( k ) with standard RL method using B

Original: Update π ( k +1) and V ( k +1) with eq.
Revised: Update π ( k +1) and V ( k +1) with eq.

Original: (1) using B , α V , α π , π ( k ) and V ( k ) //
Revised: (1) using B , α V , α π , π ( k ) and V ( k ) //

Original: Housekeeping
Revised: Housekeeping

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B S UPERVISED L EARNING
Revised: B S UPERVISED L EARNING

Original: Training
Revised: Training

Original: Testing
Revised: Testing

Original: Error in %
Revised: Error in %

Original: Rel.
Revised: Rel.

Original: Error in %
Revised: Error in %

Original: Rel.
Revised: Rel.

Original: Unmodiﬁed 0 .
Revised: Unmodiﬁed 0 .

Original: 17 ± 0 .1.
Revised: 17 ± 0 .1.

Original: 14 . 8 ± 0 .1.
Revised: 14 . 8 ± 0 .1.

Original: Noisy Labels 0 .
Revised: Noisy Labels 0 .

Original: 19 ± 0 .1.
Revised: 19 ± 0 .1.

Original: 8 ± 0 .1.
Revised: 8 ± 0 .1.

Original: Wrong Labels 0 .
Revised: Wrong Labels 0 .

Original: 20 ± 0 .1.
Revised: 20 ± 0 .1.

Original: 2 ± 0 .1.
Revised: 2 ± 0 .1.

Original: [Table] [Table]
Revised: Dataset [Table]

Original: [Table]
Revised: [Table]

Original: Here we provide additional training details and results for the supervised learning experiments performed on the CIFAR-10 dataset.
Revised: Here we provide additional training details and results for the supervised learning experiments performed on the CIFAR-10 dataset.

Original: We used a ResNet18 architecture without Batchnorm, hyperparameters for the SGD optimiser are given in table 2.
Revised: We used a ResNet18 architecture without Batchnorm, hyperparameters for the SGD optimiser are given in table 2.

Original: In table 1 we provide exact numerical values for the results in ﬁg. 1.
Revised: In table 1 we provide exact numerical values for the results in ﬁg. 1.

Original: We also provide values for the relative change in error rate due to the introduction of non-stationarities, for which the test-performance is also more affected than the train performance.
Revised: We also provide values for the relative change in error rate due to the introduction of non-stationarities, for which the test-performance is also more affected than the train performance.

Original: In ﬁg. 6, we show the same results as in ﬁg. 5, but here showing the f values used to generate D f,m on the x-Axis.
Revised: In ﬁg. 6, we show the same results as in ﬁg. 5, but here showing the f values used to generate D f,m on the x-Axis.

Original: The same ‘dips’ in performance are visible, however from this ﬁgure it is clear that
Revised: The same ‘dips’ in performance are visible, however from this ﬁgure it is clear that

Original: Dataset Size experiences it for much smaller values of f , which is unsurprising, giving the missing inﬂuences of a diverse input-data distribution.
Revised: Dataset Size experiences it for much smaller values of f , which is unsurprising, giving the missing inﬂuences of a diverse input-data distribution.

Original: Lastly, in ﬁg. 7, we provide the individual training runs used to generate ﬁg. 5(middle) and ﬁg. 6. 
Revised: Lastly, in ﬁg. 7, we provide the individual training runs used to generate ﬁg. 5(middle) and ﬁg. 6. [Figure]

Original: B. M ULTIROOM
Revised: B. M ULTIROOM

Original: In table 3 we show the hyper-parameters used for the Multiroom experiments which are shared between ‘PPO’ and ‘PPO+ITER’.
Revised: In table 3 we show the hyper-parameters used for the Multiroom experiments which are shared between ‘PPO’ and ‘PPO+ITER’.

Original: We note that our Multiroom environment uses the same modiﬁ- [Figure] [Figure] cation that was used in (Igl et al., 2019) to make the environment fully observable.
Revised: We note that our Multiroom environment uses the same modiﬁcation that was used in (Igl et al., 2019) to make the environment fully observable.

Original: In the original environment, the agent only observed its immediate surrounding from an ego-centric perspective, thereby naturally generalising across various layouts.
Revised: In the original environment, the agent only observed its immediate surrounding from an ego-centric perspective, thereby naturally generalising across various layouts.

Original: Instead, full observability introduces the need to learn how to generalise.
Revised: Instead, full observability introduces the need to learn how to generalise.

Original: Our network consists of a three layer CNN With 16, 32 and 32 ﬁlters respectively, followed by a fully connected layer of size 64.
Revised: Our network consists of a three layer CNN With 16, 32 and 32 ﬁlters respectively, followed by a fully connected layer of size 64.

Original: One max-pooling layer is used after the ﬁrst CNN layer.
Revised: One max-pooling layer is used after the ﬁrst CNN layer.

Original: We use t init = 4 × 10 7 and t distill = 4 × 10 7 (see algorithm 1) for the duration of the initial RL training phase and the following distillation phases.
Revised: We use t init = 4 × 10 7 and t distill = 4 × 10 7 (see algorithm 1) for the duration of the initial RL training phase and the following distillation phases.

Original: B. S OKOBAN
Revised: B. B OXOBAN

Original: For Sokoban, we re-use the same  hyper-parameters as for Multiroom, but with a reduced learning rate ( 1 . 

Revised: For Boxoban, we re-use the same architecture and hyper-parameters as for Multiroom, but with a reduced learning rate ( 1 .

Original: e − 04 )and doubled minibatch and batch size in order to stabilise training.
Revised: e − 04 ) in order to stabilise training.

Original: We re-use the

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  same architecture but provide all CNN layers with 64 ﬁlters and remove the max-pooling layer due to the smaller grid-size of Sokoban.
Revised: [Figure] In ﬁg. 8 we show results for wrongly chosen distillation lengths. Too short uses the same distillation length as the main results in ﬁg. 2, but continues distilling until the end of training. Because after 0 .

Revised: 5 e 9 steps the policy performance and complexity increase considerably, a longer distillation period would be required.

Revised: On the other hand, using a distillation period twice as long from the start ( Too long ) leads to slower training.

Revised: We did not experiment with increasing the distillation length over the course of training, since in our experiments earlier stopping of ITER was sufﬁcient for optimal performance.

Original: B. P ROC G EN
Revised: B. P ROC G EN

Original: In ﬁg. 8 we show all results on the various ProcGen environment from which the summary plots in the main text (ﬁgs. 3 and 4) are computed.
Revised: In ﬁg. 9 we show all results on the various ProcGen environment from which the summary plots in the main text (ﬁgs. 3 and 4) are computed.

Original: We use the same (small) IMPALA architecture as used by (Cobbe et al., 2019b).
Revised: We use the same (small) IMPALA architecture as used by (Cobbe et al., 2019b).

Original: Training is done on 4 GPUs in parallel.
Revised: Training is done on 4 GPUs in parallel.

Original: One GPU is continuously evaluating the test performance, the other three are used for training.
Revised: One GPU is continuously evaluating the test performance, the other three are used for training.

Original: Their gradients are averaged at each update step.
Revised: Their gradients are averaged at each update step.

Original: The hyper-parameters given in table 4 are per GPU .
Revised: The hyper-parameters given in table 4 are per GPU .

Original: The x -Axis in ﬁg. 8 shows the total number of consumed frames, i.e. 250 × 10 6 per training GPU.
Revised: The x -Axis in ﬁg. 9 shows the total number of consumed frames, i.e. 250 × 10 6 per training GPU.

Original: The distillation phase takes t distill = 70 × 10 6 frames (again per GPU) and we linearly anneal α π from 1 to 0 and α V from 0.
Revised: The distillation phase takes t distill = 70 × 10 6 frames (again per GPU) and we linearly anneal α π from 1 to 0 and α V from 0.

Original: to 1.
Revised: to 1.

Original: The values of α π and α V were chosen to reﬂect the relative weight between L PG and L TD in eq.
Revised: The values of α π and α V were chosen to reﬂect the relative weight between L PG and L TD in eq.

Original: (1) and no further tuning was done.
Revised: (1) and no further tuning was done.

Original: The initial RL training phase takes t init =
Revised: The initial RL training phase takes t init =

Original: 50 × 10 6 frames.
Revised: 50 × 10 6 frames.

Original: The distillation length was chosen based on preliminary experiments on BigFish by increasing its length in steps of 10 × 10 6 frames until no drop in training performance was experienced when switching to a new student.
Revised: The distillation length was chosen based on preliminary experiments on BigFish by increasing its length in steps of 10 × 10 6 frames until no drop in training performance was experienced when switching to a new student.

Original: Due to the high computation costs of running experiments on the ProcGen environment (4 GPUs for about 24h for each run), we decided to exclude environments from the original benchmark based on results presented by Cobbe et al. (2019a), ﬁgures 2 and 4.
Revised: Due to the high computation costs of running experiments on the ProcGen environment (4 GPUs for about 24h for each run) [Figure] we decided to exclude environments from the original benchmark based on results presented by Cobbe et al. (2019a) [Figure]

Revised: ﬁgures 2 and 4.

Original: We excluded environments for two different reasons, either because the generalisation gap was small (Chaser, Miner, Leaper, Boss Fight,
Revised: We excluded environments for two different reasons, either because the generalisation gap was small (Chaser, Miner, Leaper, Boss Fight,

Original: Fruitbot) or because generalisation did not improve at all during training after a very short initial jump (CaveFlyer, Maze, Heist, Plunder, Coinrun), indicating that either it was too hard, or a very simple policy already achieved reasonable performance.
Revised: Fruitbot) or because generalisation did not improve at all during training after a very short initial jump (CaveFlyer, Maze, Heist, Plunder, Coinrun) [Figure] indicating that either it was too hard, or a very simple policy already achieved reasonable performance

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]