Original: A DAPTING B EHAVIOUR FOR L EARNING P ROGRESS
Revised: A DAPTING B EHAVIOUR FOR L EARNING P ROGRESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning.
Revised: Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning.

Original: The advent of distributed agents that interact with parallel instances of the environment has enabled larger scale and greater ﬂexibility, but has not removed the need to tune or tailor exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning.
Revised: The advent of distributed agents that interact with parallel instances of the environment has enabled larger scale and greater ﬂexibility, but has not removed the need to tune or tailor exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning.

Original: We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress.
Revised: We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress.

Original: The data distribution is controlled via modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without signiﬁcant overhead.
Revised: The data distribution is controlled via modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without signiﬁcant overhead.

Original: The adaptation speed of the bandit can be increased by exploiting the factored modulation structure.
Revised: The adaptation speed of the bandit can be increased by exploiting the factored modulation structure.

Original: We demonstrate on a suite of Atari 2600 games how this uniﬁed approach produces results comparable to per-task tuning at a fraction of the cost.
Revised: We demonstrate on a suite of Atari 2600 games how this uniﬁed approach produces results comparable to per-task tuning at a fraction of the cost.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) is a general formalism modelling sequential decision making.
Revised: Reinforcement learning (RL) is a general formalism modelling sequential decision making.

Original: It aspires to be broadly applicable, making minimal assumptions about the task at hand and reducing the need for prior knowledge.
Revised: It aspires to be broadly applicable, making minimal assumptions about the task at hand and reducing the need for prior knowledge.

Original: By learning behaviour from scratch, it has the potential to surpass human expertise or tackle complex domains where human intuition is not applicable.
Revised: By learning behaviour from scratch, it has the potential to surpass human expertise or tackle complex domains where human intuition is not applicable.

Original: In practice, however, generality is often traded for performance and efﬁciency, with RL practitioners tuning algorithms, architectures and hyper-parameters to the task at hand (Hessel et al., 2019).
Revised: In practice, however, generality is often traded for performance and efﬁciency, with RL practitioners tuning algorithms, architectures and hyper-parameters to the task at hand (Hessel et al., 2019).

Original: A side-effect of this is that the resulting methods can be brittle, or difﬁcult to reliably reproduce (Nagarajan et al., 2018).
Revised: A side-effect of this is that the resulting methods can be brittle, or difﬁcult to reliably reproduce (Nagarajan et al., 2018).

Original: Exploration is one of the main aspects commonly designed or tuned speciﬁcally for the task being solved.
Revised: Exploration is one of the main aspects commonly designed or tuned speciﬁcally for the task being solved.

Original: Previous work has shown that large sample-efﬁciency gains are possible, for example, when the exploratory behaviour’s level of stochasticity is adjusted to the environment’s hazard rate (Garc´ıa & Fern´andez, 2015), or when an appropriate prior is used in large action spaces (DulacArnold et al., 2015; Czarnecki et al., 2018; Vinyals et al., 2019).
Revised: Previous work has shown that large sample-efﬁciency gains are possible, for example, when the exploratory behaviour’s level of stochasticity is adjusted to the environment’s hazard rate (Garc´ıa & Fern´andez, 2015), or when an appropriate prior is used in large action spaces (DulacArnold et al., 2015; Czarnecki et al., 2018; Vinyals et al., 2019).

Original: Ideal exploration in the presence of function approximation should be agent-centred.
Revised: Ideal exploration in the presence of function approximation should be agent-centred.

Original: It ought to focus more on generating data that supports the learning of agent at its current parameters θ , rather than making progress on objective measurements of information gathering.
Revised: It ought to focus more on generating data that supports the learning of agent at its current parameters θ , rather than making progress on objective measurements of information gathering.

Original: A useful notion here is learning progress ( LP ), deﬁned as the improvement of the learned policy π θ (Section 3).
Revised: A useful notion here is learning progress ( LP ), deﬁned as the improvement of the learned policy π θ (Section 3).

Original: The agent’s source of data is its behaviour policy.
Revised: The agent’s source of data is its behaviour policy.

Original: Beyond the conventional RL setting of a single stream of experience, distributed agents that interact with parallel copies of the environment can have multiple such data sources (Horgan et al., 2018).
Revised: Beyond the conventional RL setting of a single stream of experience, distributed agents that interact with parallel copies of the environment can have multiple such data sources (Horgan et al., 2018).

Original: In this paper, we restrict ourselves to the setting where all behaviour policies are derived from a single set of learned parameters θ , for example when θ parameterises an action-value function Q θ .
Revised: In this paper, we restrict ourselves to the setting where all behaviour policies are derived from a single set of learned parameters θ , for example when θ parameterises an action-value function Q θ .

Original: Consequently the behaviour policies are given by π ( Q θ , z ) , where each modulation z leads to meaningfully different behaviour.
Revised: Consequently the behaviour policies are given by π ( Q θ , z ) , where each modulation z leads to meaningfully different behaviour.

Original: This can be guaranteed if z is semantic (e.g. degree of stochasticity) and consistent across multiple time-steps.
Revised: This can be guaranteed if z is semantic (e.g. degree of stochasticity) and consistent across multiple time-steps.

Original: The latter is achieved by holding z ﬁxed throughout each episode (Section 2).
Revised: The latter is achieved by holding z ﬁxed throughout each episode (Section 2).

Original: We propose to estimate a proxy that is indicative of future learning progress, f ( z ) (Section 3), separately for each modulation z , and to adapt the distribution over modulations to maximize f , using a non-stationary multi-armed bandit that can exploit the factored structure of the modulations (Section 4).
Revised: We propose to estimate a proxy that is indicative of future learning progress, f ( z ) (Section 3), separately for each modulation z , and to adapt the distribution over modulations to maximize f , using a non-stationary multi-armed bandit that can exploit the factored structure of the modulations (Section 4).

Original: Figure 1 shows a diagram of all these components.
Revised: Figure 1 shows a diagram of all these components.

Original: This results in an autonomous adaptation of behaviour to the agent’s stage of learning (Section 5), varying across tasks and across time, and reducing the need for hyper-parameter tuning.
Revised: This results in an autonomous adaptation of behaviour to the agent’s stage of learning (Section 5), varying across tasks and across time, and reducing the need for hyper-parameter tuning.

Original:  M ODULATED B EHAVIOUR
Revised:  M ODULATED B EHAVIOUR

Original: As usual in RL, the objective of the agent is to ﬁnd a policy π that maximises the γ -discounted expected return G t .
Revised: As usual in RL, the objective of the agent is to ﬁnd a policy π that maximises the γ -discounted expected return G t .

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: i =

Revised: i =

Original: γ i R t
Revised: γ i R t

Original: + i , where R t is the reward obtained during the transition from time t to t + 1 .
Revised: + i , where R t is the reward obtained during the transition from time t to t + 1 .

Original: A common way to address this problem is to use methods that compute the actionvalue function Q π given by Q π ( s, a ) .
Revised: A common way to address this problem is to use methods that compute the actionvalue function Q π given by Q π ( s, a ) .

Original: =
Revised: =

Original: E [ G t | s, a ] , i.e. the expected return when starting from state s with action a
Revised: E [ G t | s, a ] , i.e. the expected return when starting from state s with action a

Original: and then following π (Puterman, 1994).
Revised: and then following π (Puterman, 1994).

Original: A richer representation of Q π that aims to capture more information about the underlying distribution of G t has been proposed by Bellemare et al.
Revised: A richer representation of Q π that aims to capture more information about the underlying distribution of G t has been proposed by Bellemare et al.

Original: (2017), and extended by Dabney et al.
Revised: (2017), and extended by Dabney et al.

Original: Instead of approximating only the mean of the return distribution, we approximate a discrete set of n quantile values q ν (where ν ∈ { 12 n , 32 n , . . .
Revised: Instead of approximating only the mean of the return distribution, we approximate a discrete set of n quantile values q ν (where ν ∈ { 12 n , 32 n , . . .

Original: , 2 n − 1 2 n } ) such that P ( Q π ≤ q ν )
Revised: , 2 n − 1 2 n } ) such that P ( Q π ≤ q ν )

Original: = ν .
Revised: = ν .

Original: Outside the beneﬁts in performance and representation learning (Such et al., 2018), these quantile estimates provide a way of inducing risk-sensitive behaviour.
Revised: Outside the beneﬁts in performance and representation learning (Such et al., 2018), these quantile estimates provide a way of inducing risk-sensitive behaviour.

Original: We approximate all q ν using a single deep neural network with parameters θ , and deﬁne the evaluation policy as the greedy one with respect to the mean estimate:
Revised: We approximate all q ν using a single deep neural network with parameters θ , and deﬁne the evaluation policy as the greedy one with respect to the mean estimate:

Original: [Equation]
Revised: [Equation]

Original: The behaviour policy is the central element of exploration: it generates exploratory behaviour (and experience therefrom) which is used to learn π θ ; ideally in such a way as to reduce the total amount of experience required to achieve good performance.
Revised: The behaviour policy is the central element of exploration: it generates exploratory behaviour (and experience therefrom) which is used to learn π θ ; ideally in such a way as to reduce the total amount of experience required to achieve good performance.

Original: Instead of a single monolithic behaviour policy, we propose to use a modulated policy to support parameterized variation.
Revised: Instead of a single monolithic behaviour policy, we propose to use a modulated policy to support parameterized variation.

Original: Its modulations z should satisfy the following criteria: they need to (i) be impactful, having a direct and meaningful effect on generated behaviour; (ii) have small dimensionality, as to quickly adapt to the needs of the learning algorithm, and interpretable semantics to ease the choice of viable ranges and initialisation; and (iii) be frugal, in the sense that they are relatively simple and computationally inexpensive to apply.
Revised: Its modulations z should satisfy the following criteria: they need to (i) be impactful, having a direct and meaningful effect on generated behaviour; (ii) have small dimensionality, as to quickly adapt to the needs of the learning algorithm, and interpretable semantics to ease the choice of viable ranges and initialisation; and (iii) be frugal, in the sense that they are relatively simple and computationally inexpensive to apply.

Original: In this work, we consider ﬁve concrete types of such modulations:
Revised: In this work, we consider ﬁve concrete types of such modulations:

Original: Temperature: a Boltzmann softmax policy based on action-logits, modulated by temperature, T .
Revised: Temperature: a Boltzmann softmax policy based on action-logits, modulated by temperature, T .

Original: Flat stochasticity: with probability (cid:15) the agent ignores the action distribution produced by the softmax, and samples an action uniformly at random ( (cid:15) -greedy).
Revised: Flat stochasticity: with probability (cid:15) the agent ignores the action distribution produced by the softmax, and samples an action uniformly at random ( (cid:15) -greedy).

Original: Per-action biases: action-logit offsets, b , to bias the agent to prefer some actions.
Revised: Per-action biases: action-logit offsets, b , to bias the agent to prefer some actions.

Original: Action-repeat probability: with probability ρ , the previous action is repeated (Machado et al., 2017).
Revised: Action-repeat probability: with probability ρ , the previous action is repeated (Machado et al., 2017).

Original: This produces chains of repeated actions with expected length 11 − ρ .
Revised: This produces chains of repeated actions with expected length 11 − ρ .

Original: Optimism: as the value function is represented by quantiles q ν , the aggregate estimate Q ω can be parameterised by an optimism exponent ω , such that ω = 0 recovers the default ﬂat average, while positive values of ω imply optimism and negative ones pessimism.
Revised: Optimism: as the value function is represented by quantiles q ν , the aggregate estimate Q ω can be parameterised by an optimism exponent ω , such that ω = 0 recovers the default ﬂat average, while positive values of ω imply optimism and negative ones pessimism.

Original: When near risk-neutral, our simple risk measure produces qualitatively similar transforms to those of Wang (2000).
Revised: When near risk-neutral, our simple risk measure produces qualitatively similar transforms to those of Wang (2000).

Original: We combine the above modulations to produce the overall z -modulated policy
Revised: We combine the above modulations to produce the overall z -modulated policy

Original: [Equation]
Revised: [Equation]

Original: where z .
Revised: where z .

Original: =
Revised: =

Original: ( T, (cid:15), b , ρ, ω ) , I x is the indicator function, and the optimism-aggregated value is
Revised: ( T, (cid:15), b , ρ, ω ) , I x is the indicator function, and the optimism-aggregated value is

Original: [Equation]
Revised: [Equation]

Original: Now that the behaviour policy can be modulated, the following two sections discuss the criteria and mechanisms for choosing modulations z .
Revised: Now that the behaviour policy can be modulated, the following two sections discuss the criteria and mechanisms for choosing modulations z .

Original:  E XPLORATION & THE E FFECTIVE A CQUISITION OF I NFORMATION
Revised:  E XPLORATION & THE E FFECTIVE A CQUISITION OF I NFORMATION

Original: A key component of a successful reinforcement learning algorithm is the ability to acquire experience (information) that allows it to make expeditious progress towards its objective of learning to act in the environment in such a way as to optimise returns over the relevant (potentially discounted) horizon.
Revised: A key component of a successful reinforcement learning algorithm is the ability to acquire experience (information) that allows it to make expeditious progress towards its objective of learning to act in the environment in such a way as to optimise returns over the relevant (potentially discounted) horizon.

Original: The types of experience that most beneﬁt an agent’s ultimate performance may differ qualitatively throughout the course of learning — a behaviour modulation that is beneﬁcial in the beginning of training often enough does not carry over to the end, as illustrated by the analysis in Figure 5.
Revised: The types of experience that most beneﬁt an agent’s ultimate performance may differ qualitatively throughout the course of learning — a behaviour modulation that is beneﬁcial in the beginning of training often enough does not carry over to the end, as illustrated by the analysis in Figure 5.

Original: However, this analysis was conducted in hindsight, and in general how to generate such experience optimally — optimal exploration in any environment — remains an open problem.
Revised: However, this analysis was conducted in hindsight, and in general how to generate such experience optimally — optimal exploration in any environment — remains an open problem.

Original: One approach is to require exploration to be in service of the agent’s future learning progress ( LP ), and to optimise this quantity during learning.
Revised: One approach is to require exploration to be in service of the agent’s future learning progress ( LP ), and to optimise this quantity during learning.

Original: Although there are multiple ways of deﬁning learning progress, in this work we opted for a task-related measure, namely the improvement of the policy in terms of expected return.
Revised: Although there are multiple ways of deﬁning learning progress, in this work we opted for a task-related measure, namely the improvement of the policy in terms of expected return.

Original: This choice of measure corresponds to the local steepness of the learning curve of the evaluation policy π θ ,
Revised: This choice of measure corresponds to the local steepness of the learning curve of the evaluation policy π θ ,

Original: [Equation]
Revised: [Equation]

Original: (1) where the expectation is over start states s 0 , the value V π ( s )
Revised: (1) where the expectation is over start states s 0 , the value V π ( s )

Original: =
Revised: =

Original: E π [ (cid:80) γ i R i | s 0 = s ] is the γdiscounted return one would expected to obtain, starting in state s and following policy
Revised: E π [ (cid:80) γ i R i | s 0 = s ] is the γdiscounted return one would expected to obtain, starting in state s and following policy

Original: π afterwards, and ∆ θ is the change in the agent’s parameters.
Revised: π afterwards, and ∆ θ is the change in the agent’s parameters.

Original: Note that this is still a limited criterion, as it is myopic and might be prone to local optima.
Revised: Note that this is still a limited criterion, as it is myopic and might be prone to local optima.

Original: As prefaced in the last section, our goal here is to deﬁne a mechanism that can switch between different behaviour modulations depending on which of them seems most promising at this point in the training process.
Revised: As prefaced in the last section, our goal here is to deﬁne a mechanism that can switch between different behaviour modulations depending on which of them seems most promising at this point in the training process.

Original: Thus in order to adapt the distribution over modulations z , we want to assess the expected LP when learning from data generated according to z -modulated behaviour:
Revised: Thus in order to adapt the distribution over modulations z , we want to assess the expected LP when learning from data generated according to z -modulated behaviour:

Original: [Equation]
Revised: [Equation]

Original: with ∆ θ ( τ, t ) the weight-change of learning from trajectory τ at time t .
Revised: with ∆ θ ( τ, t ) the weight-change of learning from trajectory τ at time t .

Original: This is a subjective utility measure, quantifying how useful τ is for a particular learning algorithm, at this stage in training.
Revised: This is a subjective utility measure, quantifying how useful τ is for a particular learning algorithm, at this stage in training.

Original: Proxies for learning progress: While LP ( z ) is a simple and clear progress metric, it is not readily available during training, so that in practice, a proxy ﬁtness f t ( z )
Revised: Proxies for learning progress: While LP ( z ) is a simple and clear progress metric, it is not readily available during training, so that in practice, a proxy ﬁtness f t ( z )

Original: ≈ LP t ( z ) needs to be used.
Revised: ≈ LP t ( z ) needs to be used.

Original: A key practical challenge is to construct f t from inexpensively measurable proxies, in a way that is sufﬁciently informative to effectively adapt the distribution over z , while being robust to noise, approximation error, state distribution shift and mismatch between the proxies and learning progress.
Revised: A key practical challenge is to construct f t from inexpensively measurable proxies, in a way that is sufﬁciently informative to effectively adapt the distribution over z , while being robust to noise, approximation error, state distribution shift and mismatch between the proxies and learning progress.

Original: The ideal choice of f ( z ) is a matter of empirical study, and this paper only scratches the surface on this topic.
Revised: The ideal choice of f ( z ) is a matter of empirical study, and this paper only scratches the surface on this topic.

Original: After some initial experimentation, we opted for the simple proxy of empirical (undiscounted) episodic return: f t ( z ) = (cid:80) a i ∼ π
Revised: After some initial experimentation, we opted for the simple proxy of empirical (undiscounted) episodic return: f t ( z ) = (cid:80) a i ∼ π

Original: ( Q θt ,z ) R i .
Revised: ( Q θt ,z ) R i .

Original: This is trivial to estimate, but it departs from LP ( z ) in a number of ways.
Revised: This is trivial to estimate, but it departs from LP ( z ) in a number of ways.

Original: First, it does not contain learner-subjective information, but this is partly mitigated through the joint use of with prioritized replay (see Section 5.1) that over-samples high error experience.
Revised: First, it does not contain learner-subjective information, but this is partly mitigated through the joint use of with prioritized replay (see Section 5.1) that over-samples high error experience.

Original: Another potential mechanism by which the episodic return can be indicative of future learning is because an improved policy tends to be preceded by some higher-return episodes – in general, there is a lag between best-seen performance and reliably reproducing it.
Revised: Another potential mechanism by which the episodic return can be indicative of future learning is because an improved policy tends to be preceded by some higher-return episodes – in general, there is a lag between best-seen performance and reliably reproducing it.

Original: Second, the ﬁtness is based on absolute returns not differences in returns as suggested by Equation 1; this makes no difference to the relative orderings of z (and the resulting probabilities induced by the bandit), but it has the beneﬁt that the non-stationarity takes a different form: a difference-based metric will appear stationary if the policy performance keeps increasing at a steady rate, but such a policy must be changing signiﬁcantly to achieve that progress, and therefore the selection mechanism should keep revisiting other modulations.
Revised: Second, the ﬁtness is based on absolute returns not differences in returns as suggested by Equation 1; this makes no difference to the relative orderings of z (and the resulting probabilities induced by the bandit), but it has the beneﬁt that the non-stationarity takes a different form: a difference-based metric will appear stationary if the policy performance keeps increasing at a steady rate, but such a policy must be changing signiﬁcantly to achieve that progress, and therefore the selection mechanism should keep revisiting other modulations.

Original: In contrast, our absolute ﬁtness naturally has this effect when paired with a non-stationary bandit, as described in the next section.
Revised: In contrast, our absolute ﬁtness naturally has this effect when paired with a non-stationary bandit, as described in the next section.

Original:  N ON - STATIONARY B ANDIT T AILORED TO L EARNING P ROGRESS
Revised:  N ON - STATIONARY B ANDIT T AILORED TO L EARNING P ROGRESS

Original: The most effective modulation scheme may differ throughout the course of learning.
Revised: The most effective modulation scheme may differ throughout the course of learning.

Original: Instead of applying a single ﬁxed modulation or ﬁxed blend, we propose an adaptive scheme, in which the choice of modulation is dynamically based on learning progress.
Revised: Instead of applying a single ﬁxed modulation or ﬁxed blend, we propose an adaptive scheme, in which the choice of modulation is dynamically based on learning progress.

Original: The adaptation process is based on a non-stationary multi-armed bandit (Besbes et al., 2014; Raj & Kalyani, 2017), where each arm corresponds to a behaviour modulation z .
Revised: The adaptation process is based on a non-stationary multi-armed bandit (Besbes et al., 2014; Raj & Kalyani, 2017), where each arm corresponds to a behaviour modulation z .

Original: The non-stationarity reﬂects the nature of the learning progress LP t ( z ) which depends on the time t in training through the parameters θ t .
Revised: The non-stationarity reﬂects the nature of the learning progress LP t ( z ) which depends on the time t in training through the parameters θ t .

Original: Because of non-stationarity, the core challenge for such bandit is to identify good modulation arms quickly, while only having access to a noisy, indirect proxy f t ( z ) of the quantity of interest LP t ( z ) .
Revised: Because of non-stationarity, the core challenge for such bandit is to identify good modulation arms quickly, while only having access to a noisy, indirect proxy f t ( z ) of the quantity of interest LP t ( z ) .

Original: However, our setting also presents an unusual advantage: the bandit does not need to identify the best z , as in practice it sufﬁces to spread probability among all arms that produce reasonably useful experience for learning.
Revised: However, our setting also presents an unusual advantage: the bandit does not need to identify the best z , as in practice it sufﬁces to spread probability among all arms that produce reasonably useful experience for learning.

Original: Concretely, our bandit samples a modulation z ∈ { z 1 , . .
Revised: Concretely, our bandit samples a modulation z ∈ { z 1 , . .

Original: , z K } according to the probability that it results in higher than usual ﬁtness (measured as the mean over a recent length-
Revised: , z K } according to the probability that it results in higher than usual ﬁtness (measured as the mean over a recent length-

Original: h window):
Revised: h window):

Original: [Equation]
Revised: [Equation]

Original: where m t .
Revised: where m t .

Original: = 1 h
Revised: = 1 h

Original: [Equation]
Revised: [Equation]

Original: Note that m t depends on the payoffs of the actually sampled modulations z t − h : t − 1 , allowing the bandit to become progressively more selective (if m t keeps increasing).
Revised: Note that m t depends on the payoffs of the actually sampled modulations z t − h : t − 1 , allowing the bandit to become progressively more selective (if m t keeps increasing).

Original: Estimation: For simplicity, P t ( z ) is inferred based on the empirical data within a recent time window of the same horizon h that is used to compute m t .
Revised: Estimation: For simplicity, P t ( z ) is inferred based on the empirical data within a recent time window of the same horizon h that is used to compute m t .

Original: Concretely, P t ( z ) .
Revised: Concretely, P t ( z ) .

Original: = µ t ( z ) / (cid:80) z (cid:48) µ t ( z (cid:48) ) with the preferences µ t ( z ) ≈ P
Revised: = µ t ( z ) / (cid:80) z (cid:48) µ t ( z (cid:48) ) with the preferences µ t ( z ) ≈ P

Original: ( f t ( z ) ≥ m t ) deﬁned as
Revised: ( f t ( z ) ≥ m t ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where n ( z, h ) is the number of times that z was chosen in the corresponding time window.
Revised: where n ( z, h ) is the number of times that z was chosen in the corresponding time window.

Original: We encode a prior preference of 1 2 in the absence of other evidence, as an additional (ﬁctitious) sample.
Revised: We encode a prior preference of 1 2 in the absence of other evidence, as an additional (ﬁctitious) sample.

Original: [Figure]
Revised: [Figure]

Original: Adaptive horizon: The choice of h can be tuned as a hyper-parameter, but in order to remove all hyper-parameters from the bandit, we adapt it online instead.
Revised: Adaptive horizon: The choice of h can be tuned as a hyper-parameter, but in order to remove all hyper-parameters from the bandit, we adapt it online instead.

Original: The update is based on a regression accuracy criterion, weighted by how often the arm is pulled.
Revised: The update is based on a regression accuracy criterion, weighted by how often the arm is pulled.

Original: For the full description, see Appendix A.
Revised: For the full description, see Appendix A.

Original: Factored structure: As we have seen in Section 2, our concrete modulations z have additional factored structure that can be exploited.
Revised: Factored structure: As we have seen in Section 2, our concrete modulations z have additional factored structure that can be exploited.

Original: For that we propose to use a separate sub-bandit (each deﬁned as above) for each dimension j of z .
Revised: For that we propose to use a separate sub-bandit (each deﬁned as above) for each dimension j of z .

Original: The full modulation z is assembled from the z j independently sampled from the sub-bandits.
Revised: The full modulation z is assembled from the z j independently sampled from the sub-bandits.

Original: This way, denoting by K j the number of arms for z j , the total number of arms to model is (cid:80) j K j , which is a signiﬁcant reduction from the number of arms in the single ﬂattened space (cid:81)
Revised: This way, denoting by K j the number of arms for z j , the total number of arms to model is (cid:80) j K j , which is a signiﬁcant reduction from the number of arms in the single ﬂattened space (cid:81)

Original: j K j .
Revised: j K j .

Original: This allows for dramatically faster adaptation in the bandit (see Figure 2).
Revised: This allows for dramatically faster adaptation in the bandit (see Figure 2).

Original: On the other hand, from the perspective of each sub-bandit, there is now another source of non-stationarity due to other sub-bandits shifting their distributions.
Revised: On the other hand, from the perspective of each sub-bandit, there is now another source of non-stationarity due to other sub-bandits shifting their distributions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The central claim of this paper is that the best ﬁxed hyper-parameters in hindsight for behaviour differ widely across tasks, and that an adaptive approach obtains similar performance to the best choice without costly per-task tuning.
Revised: The central claim of this paper is that the best ﬁxed hyper-parameters in hindsight for behaviour differ widely across tasks, and that an adaptive approach obtains similar performance to the best choice without costly per-task tuning.

Original: We report a broad collection of empirical results on Atari 2600 (Bellemare et al., 2013) that substantiate this claim, and validate the effectiveness of the proposed components.
Revised: We report a broad collection of empirical results on Atari 2600 (Bellemare et al., 2013) that substantiate this claim, and validate the effectiveness of the proposed components.

Original: From our results, we distill qualitative descriptions of the adaptation dynamics.
Revised: From our results, we distill qualitative descriptions of the adaptation dynamics.

Original: To isolate effects, independent experiments may use all or subsets of the dimensions of z .
Revised: To isolate effects, independent experiments may use all or subsets of the dimensions of z .

Original: Two initial experiments in a toy grid-world setting are reported in Figure 2.
Revised: Two initial experiments in a toy grid-world setting are reported in Figure 2.

Original: They demonstrate that the proposed bandit works well in both stationary and non-stationary settings.
Revised: They demonstrate that the proposed bandit works well in both stationary and non-stationary settings.

Original: Moreover, they highlight the beneﬁts of using the exact learning progress LP ( z ) , and the gap incurred when using less informative proxies f ( z ) .
Revised: Moreover, they highlight the beneﬁts of using the exact learning progress LP ( z ) , and the gap incurred when using less informative proxies f ( z ) .

Original: They also indicate that the factored approach can deliver a substantial speed-up.
Revised: They also indicate that the factored approach can deliver a substantial speed-up.

Original: Details of this setting are described in Appendix B.
Revised: Details of this setting are described in Appendix B.

Original: 5. E XPERIMENTAL S ETUP : A TARI
Revised: 5. E XPERIMENTAL S ETUP : A TARI

Original: Our Atari agent is a distributed system inspired by Impala (Espeholt et al., 2018) and Ape-X (Horgan et al., 2018), consisting of one learner (on GPU), multiple actors (on CPUs), and a bandit providing modulations to the actors.
Revised: Our Atari agent is a distributed system inspired by Impala (Espeholt et al., 2018) and Ape-X (Horgan et al., 2018), consisting of one learner (on GPU), multiple actors (on CPUs), and a bandit providing modulations to the actors.

Original: On each episode t , an actor queries the bandit for a modulation z t , and the learner for the latest network weights θ t .
Revised: On each episode t , an actor queries the bandit for a modulation z t , and the learner for the latest network weights θ t .

Original: At episode end, it reports a ﬁtness value f t ( z t ) to the bandit, and adds the collected experience to a replay table for the learner.
Revised: At episode end, it reports a ﬁtness value f t ( z t ) to the bandit, and adds the collected experience to a replay table for the learner.

Original: For stability and reliability, we enforce a ﬁxed ratio between experience generated and learning steps, making actors and learner run at the same pace.
Revised: For stability and reliability, we enforce a ﬁxed ratio between experience generated and learning steps, making actors and learner run at the same pace.

Original: Our agents learn a policy from 200 million environment frames in 10-12h wall-clock time (compared to a GPU-week for the state-of-art Rainbow agent (Hessel et al., 2018)).
Revised: Our agents learn a policy from 200 million environment frames in 10-12h wall-clock time (compared to a GPU-week for the state-of-art Rainbow agent (Hessel et al., 2018)).

Original: Besides distributed experience collection (i.e., improved experimental turnaround time), algorithmic elements of the learner are similar to Rainbow: the updates use multi-step double Q-learning, with distributional quantile regression (Dabney et al., 2018) and prioritized experience replay (Schaul et al., 2015).
Revised: Besides distributed experience collection (i.e., improved experimental turnaround time), algorithmic elements of the learner are similar to Rainbow: the updates use multi-step double Q-learning, with distributional quantile regression (Dabney et al., 2018) and prioritized experience replay (Schaul et al., 2015).

Original: All hyper-parameters (besides those determined by z ) are kept ﬁxed across all games and all experiments; these are listed in Appendix C alongside default values of z .
Revised: All hyper-parameters (besides those determined by z ) are kept ﬁxed across all games and all experiments; these are listed in Appendix C alongside default values of z .

Original: These allow us to generate competitive baseline results ( 118 ± 6% median human-normalised score) with a so-called reference setting (solid black in all learning curves) which sets the exploration parameters to that is most commonly used in the literature ( (cid:15)
Revised: These allow us to generate competitive baseline results ( 118 ± 6% median human-normalised score) with a so-called reference setting (solid black in all learning curves) which sets the exploration parameters to that is most commonly used in the literature ( (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 01 , ω = 0 , T = 0 , b = 0 , ρ = 0 ).
Revised: 01 , ω = 0 , T = 0 , b = 0 , ρ = 0 ).

Original: [Figure]
Revised: [Figure]

Original: If not mentioned otherwise, all aggregate results are across 15 games listed in Appendix D and at least N = 5 independent runs (seeds).
Revised: If not mentioned otherwise, all aggregate results are across 15 games listed in Appendix D and at least N = 5 independent runs (seeds).

Original: Learning curves shown are evaluations of the greedy policy after the agent has experienced the corresponding number of environment frames.
Revised: Learning curves shown are evaluations of the greedy policy after the agent has experienced the corresponding number of environment frames.

Original: To aggregate scores across these ﬁfteen games we use relative rank , an ordinal statistic that weighs each game equally (despite different score scales) and highlights relative differences between variants.
Revised: To aggregate scores across these ﬁfteen games we use relative rank , an ordinal statistic that weighs each game equally (despite different score scales) and highlights relative differences between variants.

Original: Concretely, the performance outcome G ( game, seed, variant ) is deﬁned as the average return of the greedy policy across the last 10% of the run (20 million frames).
Revised: Concretely, the performance outcome G ( game, seed, variant ) is deﬁned as the average return of the greedy policy across the last 10% of the run (20 million frames).

Original: All outcomes G ( game, · , · ) are then jointly ranked, and the corresponding ranks are averaged across seeds.
Revised: All outcomes G ( game, · , · ) are then jointly ranked, and the corresponding ranks are averaged across seeds.

Original: The averaged ranks are normalized to fall between 0 and 1, such that a normalized rank of 1 corresponds to all N seeds of a variant being ranked at the top N positions in the joint ranking.
Revised: The averaged ranks are normalized to fall between 0 and 1, such that a normalized rank of 1 corresponds to all N seeds of a variant being ranked at the top N positions in the joint ranking.

Original: Finally, the relative ranks for each variant are averaged across all games.
Revised: Finally, the relative ranks for each variant are averaged across all games.

Original: See also Appendix D.
Revised: See also Appendix D.

Original: 5. Q UANTIFYING THE T UNING C HALLENGES
Revised: 5. Q UANTIFYING THE T UNING C HALLENGES

Original: It is widely appreciated that the best hyper-parameters differ per Atari game.
Revised: It is widely appreciated that the best hyper-parameters differ per Atari game.

Original: Figure 3 illustrates this point for multiple classes of modulations (different arms come out on top in different games), while
Revised: Figure 3 illustrates this point for multiple classes of modulations (different arms come out on top in different games), while

Original: Figure 4 quantiﬁes this phenomenon across 15 games and 4 modulation classes and ﬁnds that this effect holds in general.
Revised: Figure 4 quantiﬁes this phenomenon across 15 games and 4 modulation classes and ﬁnds that this effect holds in general.

Original: If early performance were indicative of ﬁnal performance, the cost of tuning could be reduced.
Revised: If early performance were indicative of ﬁnal performance, the cost of tuning could be reduced.

Original: We quantify how much performance would be lost if the best ﬁxed arm were based on the ﬁrst 10% of the run.
Revised: We quantify how much performance would be lost if the best ﬁxed arm were based on the ﬁrst 10% of the run.

Original: Figure 5 shows that the mismatch is often substantial.
Revised: Figure 5 shows that the mismatch is often substantial.

Original: This also indicates the best choice is nonstationary : what is good in early learning may not be good later on — an issue sometimes addressed by hand-crafted schedules (e.g., DQN linearly decreases the value of (cid:15) (Mnih et al., 2015)).
Revised: This also indicates the best choice is nonstationary : what is good in early learning may not be good later on — an issue sometimes addressed by hand-crafted schedules (e.g., DQN linearly decreases the value of (cid:15) (Mnih et al., 2015)).

Original: Another approach is to choose not to choose, that is, feed experience from the full set of choices to the learner, an approach taken, e.g., in (Horgan et al., 2018).
Revised: Another approach is to choose not to choose, that is, feed experience from the full set of choices to the learner, an approach taken, e.g., in (Horgan et al., 2018).

Original: However, this merely shifts the problem, as it in turn necessitates tuning this set of choices.
Revised: However, this merely shifts the problem, as it in turn necessitates tuning this set of choices.

Original: Figure 6 shows that the difference between a naive and a carefully curated set can indeed be very large (Table 4 in Appendix C lists all these sets).
Revised: Figure 6 shows that the difference between a naive and a carefully curated set can indeed be very large (Table 4 in Appendix C lists all these sets).

Original: 5. A DAPTING INSTEAD OF T UNING
Revised: 5. A DAPTING INSTEAD OF T UNING

Original: It turns out that adapting the distribution over z as learning progresses effectively addresses the three tuning challenges discussed above (per-task differences, early-late mismatch, handling sets).
Revised: It turns out that adapting the distribution over z as learning progresses effectively addresses the three tuning challenges discussed above (per-task differences, early-late mismatch, handling sets).

Original: Figure 6 shows that the bandit can quickly suppress the choices of harmful elements in a noncurated set; in other words, the set does not need to be carefully tuned.
Revised: Figure 6 shows that the bandit can quickly suppress the choices of harmful elements in a noncurated set; in other words, the set does not need to be carefully tuned.

Original: At the same time, a game-
Revised: At the same time, a game-

Original: [Figure]
Revised: [Figure]

Original: tennis star_gunner space_invaders seaquest qbert private_eye ms_pacman hero frostbite demon_attack breakout asterix Performance drop when picking best fixed z early speciﬁc schedule emerges from the non-stationary adaptation, for example recovering an (cid:15) -schedule reminiscent of the hand-crafted one in DQN (Mnih et al., 2015) (see Figure 16 in Appendix E).
Revised: tennis star_gunner space_invaders seaquest qbert private_eye ms_pacman hero frostbite demon_attack breakout asterix Performance drop when picking best fixed z early speciﬁc schedule emerges from the non-stationary adaptation, for example recovering an (cid:15) -schedule reminiscent of the hand-crafted one in DQN (Mnih et al., 2015) (see Figure 17 in Appendix E).

Original: Finally, the overall performance of the bandit is similar to that of the best ﬁxed choice and not far from an “oracle” that picks the best ﬁxed z per game in hindsight (Figure 4).
Revised: Finally, the overall performance of the bandit is similar to that of the best ﬁxed choice and not far from an “oracle” that picks the best ﬁxed z per game in hindsight (Figure 4).

Original: A number of other interesting qualitative dynamics emerge in our setting (Appendix E): action biases are used initially and later suppressed (e.g., on S EAQUEST , Figure 18); the usefulness of action repeats varies across training (e.g., on H.E.R.O., Figure 17).
Revised: A number of other interesting qualitative dynamics emerge in our setting (Appendix E): action biases are used initially and later suppressed (e.g., on S EAQUEST , Figure 19); the usefulness of action repeats varies across training (e.g., on H.E.R.O., Figure 18).

Revised: Figure 16 looks at additional bandit baselines and ﬁnds that addressing the non-stationarity is critical (see Appendix E.3).

Original: Finally, our approach generalizes beyond a single class of modulations; all proposed dimensions can adapt simultaneously within a single run, using a factored bandit to handle the combinatorial space.
Revised: Finally, our approach generalizes beyond a single class of modulations; all proposed dimensions can adapt simultaneously within a single run, using a factored bandit to handle the combinatorial space.

Original: Figure 13 shows this yields similar performance to adapting within one class.
Revised: Figure 13 shows this yields similar performance to adapting within one class.

Original: In a few games this outperforms the best ﬁxed choice 1 in hindsight; see Figure 6 (‘combo’) and Figure 7; presumably because of the added dynamic adaptation to the learning process.
Revised: In a few games this outperforms the best ﬁxed choice 1 in hindsight; see Figure 6 (‘combo’) and Figure 7; presumably because of the added dynamic adaptation to the learning process.

Original: On the entire set of 57 Atari games, the bandit achieves similar performance ( 113 ± 2% median human-normalized score) to our ﬁxed, tuned reference setting ( 118 ± 6% ), despite operating on 60 different combinations of modulations.
Revised: On the entire set of 57 Atari games, the bandit achieves similar performance ( 113 ± 2% median human-normalized score) to our ﬁxed, tuned reference setting ( 118 ± 6% ), despite operating on 60 different combinations of modulations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Here we focus on two facets of our research: its relation to exploration, and hyper-parameter tuning.
Revised: Here we focus on two facets of our research: its relation to exploration, and hyper-parameter tuning.

Original: First, our work can be seen as building on a rich literature on exploration through intrinsic motivation aimed at maximising learning progress.
Revised: First, our work can be seen as building on a rich literature on exploration through intrinsic motivation aimed at maximising learning progress.

Original: As the true learning progress is not readily available during training, much of this work targets one of a number of proxies: empirical return (Jaderberg et al., 2017); change in parameters, policy, or value function (Itti & Baldi, 2006); magnitude of training loss (Mirolli & Baldassarre, 2013; Schmidhuber, 1991); error reduction or derivative (Schmidhuber, 1991; Oudeyer et al., 2007); expected accuracy improvement (Misra et al., 2018); compression progress (Schmidhuber, 2008); reduction in uncertainty; improvement of value accuracy; or change in distribution of encountered states.
Revised: As the true learning progress is not readily available during training, much of this work targets one of a number of proxies: empirical return (Jaderberg et al., 2017); change in parameters, policy, or value function (Itti & Baldi, 2006); magnitude of training loss (Mirolli & Baldassarre, 2013; Schmidhuber, 1991); error reduction or derivative (Schmidhuber, 1991; Oudeyer et al., 2007); expected accuracy improvement (Misra et al., 2018); compression progress (Schmidhuber, 2008); reduction in uncertainty; improvement of value accuracy; or change in distribution of encountered states.

Original: Some of these have the desirable property that if the proxy is zero, so is LP .
Revised: Some of these have the desirable property that if the proxy is zero, so is LP .

Original: However, these proxies themselves may only be available in approximated form, and these approximations tend to be highly dependent on the state distribution under which they are evaluated, which is subject to continual shift due to the changes in policy.
Revised: However, these proxies themselves may only be available in approximated form, and these approximations tend to be highly dependent on the state distribution under which they are evaluated, which is subject to continual shift due to the changes in policy.

Original: As a result, direct comparison between different learning algorithms under these proxies tends to be precarious.
Revised: As a result, direct comparison between different learning algorithms under these proxies tends to be precarious.

Original: Second, our adaptive behaviour modulation can be viewed as an alternative to per-task hyperparameter tuning, or hyper-parameter tuning with cross-task transfer (Golovin et al., 2017), and can be compared to other works attempting to reduce the need for this common practice.
Revised: Second, our adaptive behaviour modulation can be viewed as an alternative to per-task hyperparameter tuning, or hyper-parameter tuning with cross-task transfer (Golovin et al., 2017), and can be compared to other works attempting to reduce the need for this common practice.

Original: (Note that the best-ﬁxed-arm in our experiments is equivalent to explicitly tuning the modulations as hyperparameters.)
Revised: (Note that the best-ﬁxed-arm in our experiments is equivalent to explicitly tuning the modulations as hyperparameters.)

Original: Though often performed manually, hyper-parameter tuning can be improved by random search (Bergstra et al., 2011), but in either case requires many full training cycles, whereas our work optimises the modulations on-the-ﬂy during a single training run.
Revised: Though often performed manually, hyper-parameter tuning can be improved by random search (Bergstra et al., 2011), but in either case requires many full training cycles, whereas our work optimises the modulations on-the-ﬂy during a single training run.

Original: Like our method, Population Based Training (PBT, Jaderberg et al., 2017) and meta-gradient
Revised: Like our method, Population Based Training (PBT, Jaderberg et al., 2017) and meta-gradient

Original: RL (Andrychowicz et al., 2016; Xu et al., 2018) share the property of dynamically adapting hyperparameters throughout agent training.
Revised: RL (Andrychowicz et al., 2016; Xu et al., 2018) share the property of dynamically adapting hyperparameters throughout agent training.

Original: However, these methods exist in a distinctly different problem setting: PBT assumes the ability to run multiple independent learners in parallel with separate experience.
Revised: However, these methods exist in a distinctly different problem setting: PBT assumes the ability to run multiple independent learners in parallel with separate experience.

Original: Its cost grows linearly with the population size (typically > 10 ), but it can tune other hyper-parameters than our approach (such as learning rates).
Revised: Its cost grows linearly with the population size (typically > 10 ), but it can tune other hyper-parameters than our approach (such as learning rates).

Original: Meta-gradient RL, on the other hand, assumes that the ﬁtness is a differentiable function of the hyper-parameters, which may not generally hold for exploration hyper-parameters.
Revised: Meta-gradient RL, on the other hand, assumes that the ﬁtness is a differentiable function of the hyper-parameters, which may not generally hold for exploration hyper-parameters.

Original: While our method focuses on modulating behaviour in order to shape the experience stream for effective learning, a related but complementary approach is to ﬁlter or prioritize the generated experience when sampling from replay.
Revised: While our method focuses on modulating behaviour in order to shape the experience stream for effective learning, a related but complementary approach is to ﬁlter or prioritize the generated experience when sampling from replay.

Original: Classically, replay prioritization has been based on TD error, a simple proxy for the learning progress conferred by an experience sample (Schaul et al., 2015).
Revised: Classically, replay prioritization has been based on TD error, a simple proxy for the learning progress conferred by an experience sample (Schaul et al., 2015).

Original: More recently, however, learned and thereby more adaptive prioritization schemes have been proposed (Zha et al., 2019), with (approximate) learning progress as the objective function.
Revised: More recently, however, learned and thereby more adaptive prioritization schemes have been proposed (Zha et al., 2019), with (approximate) learning progress as the objective function.

Original:  D ISCUSSION & F UTURE W ORK
Revised:  D ISCUSSION & F UTURE W ORK

Original: Reiterating one of our key observations: the qualitative properties of experience generated by an agent impact its learning, in a way that depends on characteristics of the task, current learning parameters, and the design of the agent and its learning algorithm.
Revised: Reiterating one of our key observations: the qualitative properties of experience generated by an agent impact its learning, in a way that depends on characteristics of the task, current learning parameters, and the design of the agent and its learning algorithm.

Original: We have demonstrated that by adaptively using simple, direct modulations of the way an agent generates experience, we can improve the efﬁciency of learning by adapting to the dynamics of the learning process and the speciﬁc requirements of the task.
Revised: We have demonstrated that by adaptively using simple, direct modulations of the way an agent generates experience, we can improve the efﬁciency of learning by adapting to the dynamics of the learning process and the speciﬁc requirements of the task.

Original: Our proposed method 2 has the potential to accelerate RL research by reducing the burden of hyper-parameter tuning or the requirement for hand-designed strategies, and does so without incurring the computational overhead of some of the alternatives.
Revised: Our proposed method 2 has the potential to accelerate RL research by reducing the burden of hyper-parameter tuning or the requirement for hand-designed strategies, and does so without incurring the computational overhead of some of the alternatives.

Original: The work presented in this paper represents a ﬁrst stab at exploiting adaptive modulations to the dynamics of learning, and there are many natural ways of extending this work.
Revised: The work presented in this paper represents a ﬁrst stab at exploiting adaptive modulations to the dynamics of learning, and there are many natural ways of extending this work.

Original: For instance, such an approach need not be constrained to draw only from experiences generated by the agent; the agent can also leverage demonstrations provided by humans or by other agents.
Revised: For instance, such an approach need not be constrained to draw only from experiences generated by the agent; the agent can also leverage demonstrations provided by humans or by other agents.

Original: Having an adaptive system control the use of data relieves system designers of the need to curate such data to be of high quality – an adaptive system can learn to simply ignore data sources that are not useful (or which have outlived their usefulness), as our bandit has done in the case of choosing modulations to generate experiences with (e.g., Figures16, 17, 18).
Revised: Having an adaptive system control the use of data relieves system designers of the need to curate such data to be of high quality – an adaptive system can learn to simply ignore data sources that are not useful (or which have outlived their usefulness), as our bandit has done in the case of choosing modulations to generate experiences with (e.g., Figures 17, 18, 19).

Original: A potential limitation of our proposal is the assumption that a modulation remains ﬁxed for the duration of an episode.
Revised: A potential limitation of our proposal is the assumption that a modulation remains ﬁxed for the duration of an episode.

Original: This restriction could be lifted, and one can imagine scenarios in which the modulation used might depend on time or the underlying state.
Revised: This restriction could be lifted, and one can imagine scenarios in which the modulation used might depend on time or the underlying state.

Original: For example, an agent might generate more useful exploratory experiences by having low stochasticity in the initial part of an episode, but switching to have higher entropy once it reaches an unexplored region of state space.
Revised: For example, an agent might generate more useful exploratory experiences by having low stochasticity in the initial part of an episode, but switching to have higher entropy once it reaches an unexplored region of state space.

Original: There is also considerable scope to expand the set of modulations used.
Revised: There is also considerable scope to expand the set of modulations used.

Original: A particularly promising avenue might be to consider adding noise in parameter space, and controlling the variance (Fortunato et al., 2018;
Revised: A particularly promising avenue might be to consider adding noise in parameter space, and controlling the variance (Fortunato et al., 2018;

Original: Plappert et al., 2018).
Revised: Plappert et al., 2018).

Original: In addition, previous works have shown that agents can learn diverse behaviours conditioned on a latent policy embedding (Eysenbach et al., 2018;
Revised: In addition, previous works have shown that agents can learn diverse behaviours conditioned on a latent policy embedding (Eysenbach et al., 2018;

Original: Haarnoja et al., 2018), goal (Ghosh et al., 2018; Nair et al., 2018) or task speciﬁcation (Borsa et al., 2019).
Revised: Haarnoja et al., 2018), goal (Ghosh et al., 2018; Nair et al., 2018) or task speciﬁcation (Borsa et al., 2019).

Original: A bandit could potentially be exposed to modulating the choices in abstract task space, which could be a powerful driver for more directed exploration.
Revised: A bandit could potentially be exposed to modulating the choices in abstract task space, which could be a powerful driver for more directed exploration.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: (omitted for anonymity)
Revised: (omitted for anonymity)

Original: A A DAPTIVE B ANDIT
Revised: A A DAPTIVE B ANDIT

Original: In this section we will brieﬂy revisit the adaptive bandit proposed in Section 4 and provide more of the details behind the adaptability of its horizon.
Revised: In this section we will brieﬂy revisit the adaptive bandit proposed in Section 4 and provide more of the details behind the adaptability of its horizon.

Original: For clarity, we start by restating the setting and key quantities: the reference ﬁtness m t , the active horizon h t and the observed ﬁtness function f t : Z (cid:55)→ R , where
Revised: For clarity, we start by restating the setting and key quantities: the reference ﬁtness m t , the active horizon h t and the observed ﬁtness function f t : Z (cid:55)→ R , where

Original: Z = { z 1 , . .
Revised: Z = { z 1 , . .

Original: , z K } deﬁnes a modulation class.
Revised: , z K } deﬁnes a modulation class.

Original: The bandit samples a modulation z ∈ { z 1 , . .
Revised: The bandit samples a modulation z ∈ { z 1 , . .

Original: , z K } according to the probability that this z will result in higher than average ﬁtness (within a recent length-
Revised: , z K } according to the probability that this z will result in higher than average ﬁtness (within a recent length-

Original: h window):
Revised: h window):

Original: [Equation]
Revised: [Equation]

Original: Adapting z -probabilities.
Revised: Adapting z -probabilities.

Original: For simplicity, P t ( z ) is inferred based on the empirical data within a recent time window of the same horizon h that is used to compute m t .
Revised: For simplicity, P t ( z ) is inferred based on the empirical data within a recent time window of the same horizon h that is used to compute m t .

Original: Concretely, P t ( z ) .
Revised: Concretely, P t ( z ) .

Original: = µ t ( z ) / (cid:80) z (cid:48) µ t ( z (cid:48) ) with the preferences µ t ( z ) ≈ P
Revised: = µ t ( z ) / (cid:80) z (cid:48) µ t ( z (cid:48) ) with the preferences µ t ( z ) ≈ P

Original: ( f t ( z ) ≥ m t ) deﬁned as
Revised: ( f t ( z ) ≥ m t ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where n ( z, h ) is the number of times that z was chosen in the corresponding time window.
Revised: where n ( z, h ) is the number of times that z was chosen in the corresponding time window.

Original: We encode a prior preference of 12 in the absence of other evidence, as an additional (ﬁctitious) sample.
Revised: We encode a prior preference of 12 in the absence of other evidence, as an additional (ﬁctitious) sample.

Original: Adapting the horizon.
Revised: Adapting the horizon.

Original: As motivated in Section 4, the discrete horizon size h t = h t − 1 +1 is adapted in order to improve regression accuracy L t ( h ) :
Revised: As motivated in Section 4, the discrete horizon size h t = h t − 1 +1 is adapted in order to improve regression accuracy L t ( h ) :

Original: [Equation]
Revised: [Equation]

Original: where f ( z t ) is the ﬁtness of the modulation z t chosen at time t , and
Revised: where f ( z t ) is the ﬁtness of the modulation z t chosen at time t , and

Original: [Equation]
Revised: [Equation]

Original: This objective is not differentiable w.r.t.
Revised: This objective is not differentiable w.r.t.

Original: h , so we perform a ﬁnite-difference step.
Revised: h , so we perform a ﬁnite-difference step.

Original: As the horizon cannot grow beyond the amount of available data, the ﬁnite-difference is not symmetric around h t .
Revised: As the horizon cannot grow beyond the amount of available data, the ﬁnite-difference is not symmetric around h t .

Original: Concretely, at every step we evaluate two candidates: one according to the current horizon h t ,
Revised: Concretely, at every step we evaluate two candidates: one according to the current horizon h t ,

Original: L t ( h t ) , and one proposing a shrinkage in the effective horizon, L t ( h (cid:48) ) , where the new candidate horizon is given by:
Revised: L t ( h t ) , and one proposing a shrinkage in the effective horizon, L t ( h (cid:48) ) , where the new candidate horizon is given by:

Original: [Equation]
Revised: [Equation]

Original: Thus the new horizon proposes a shrinkage of up to η = 2% per step, but is never allowed to shrink beyond twice the number of arms K .
Revised: Thus the new horizon proposes a shrinkage of up to η = 2% per step, but is never allowed to shrink beyond twice the number of arms K .

Original: Given a current sample of the ﬁtness function f t ( z t ) , we probe which of these two candidates h t or h (cid:48) best explains it, by comparing L t ( h t ) and L t ( h (cid:48) ) .
Revised: Given a current sample of the ﬁtness function f t ( z t ) , we probe which of these two candidates h t or h (cid:48) best explains it, by comparing L t ( h t ) and L t ( h (cid:48) ) .

Original: If the shorter horizon seems to explain the new data point better, we interpret this as a sign of non-stationarity in the process and propose a shrinkage proportional to the relative error prediction reduction L t ( h t ) −L t ( h (cid:48) )
Revised: If the shorter horizon seems to explain the new data point better, we interpret this as a sign of non-stationarity in the process and propose a shrinkage proportional to the relative error prediction reduction L t ( h t ) −L t ( h (cid:48) )

Original: L t ( h t ) , namely:
Revised: L t ( h t ) , namely:

Original: [Equation]
Revised: [Equation]

Original: Factored bandits .
Revised: Factored bandits .

Original: In the case of factored sub-bandits, they each maintain their own independent horizon h ; we have not investigated whether sharing it would be beneﬁcial.
Revised: In the case of factored sub-bandits, they each maintain their own independent horizon h ; we have not investigated whether sharing it would be beneﬁcial.

Original: B L AVA W ORLD E XPERIMENTS
Revised: B L AVA W ORLD E XPERIMENTS

Original: In this section we will describe in further details the experiments behind Figure 2.
Revised: In this section we will describe in further details the experiments behind Figure 2.

Original: These were conducted on LavaWorld, a small (96 states), deterministic four-rooms-style navigation domain, with deadly lava instead of walls.
Revised: These were conducted on LavaWorld, a small (96 states), deterministic four-rooms-style navigation domain, with deadly lava instead of walls.

Original: We chose this domain to illustrate what our proposed adaptive mechanism (Section 4) would do under somewhat idealised conditions where the learning is tabular and we can compute the ground-truth LP ( z ) and assess oracle performance.
Revised: We chose this domain to illustrate what our proposed adaptive mechanism (Section 4) would do under somewhat idealised conditions where the learning is tabular and we can compute the ground-truth LP ( z ) and assess oracle performance.

Original: This investigation allows us to ﬁrst see how well the bandit can deal with the kind of non-stationary arising from an
Revised: This investigation allows us to ﬁrst see how well the bandit can deal with the kind of non-stationary arising from an

Original: RL learning process (entangled with exploration).
Revised: RL learning process (entangled with exploration).

Original: In this setting, we consider three modulation classes, (cid:15), T, b , where b can boost the logits for any of the 4 available actions.
Revised: In this setting, we consider three modulation classes, (cid:15), T, b , where b can boost the logits for any of the 4 available actions.

Original: The sets of modulations are (cid:15), T ∈ { 0 . 01 , 0 .
Revised: The sets of modulations are (cid:15), T ∈ { 0 . 01 , 0 .

Original: 1 , 1 } and b i ∈ { 0 , 0 . 1 } resulting in 31 unique modulated (stochastic) policies for each Q-function (see Figure 8).
Revised: 1 , 1 } and b i ∈ { 0 , 0 . 1 } resulting in 31 unique modulated (stochastic) policies for each Q-function (see Figure 8).

Original: Q-functions are look-up tables of size (96 × 4) as this domain contains 96 unique states.
Revised: Q-functions are look-up tables of size (96 × 4) as this domain contains 96 unique states.

Original: The single start state is in the top-left corner, and the single rewarding state is in the top-left corner of the top-right room, and is also absorbing.
Revised: The single start state is in the top-left corner, and the single rewarding state is in the top-left corner of the top-right room, and is also absorbing.

Original: We treat the discount γ = 0 .
Revised: We treat the discount γ = 0 .

Original: 99 as probability of continuation, and terminate episodes stochastically based on this, or when the agent hits lava.
Revised: 99 as probability of continuation, and terminate episodes stochastically based on this, or when the agent hits lava.

Original: We study the behaviour of the system in two settings: one stationary, one non-stationary.
Revised: We study the behaviour of the system in two settings: one stationary, one non-stationary.

Original: In the stationary setting (Figure 2, left), we considered modulation behaviours π ( ·| s, z ) that do not change over time.
Revised: In the stationary setting (Figure 2, left), we considered modulation behaviours π ( ·| s, z ) that do not change over time.

Original: They are computed based on a ’dummy’ action-value function Q , as described in
Revised: They are computed based on a ’dummy’ action-value function Q , as described in

Original: Section 2, but this value does not change over time.
Revised: Section 2, but this value does not change over time.

Original: The learning process is tabular and independent of this behaviour generating value Q .
Revised: The learning process is tabular and independent of this behaviour generating value Q .

Original: In this case, we compute the cumulative probability that an executed policy encounters the single sparse reward (‘expected reward’) as a function of the number of episodes, where we assume that the policy will be perfect after the ﬁrst reward event.
Revised: In this case, we compute the cumulative probability that an executed policy encounters the single sparse reward (‘expected reward’) as a function of the number of episodes, where we assume that the policy will be perfect after the ﬁrst reward event.

Original: The LP ( z ) signal given to the oracle bandit is the true (stationary) expectation of this event for every modulation z .
Revised: The LP ( z ) signal given to the oracle bandit is the true (stationary) expectation of this event for every modulation z .

Original: Given the extreme reward sparsity and the absence of learning, there is no obvious choice for a proxy measure f , so the non-oracle bandit reverts to a uniform choice over z .
Revised: Given the extreme reward sparsity and the absence of learning, there is no obvious choice for a proxy measure f , so the non-oracle bandit reverts to a uniform choice over z .

Original: The reference Q-values are the optimal ones for the task.
Revised: The reference Q-values are the optimal ones for the task.

Original: The results presented are averaged over 10 runs.
Revised: The results presented are averaged over 10 runs.

Original: Secondly, we considered a non-stationary setting (Figure 2, right) similar to the one above, but where the Q-values behind the modulated behaviour are learned over time.
Revised: Secondly, we considered a non-stationary setting (Figure 2, right) similar to the one above, but where the Q-values behind the modulated behaviour are learned over time.

Original: This is akin to the actual regime of operation this system would encounter in practice, although the learning of these values is idealised.
Revised: This is akin to the actual regime of operation this system would encounter in practice, although the learning of these values is idealised.

Original: These are initialised at zero, and the only update (given the absence of rewards) is to suppress the Q-value of any encountered transition that hits lava by 0 .
Revised: These are initialised at zero, and the only update (given the absence of rewards) is to suppress the Q-value of any encountered transition that hits lava by 0 .

Original: 1 ; this learning update happens instantaneously.
Revised: 1 ; this learning update happens instantaneously.

Original: In other words, the agent does a random walk, but over time it dies in many different ways.
Revised: In other words, the agent does a random walk, but over time it dies in many different ways.

Original: The reported “expected reward” is again the probability that the policy induced by
Revised: The reported “expected reward” is again the probability that the policy induced by

Original: Q t encounters the ﬁnal reward.
Revised: Q t encounters the ﬁnal reward.

Original: In this case, a reasonable proxy f ( z ) for LP ( z ) is the binary signal on whether something was learned from the last episode (by encountering a new into-lava transition), which obtains a large fraction of the oracle’s performance.
Revised: In this case, a reasonable proxy f ( z ) for LP ( z ) is the binary signal on whether something was learned from the last episode (by encountering a new into-lava transition), which obtains a large fraction of the oracle’s performance.

Original: C A TARI H YPER - PARAMETERS
Revised: C A TARI H YPER - PARAMETERS

Original: In this section we record the setting used in our Atari experiments: hyperparameters of the agents (Tables 2 and 3), environment and preprocessing (Table 1), and modulation sets for our behaviours (Table 4).
Revised: In this section we record the setting used in our Atari experiments: hyperparameters of the agents (Tables 2 and 3), environment and preprocessing (Table 1), and modulation sets for our behaviours (Table 4).

Original: Reference modulations Unless speciﬁed otherwise, we use the following modulations by default: (cid:15) = 0 .
Revised: Reference modulations Unless speciﬁed otherwise, we use the following modulations by default: (cid:15) = 0 .

Original: 01 , temperature T = 0 .
Revised: 01 , temperature T = 0 .

Original: 00001 (for tie-breaking between equal-valued actions), biases b = 0 , optimism ω
Revised: 00001 (for tie-breaking between equal-valued actions), biases b = 0 , optimism ω

Original: = 0 , repeat probability ρ = 0 .
Revised: = 0 , repeat probability ρ = 0 .

Original: This corresponds to the most commonly used settings in the literature.
Revised: This corresponds to the most commonly used settings in the literature.

Original: On learning curve plots, this ﬁxed setting is always shown in black.
Revised: On learning curve plots, this ﬁxed setting is always shown in black.

Original: Modulation sets The sets of curated and non-curated modulations we use are described in Table 4.
Revised: Modulation sets The sets of curated and non-curated modulations we use are described in Table 4.

Original: Curated values were chosen based on the results in Figure 4.
Revised: Curated values were chosen based on the results in Figure 4.

Original: Fixed hyperparameters The hyper-parameters used by our Atari agent are close to defaults used in the literature, with a few modiﬁcation to improve learning stability in our highly distributed setting.
Revised: Fixed hyperparameters The hyper-parameters used by our Atari agent are close to defaults used in the literature, with a few modiﬁcation to improve learning stability in our highly distributed setting.

Original: For preprocessing and agent architecture, we use DQN settings detailed in Table 1 and
Revised: For preprocessing and agent architecture, we use DQN settings detailed in Table 1 and

Original: Table 2.
Revised: Table 2.

Original: Table 3 summarizes the other hyper-parameters used by our Atari agent.
Revised: Table 3 summarizes the other hyper-parameters used by our Atari agent.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D E VALUATION
Revised: D E VALUATION

Original: In this section we detail the evaluation settings used for our Atari experiments, as well as the metrics used to aggregate results across games in Figure 5 in the main text.
Revised: In this section we detail the evaluation settings used for our Atari experiments, as well as the metrics used to aggregate results across games in Figure 5 in the main text.

Original: Games We evaluate on a set of 15 games chosen for their different learning characteristics:
Revised: Games We evaluate on a set of 15 games chosen for their different learning characteristics:

Original: A STERIX , B REAKOUT , D EMON A TTACK , F ROSTBITE , H.E.R.O., M S .
Revised: A STERIX , B REAKOUT , D EMON A TTACK , F ROSTBITE , H.E.R.O., M S .

Original: P AC -M
Revised: P AC -M

Original: AN , P RIVATE E YE ,
Revised: AN , P RIVATE E YE ,

Original: Q* BERT , S EAQUEST , S PACE I NVADERS , S TAR G UNNER , T ENNIS , V ENTURE , Y ARS ’ R EVENGE and Z AXXON .
Revised: Q* BERT , S EAQUEST , S PACE I NVADERS , S TAR G UNNER , T ENNIS , V ENTURE , Y ARS ’ R EVENGE and Z AXXON .

Original: Human-normalised scores are computed following the procedure in (Mnih et al., 2015), but differing in that we evaluate online (without interrupting training), average over policies with different weights θ t (not freezing them), and aggregate over 20 million frames per point instead of 1 million.
Revised: Human-normalised scores are computed following the procedure in (Mnih et al., 2015), but differing in that we evaluate online (without interrupting training), average over policies with different weights θ t (not freezing them), and aggregate over 20 million frames per point instead of 1 million.

Original: Metrics The relative rank statistic in Section 5.1 are normalized to fall between 0 and 1 for any set of outcomes G ( game, · , variant ) for any number of seeds N .
Revised: Metrics The relative rank statistic in Section 5.1 are normalized to fall between 0 and 1 for any set of outcomes G ( game, · , variant ) for any number of seeds N .

Original: For this we simply scale the raw average ranks by their minimal and maximal values: N + and N + + N + where N + is the number of other outcomes this variant is jointly ranked with.
Revised: For this we simply scale the raw average ranks by their minimal and maximal values: N + and N + + N + where N + is the number of other outcomes this variant is jointly ranked with.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: In Figure
Revised: In Figure

Original: [Table]
Revised: [Table]

Original: we use a different metric to compare the effect of a chosen modulation early into the learning process.
Revised: we use a different metric to compare the effect of a chosen modulation early into the learning process.

Original: For this we compute E s [ G ( s ) z ] , the average episode return of modulation z at the end of training across all seeds s ∈ S , and z 0 , the modulation with highest average episode returns at the beginning of training (ﬁrst 10% of the run).
Revised: For this we compute E s [ G ( s ) z ] , the average episode return of modulation z at the end of training across all seeds s ∈ S , and z 0 , the modulation with highest average episode returns at the beginning of training (ﬁrst 10% of the run).

Original: Based on this, we compute the normalised drop in performance resulting from committing prematurely to z 0 :
Revised: Based on this, we compute the normalised drop in performance resulting from committing prematurely to z 0 :

Original: Performance drop ( z ) .
Revised: Performance drop ( z ) .

Original: =
Revised: =

Original: E s
Revised: E s

Original: [ G ( s ) z 0 ] − E s [
Revised: [ G ( s ) z 0 ] − E s [

Original: [Equation]
Revised: [Equation]

Original: where z + = arg max z ∈Z E s [ G ( s ) z ] , z − = arg min
Revised: where z + = arg max z ∈Z E s [ G ( s ) z ] , z − = arg min

Original: z ∈Z E s [ G ( s ) z ] and Z the modulation class considered in the study ( (cid:15) ’s, temperatures T , action repeat probabilities ρ , and optimism ω ).
Revised: z ∈Z E s [ G ( s ) z ] and Z the modulation class considered in the study ( (cid:15) ’s, temperatures T , action repeat probabilities ρ , and optimism ω ).

Original: E A DDITIONAL A TARI R ESULTS E. P ER -T ASK N ON - STATIONARITY
Revised: E A DDITIONAL A TARI R ESULTS E. P ER -T ASK N ON - STATIONARITY

Original: In this section we report detailed results which were presented in aggregate in Figure 4.
Revised: In this section we report detailed results which were presented in aggregate in Figure 4.

Original: Speciﬁcally, these results show that (1) the most effective set of modulations varies by game, (2) different modulations are preferable on different games, and (3) the non-stationary bandit performs comparably with the best choice of modulation class on all games.
Revised: Speciﬁcally, these results show that (1) the most effective set of modulations varies by game, (2) different modulations are preferable on different games, and (3) the non-stationary bandit performs comparably with the best choice of modulation class on all games.

Original: In the next few ﬁgures we give per-game performance comparing ﬁxed-arm modulations with the adaptive bandit behaviour for modulation epsilon (Figure 9), temperature (Figure 10), action repeats (Figure 11), and optimism (Figure 12).
Revised: In the next few ﬁgures we give per-game performance comparing ﬁxed-arm modulations with the adaptive bandit behaviour for modulation epsilon (Figure 9), temperature (Figure 10), action repeats (Figure 11), and optimism (Figure 12).

Original: For reference, we include the performance of a uniform bandit (dashed-line in the ﬁgures), over the same modulation set as the bandits, as well as the best parameter setting across games ( reference solid black line in the ﬁgures).
Revised: For reference, we include the performance of a uniform bandit (dashed-line in the ﬁgures), over the same modulation set as the bandits, as well as the best parameter setting across games ( reference solid black line in the ﬁgures).

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: E. C OMBINATORIAL B ANDITS
Revised: E. C OMBINATORIAL B ANDITS

Original: In this section we include additional results for the different combinatorial bandits run on the curated and extended sets (see Table 4).
Revised: In this section we include additional results for the different combinatorial bandits run on the curated and extended sets (see Table 4).

Original: Most of these experiments were run on subsets of the curated/extended sets across modulations, rather than the full Cartesian product.
Revised: Most of these experiments were run on subsets of the curated/extended sets across modulations, rather than the full Cartesian product.

Original: As a convention, whenever a modulation class is omitted from the experiment name, the value for this class is set to the default reference value reported in Section C
Revised: As a convention, whenever a modulation class is omitted from the experiment name, the value for this class is set to the default reference value reported in Section C

Original: ( Reference modulations ).
Revised: ( Reference modulations ).

Original: Thus, for instance if we refer to a per-class-modulation bandit, say optimism ω , the modulations z for this class would
Revised: Thus, for instance if we refer to a per-class-modulation bandit, say optimism ω , the modulations z for this class would

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: be the ones reported in Table 4 (line 4)
Revised: be the ones reported in Table 4 (line 4)

Original: [Figure]
Revised: [Figure]

Original: while all other modulation dimensions would be kept ﬁxed to their reference values.
Revised: while all other modulation dimensions would be kept ﬁxed to their reference values.

Original: Figure 13 shows that the combined bandit performs competitively compared with per-factor bandits (the same adaptive bandit but restricted to one class of modulation).
Revised: Figure 13 shows that the combined bandit performs competitively compared with per-factor bandits (the same adaptive bandit but restricted to one class of modulation).

Original: In particular, it is worth noting that the per-factor bandit that performs best is game dependent.
Revised: In particular, it is worth noting that the per-factor bandit that performs best is game dependent.

Original: Nevertheless, the combined bandit, considering modulations across many of these dimensions, manages to recover a competitive performance across most games
Revised: Nevertheless, the combined bandit, considering modulations across many of these dimensions, manages to recover a competitive performance across most games

Original: [Figure]
Revised: [Figure]

Original: In Figure 14, we include a comparison plot between the combinatorial bandit on the curated set of 3 modulation classes ( (cid:15), ρ, ω ) , its uniform counterpart on the same set and the reference ﬁxed arm across games.
Revised: In Figure 14, we include a comparison plot between the combinatorial bandit on the curated set of 3 modulation classes ( (cid:15), ρ, ω ) , its uniform counterpart on the same set and the reference ﬁxed arm across games.

Original: First thing to notice is that on the curated set the uniform bandit is quite competitive, validating our initial observation that the problem of tuning can be shifted a level above, by carefully curating a set of good candidates.
Revised: First thing to notice is that on the curated set the uniform bandit is quite competitive, validating our initial observation that the problem of tuning can be shifted a level above, by carefully curating a set of good candidates.

Original: We can additionally see that the adaptive mechanism tends to fall in-between these two extremes: an uninformed arm selection and tuned arm selection.
Revised: We can additionally see that the adaptive mechanism tends to fall in-between these two extremes: an uninformed arm selection and tuned arm selection.

Original: We can see that the adaptive mechanism can recover a behaviour close to uniform in some games (H.E.R.O.,
Revised: We can see that the adaptive mechanism can recover a behaviour close to uniform in some games (H.E.R.O.,

Original: Y ARS ’ R EVENGE ), while maintaining the ability to recover something akin to best arm identiﬁcation in other games (see A STERIX ).
Revised: Y ARS ’ R EVENGE ), while maintaining the ability to recover something akin to best arm identiﬁcation in other games (see A STERIX ).

Original: Moreover there are (rare) instances, see Z AXXON , where the bandit outperforms both of these extremes.
Revised: Moreover there are (rare) instances, see Z AXXON , where the bandit outperforms both of these extremes.

Original: [Figure]
Revised: [Figure]

Original: Finally, in Figure 15 we include a plot comparing the performance of a combinatorial bandit on the full curated and extended modulation sets.
Revised: In Figure 15 we include a plot comparing the performance of a combinatorial bandit on the full curated and extended modulation sets.

Original: These are bandits acting on across all modulation classes outlined in Table 4.
Revised: These are bandits acting on across all modulation classes outlined in Table 4.

Original: As a reference, we include the performance of the per-class modulation bandits, as in Figure 13.
Revised: As a reference, we include the performance of the per-class modulation bandits, as in Figure 13.

Original: The bias modulation class was omitted as modulating exclusively within this class leads to very poor performance as policies tend to lock into a particular preference.
Revised: The bias modulation class was omitted as modulating exclusively within this class leads to very poor performance as policies tend to lock into a particular preference.

Original: We can also see a negative impact on the overall performance when adding a bias set to the set of modulations the bandit operates on, as one can see from Figure 15 (magenta line).
Revised: We can also see a negative impact on the overall performance when adding a bias set to the set of modulations the bandit operates on, as one can see from Figure 15 (magenta line).

Original: This is why we opted not to include this set in the extended bandit experiments reported in Figure 6 and restricted ourselves to the other 4 extended modulation sets
Revised: This is why we opted not to include this set in the extended bandit experiments reported in Figure 6 and restricted ourselves to the other 4 extended modulation sets

Original: [Figure]
Revised: [Figure]

Revised: E. O THER B ANDIT B ASELINES

Revised: Finally, in Figure 16 we provide a comparison to other more established bandit algorithms,

Revised: UCB (Auer, 2002; Kaufmann et al., 2012) and Thompson Sampling (Thompson, 1933; Chapelle & Li, 2011), that would need to learn with modulation to use.

Revised: The results in this ﬁgure are averaged across 5 seeds, and jointly modulate across three classes ( (cid:15), ρ, ω ).

Revised: We also include resulting learning curves for our proposed adaptation mechanism, the bandit described in Section 4, as well as uniform.

Revised: The ﬁrst thing to notice is that the stationary bandits, UCB and Thompson Sampling, are sometimes signiﬁcantly worse than uniform, indicating that they prematurely lock into using modulations that may be good initially, but don’t help for long-term performance.

Revised: We have already seen signs of this non-stationarity in the analysis in Figure 5 which shows that early commitment based on the evidence seen in the ﬁrst part of training can be premature and might hinder the overall performance.

Revised: In contrast, our proposed bandit can adapt to the non-stationarity present in the learning process, resulting in performance that is on par or better than these baselines in most of these games (with only one exception, in Y ARS ’ R EVENGE , where it matches the performance of the uniform bandit).

Revised: In that context, it is worth highlighting that the best alternative baseline (UCB, Thompson Sampling,

Revised: Uniform) differs from game to game, so outperforming all of them is a signiﬁcant result.

Revised: Another point is that UCB and Thompson Sampling still require some hyper-parameter tuning (we report the results for the best setting we found), and thus add extra tuning complexity, while our approach is hyper-parameter-free.

Revised: [Figure]

Revised: (blue) varies considerably from one game to another, while our proposed adaptive mechanism (‘bandit’, red) seems to deal quite effectively with this variability and achieves very good performance across almost all games.

Revised: Note that the bandit and uniform results here correspond to the extended ‘combo’ results in Figure 4.

Original: E. B EHAVIOUR OF THE N ON - STATIONARY B ANDIT
Revised: E. B EHAVIOUR OF THE N ON - STATIONARY B ANDIT

Original: In the previous section we saw that the bandit is able to effectively modulate the behaviour policy to give robust performance across games.
Revised: In the previous section we saw that the bandit is able to effectively modulate the behaviour policy to give robust performance across games.

Original: In this section we dive in deeper to analyse the precise modulations applied by the bandit over time and per-game.
Revised: In this section we dive in deeper to analyse the precise modulations applied by the bandit over time and per-game.

Original: We see that the bandit does indeed adaptively change the modulation over time and in a game-speciﬁc way.
Revised: We see that the bandit does indeed adaptively change the modulation over time and in a game-speciﬁc way.

Original: With these results we aim to look into the choices of the bandit across the learning process and in multiple games.
Revised: With these results we aim to look into the choices of the bandit across the learning process, and in multiple games.

Original: Most of the effect seems to be present in early stages of training.
Revised: Most of the effect seems to be present in early stages of training.

Original: • Epsilon schedules: Figure 16 shows the evolution of the value of (cid:15) by the bandit over time.
Revised: • Epsilon schedules: Figure 17 shows the evolution of the value of (cid:15) by the bandit over time.

Original: We see that this gives rise to an adaptive type of epsilon schedule, where early in training (usually the ﬁrst few million frames) large values are preferred, and as training progresses smaller values are preferred.
Revised: We see that this gives rise to an adaptive type of epsilon schedule, where early in training (usually the ﬁrst few million frames) large values are preferred, and as training progresses smaller values are preferred.

Original: This leads to a gradually increasingly greedy behaviour policy.
Revised: This leads to a gradually increasingly greedy behaviour policy.

Original: • Action repeats decay: Figure 17 shows the bandit modulated values for action repeats.
Revised: • Action repeats decay: Figure 18 shows the bandit modulated values for action repeats.

Original: We observe that as the agent progresses it can beneﬁt from more resolution in the policy.
Revised: We observe that as the agent progresses it can beneﬁt from more resolution in the policy.

Original: Thus, the agent adaptively moves to increasingly prefer low action repeat probability over time, with a prolonged period of non-preference early in training.
Revised: Thus, the agent adaptively moves to increasingly prefer low action repeat probability over time, with a prolonged period of non-preference early in training.

Original: • S EAQUEST :
Revised: • S EAQUEST :

Original: Figure 18 shows the evolution of the sampling distributions of a combined bandit with access to all modulation dimensions: T , (cid:15) , b , ρ , and ω .
Revised: Figure 19 shows the evolution of the sampling distributions of a combined bandit with access to all modulation dimensions: T , (cid:15) , b , ρ , and ω .

Original: Despite having over 7.5 million combinations of modulation values, the bandit can efﬁciently learn the quality of different arms.
Revised: Despite having over 7.5 million combinations of modulation values, the bandit can efﬁciently learn the quality of different arms.

Original: For instance, the agent quickly learns to prefer the down action over the up action, to avoid extreme left/right biases, and to avoid suppressing the ﬁre action, which is consistent with our intuition (in S EAQUEST , the player must move below the sea level to receive points, avoid the left and right boundaries, and ﬁre at incoming enemies).
Revised: For instance, the agent quickly learns to prefer the down action over the up action, to avoid extreme left/right biases, and to avoid suppressing the ﬁre action, which is consistent with our intuition (in S EAQUEST , the player must move below the sea level to receive points, avoid the left and right boundaries, and ﬁre at incoming enemies).

Original: Moreover, as in the case of single arm bandits discussed above, the combined bandit prefers more stochastic choices of (cid:15) and temperature in the beginning of training and more deterministic settings later in training, and the action repeat probability decays over time.
Revised: Moreover, as in the case of single arm bandits discussed above, the combined bandit prefers more stochastic choices of (cid:15) and temperature in the beginning of training and more deterministic settings later in training, and the action repeat probability decays over time.

Original: by the bandit across time (log-scale) on the extended set of (cid:15) .

Original: Note that in general the values of (cid:15) start high and gradually diminish, but the emerging schedules are quite different across games.