Original: W
Revised: W

Original: HEN SHOULD AGENTS EXPLORE ?
Revised: HEN SHOULD AGENTS EXPLORE ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Exploration remains a central challenge for reinforcement learning (RL).
Revised: Exploration remains a central challenge for reinforcement learning (RL).

Original: Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best).
Revised: Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best).

Original: In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of switching between modes.
Revised: In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of switching between modes.

Original: This paper presents an initial study of mode-switching, nonmonolithic exploration for RL.
Revised: This paper presents an initial study of mode-switching, nonmonolithic exploration for RL.

Original: We investigate different modes to switch between, at what timescales it makes sense to switch and what signals make for good switching triggers.
Revised: We investigate different modes to switch between, at what timescales it makes sense to switch and what signals make for good switching triggers.

Original: We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables ﬂexibility without an accompanying hyper-parameter-tuning burden.
Revised: We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables ﬂexibility without an accompanying hyper-parameter-tuning burden.

Original: Finally, we report a promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales
Revised: Finally, we report a promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The trade-off between exploration and exploitation is described as the crux of learning and behaviour across many domains, not just reinforcement learning (Sutton & Barto, 2018), but also in decision making (Cohen et al., 2007), evolutionary biology (Cremer et al., 2019), ecology (Kembro et al., 2019), neuroscience (e.g., focused versus diffuse search in visual attention (Wolfe et al., 1989), dopamine regulations (Chakroun et al., 2020)), cognitive sciences (Hills et al., 2015), as well as psychology and psychiatry (Addicott et al., 2017).
Revised: The trade-off between exploration and exploitation is described as the crux of learning and behaviour across many domains, not just reinforcement learning (Sutton & Barto, 2018), but also in decision making (Cohen et al., 2007), evolutionary biology (Cremer et al., 2019), ecology (Kembro et al., 2019), neuroscience (e.g., focused versus diffuse search in visual attention (Wolfe et al., 1989), dopamine regulations (Chakroun et al., 2020)), cognitive sciences (Hills et al., 2015), as well as psychology and psychiatry (Addicott et al., 2017).

Original: In a nutshell, exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward, but which could ultimately be more valuable than the familiar options.
Revised: In a nutshell, exploration is about the balance between taking the familiar choice that is known to be rewarding and learning about unfamiliar options of uncertain reward, but which could ultimately be more valuable than the familiar options.

Original: Ample literature has studied the question of how much to explore, that is how to set the overall trade-off (and how to adjust it over the course of learning) (Jaksch et al., 2010; Capp ´ e et al., 2013;
Revised: Ample literature has studied the question of how much to explore, that is how to set the overall trade-off (and how to adjust it over the course of learning) (Jaksch et al., 2010; Capp ´ e et al., 2013;

Original: Lattimore & Szepesv ´ ari, 2020; Thrun, 1992), and the question of how to explore, namely how to choose exploratory actions (e.g., randomly, optimistically, intrinsically motivated, or otherwise)
Revised: Lattimore & Szepesv ´ ari, 2020; Thrun, 1992), and the question of how to explore, namely how to choose exploratory actions (e.g., randomly, optimistically, intrinsically motivated, or otherwise)

Original: (Schmidhuber, 1991; Oudeyer & Kaplan, 2009; Linke et al., 2019).
Revised: (Schmidhuber, 1991; Oudeyer & Kaplan, 2009; Linke et al., 2019).

Original: In contrast, the question of when to explore has been studied very little, possibly because it does not arise in bandit problems, where a lot of exploration methods are rooted.
Revised: In contrast, the question of when to explore has been studied very little, possibly because it does not arise in bandit problems, where a lot of exploration methods are rooted.

Original: The ‘when’ question and its multiple facets are the subjects of this paper.
Revised: The ‘when’ question and its multiple facets are the subjects of this paper.

Original: We believe that addressing it could lead to more intentional forms of exploration.
Revised: We believe that addressing it could lead to more intentional forms of exploration.

Original: Consider an agent that has access to two modes of behaviour, an ‘explore’ mode and an ‘exploit’ mode (e.g., a random policy and a greedy policy, as in ε -greedy).
Revised: Consider an agent that has access to two modes of behaviour, an ‘explore’ mode and an ‘exploit’ mode (e.g., a random policy and a greedy policy, as in ε -greedy).

Original: Even when assuming that the overall proportion of exploratory steps is ﬁxed, the agent still has multiple degrees of freedom: it can explore more at the beginning of training and less in later phases; it may take single exploratory steps or execute prolonged periods of exploration; it may prefer exploratory steps early or late within an episode; and it could trigger the onset (or end) of an exploratory period based on various criteria.
Revised: Even when assuming that the overall proportion of exploratory steps is ﬁxed, the agent still has multiple degrees of freedom: it can explore more at the beginning of training and less in later phases; it may take single exploratory steps or execute prolonged periods of exploration; it may prefer exploratory steps early or late within an episode; and it could trigger the onset (or end) of an exploratory period based on various criteria.

Original: Animals and humans exhibit non-trivial behaviour in all of these dimensions, presumably encoding useful inductive biases that way (Power, 1999).
Revised: Animals and humans exhibit non-trivial behaviour in all of these dimensions, presumably encoding useful inductive biases that way (Power, 1999).

Original: Humans make use of multiple effective strategies, such as selectively exploring options with high uncertainty (a form of directed, or informationseeking exploration), and increasing the randomness of their choices when they are more uncertain (Gershman, 2018; Gershman & Tzovaras, 2018; Ebitz et al., 2019).
Revised: Humans make use of multiple effective strategies, such as selectively exploring options with high uncertainty (a form of directed, or informationseeking exploration), and increasing the randomness of their choices when they are more uncertain (Gershman, 2018; Gershman & Tzovaras, 2018; Ebitz et al., 2019).

Original: Monkeys use directed exploration to manage explore-exploit trade-offs, and these signals are coded in motivational brain regions (Costa et al., 2019).
Revised: Monkeys use directed exploration to manage explore-exploit trade-offs, and these signals are coded in motivational brain regions (Costa et al., 2019).

Original: Patients with schizophrenia register changes in directed exploration and experience low-grade inﬂammation when shifting from exploitation to random exploration (Waltz et al., 2020; Cathomas et al., 2021).
Revised: Patients with schizophrenia register changes in directed exploration and experience low-grade inﬂammation when shifting from exploitation to random exploration (Waltz et al., 2020; Cathomas et al., 2021).

Original: This diversity is what motivates us to study which of these can beneﬁt
Revised: This diversity is what motivates us to study which of these can beneﬁt

Original: RL agents in turn, by expanding the class of exploratory behaviours beyond the commonly used monolithic ones (where modes are merged homogeneously in time).
Revised: RL agents in turn, by expanding the class of exploratory behaviours beyond the commonly used monolithic ones (where modes are merged homogeneously in time).

Original:  M ETHODS
Revised:  M ETHODS

Original: The objective of an RL agent is to learn a policy that maximises external reward.
Revised: The objective of an RL agent is to learn a policy that maximises external reward.

Original: At the high level, it achieves this by interleaving two processes: generating new experience by interacting with the environment using a behaviour policy (exploration) and updating its policy using this experience (learning).
Revised: At the high level, it achieves this by interleaving two processes: generating new experience by interacting with the environment using a behaviour policy (exploration) and updating its policy using this experience (learning).

Original: As RL is applied to increasingly ambitious tasks, the challenge for exploration becomes to keep producing diverse experience, because if something has not been encountered, it cannot be learned.
Revised: As RL is applied to increasingly ambitious tasks, the challenge for exploration becomes to keep producing diverse experience, because if something has not been encountered, it cannot be learned.

Original: Our central argument is therefore simple: a monolithic, time-homogeneous behaviour policy is strictly less diverse than a heterogeneous mode-switching one, and the former may hamstring the agent’s performance.
Revised: Our central argument is therefore simple: a monolithic, time-homogeneous behaviour policy is strictly less diverse than a heterogeneous mode-switching one, and the former may hamstring the agent’s performance.

Original: As an illustrative example, consider a human learning how to ride a bike (explore), while maintaining their usual happiness through food, sleep, work (exploit): there is a stark contrast between a monolithic, time-homogeneous behaviour that interleaves a twist of the handlebar or a turn of a pedal once every few minutes or so, and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling.
Revised: As an illustrative example, consider a human learning how to ride a bike (explore), while maintaining their usual happiness through food, sleep, work (exploit): there is a stark contrast between a monolithic, time-homogeneous behaviour that interleaves a twist of the handlebar or a turn of a pedal once every few minutes or so, and the mode-switching behaviour that dedicates prolonged periods of time exclusively to acquiring the new skill of cycling.

Original: While the choice of behaviour in pure exploit mode is straightforward, namely the greedy pursuit of external reward (or best guess thereof), denoted by G , there are numerous viable choices for behaviour in a pure explore mode (denoted by X ).
Revised: While the choice of behaviour in pure exploit mode is straightforward, namely the greedy pursuit of external reward (or best guess thereof), denoted by G , there are numerous viable choices for behaviour in a pure explore mode (denoted by X ).

Original: In this paper we consider two standard ones:
Revised: In this paper we consider two standard ones:

Original: X U , the naive uniform random policy, and X I , an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation (RND, (Burda et al., 2018)).
Revised: X U , the naive uniform random policy, and X I , an intrinsically motivated behaviour that exclusively pursues a novelty measure based on random network distillation (RND, (Burda et al., 2018)).

Original: See
Revised: See

Original: Section 4 and Appendix B for additional possibilities of X .
Revised: Section 4 and Appendix B for additional possibilities of X .

Original: In this paper we choose ﬁxed behaviours for these modes, and focus solely on the question of when to switch between them.
Revised: In this paper we choose ﬁxed behaviours for these modes, and focus solely on the question of when to switch between them.

Original: In our setting, overall proportion of exploratory steps (the how much ), denoted by p X , is not directly controlled but derives from the when .
Revised: In our setting, overall proportion of exploratory steps (the how much ), denoted by p X , is not directly controlled but derives from the when .

Original: 2. G RANULARITY
Revised: 2. G RANULARITY

Original: An exploration period is an uninterrupted sequence of steps in explore mode.
Revised: An exploration period is an uninterrupted sequence of steps in explore mode.

Original: We consider four choices of temporal granularity for exploratory periods, also illustrated in Figure 1.
Revised: We consider four choices of temporal granularity for exploratory periods, also illustrated in Figure 1.

Original: Step-level exploration is the most common scenario, where the decision to explore is taken independently at each step, affecting one action.
Revised: Step-level exploration is the most common scenario, where the decision to explore is taken independently at each step, affecting one action.

Original: The canonical example is ε -greedy
Revised: The canonical example is ε -greedy

Original: (Fig.1:C).
Revised: (Fig.1:C).

Original: Experiment-level exploration is the other extreme, where all behaviour during training is produced in explore mode, and learning is off-policy (the greedy policy is only used for evaluation).
Revised: Experiment-level exploration is the other extreme, where all behaviour during training is produced in explore mode, and learning is off-policy (the greedy policy is only used for evaluation).

Original: This scenario is also very common, with most forms of intrinsic motivation falling into this category, namely pursuing reward with an intrinsic bonus throughout training (Fig.1:A).
Revised: This scenario is also very common, with most forms of intrinsic motivation falling into this category, namely pursuing reward with an intrinsic bonus throughout training (Fig.1:A).

Original: Episode-level exploration is the case where the mode is ﬁxed for an entire episode at a time (e.g., training games versus tournament matches in a sport), see Fig.1:
Revised: Episode-level exploration is the case where the mode is ﬁxed for an entire episode at a time (e.g., training games versus tournament matches in a sport), see Fig.1:

Original: B. This has been investigated for simple cases, where the policy’s level of stochasticity is sampled at the beginning of each episode (Horgan et al., 2018; Kapturowski et al., 2019; Zha et al., 2021).
Revised: B. This has been investigated for simple cases, where the policy’s level of stochasticity is sampled at the beginning of each episode (Horgan et al., 2018; Kapturowski et al., 2019; Zha et al., 2021).

Original: Intra-episodic exploration is what falls in-between step- and episode-level exploration, where exploration periods last for multiple steps, but less than a full episode.
Revised: Intra-episodic exploration is what falls in-between step- and episode-level exploration, where exploration periods last for multiple steps, but less than a full episode.

Original: This is the least commonly studied scenario, and will form the bulk of our investigations (Fig.1:D,E,F,G).
Revised: This is the least commonly studied scenario, and will form the bulk of our investigations (Fig.1:D,E,F,G).

Original: We denote the length of an exploratory period by n X (and similarly n G for exploit mode).
Revised: We denote the length of an exploratory period by n X (and similarly n G for exploit mode).

Original: To characterise granularity, our summary statistic of choice is med X := median( n X ) .
Revised: To characterise granularity, our summary statistic of choice is med X := median( n X ) .

Original: Note that there are two possible units for these statistics: the raw steps or the proportion of the episode length L .
Revised: Note that there are two possible units for these statistics: the raw steps or the proportion of the episode length L .

Original: The latter has different (relative) semantics, but may be more appropriate when episode lengths vary widely across training.
Revised: The latter has different (relative) semantics, but may be more appropriate when episode lengths vary widely across training.

Original: We denote it as rmed X := median( n X /L ) .
Revised: We denote it as rmed X := median( n X /L ) .

Original: 2. S WITCHING MECHANISMS
Revised: 2. S WITCHING MECHANISMS

Original: Granularity is but the coarsest facet of the ‘when’ question, but the more precise intra-episode timings matter too, namely when exactly to start and when to stop an exploratory period.
Revised: Granularity is but the coarsest facet of the ‘when’ question, but the more precise intra-episode timings matter too, namely when exactly to start and when to stop an exploratory period.

Original: This section introduces two mechanisms, blind and informed switching.
Revised: This section introduces two mechanisms, blind and informed switching.

Original: It is worth highlighting that, in general, the mechanism (or its time resolution) for entering explore mode differs from the one for exiting it (to enter exploit mode) – this asymmetry is crucial to obtain ﬂexible overall amounts of exploration.
Revised: It is worth highlighting that, in general, the mechanism (or its time resolution) for entering explore mode differs from the one for exiting it (to enter exploit mode) – this asymmetry is crucial to obtain ﬂexible overall amounts of exploration.

Original: If switching were symmetric , the proportion would be p X ≈ 0 .
Revised: If switching were symmetric , the proportion would be p X ≈ 0 .

Original: Blind switching
Revised: Blind switching

Original: The simplest switching mechanism does not take any state into account (thus, we call it blind ) and is only concerned with producing switches at some desired time resolution.
Revised: The simplest switching mechanism does not take any state into account (thus, we call it blind ) and is only concerned with producing switches at some desired time resolution.

Original: It can be implemented deterministically through a counter (e.g., enter explore mode after 100 exploit mode steps), or probabilistically (e.g., at each step, enter explore mode with probability 0 . 01 ).
Revised: It can be implemented deterministically through a counter (e.g., enter explore mode after 100 exploit mode steps), or probabilistically (e.g., at each step, enter explore mode with probability 0 . 01 ).

Original: Its expected duration can be parameterised in terms of raw steps or in terms of fractional episode length.
Revised: Its expected duration can be parameterised in terms of raw steps or in terms of fractional episode length.

Original: The opposite of blind switching is informed switching, as discussed below.
Revised: The opposite of blind switching is informed switching, as discussed below.

Original: Informed switching
Revised: Informed switching

Original: Going beyond blind switching opens up another rich set of design choices, with switching informed by the agent’s internal state.
Revised: Going beyond blind switching opens up another rich set of design choices, with switching informed by the agent’s internal state.

Original: There are two parts: ﬁrst, a scalar trigger signal is produced by the agent at each step, using its current information – drawing inspiration from human behaviour, we view the triggering signal as a proxy for uncertainty (Schulz et al., 2019): when uncertainty is high, the agent will switch to explore.
Revised: There are two parts: ﬁrst, a scalar trigger signal is produced by the agent at each step, using its current information – drawing inspiration from human behaviour, we view the triggering signal as a proxy for uncertainty (Schulz et al., 2019): when uncertainty is high, the agent will switch to explore.

Original: Second, a binary switching decision is taken based on the trigger signal (for example, by comparing it to a threshold).
Revised: Second, a binary switching decision is taken based on the trigger signal (for example, by comparing it to a threshold).

Original: Again, triggering will generally not be symmetric between entering and exiting an exploratory period.
Revised: Again, triggering will generally not be symmetric between entering and exiting an exploratory period.

Original: To keep this paper focused, we will look at one such informed trigger, dubbed ‘value promise discrepancy’ (see Appendix B for additional competitive variants).
Revised: To keep this paper focused, we will look at one such informed trigger, dubbed ‘value promise discrepancy’ (see Appendix B for additional competitive variants).

Original: This is an online proxy of how much of the reward that the agent’s past value estimate promised ( k steps ago) have actually come about.
Revised: This is an online proxy of how much of the reward that the agent’s past value estimate promised ( k steps ago) have actually come about.

Original: The intuition is that in uncertain parts of state space, this discrepancy will generally be larger than when everything goes as expected.
Revised: The intuition is that in uncertain parts of state space, this discrepancy will generally be larger than when everything goes as expected.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where V ( s ) is the agent’s value estimate at state s , R is the reward, and γ is a discount factor.
Revised: where V ( s ) is the agent’s value estimate at state s , R is the reward, and γ is a discount factor.

Original: Starting mode
Revised: Starting mode

Original: When periods last for a signiﬁcant fraction of episode length, it also matters how the sequence is initialised, i.e., whether an episode starts in explore or in exploit mode, or more generally, whether the agent explores more early in an episode or more later on.
Revised: When periods last for a signiﬁcant fraction of episode length, it also matters how the sequence is initialised, i.e., whether an episode starts in explore or in exploit mode, or more generally, whether the agent explores more early in an episode or more later on.

Original: It is conceivable that the best choice among these is domain dependent (see Figure 6): in most scenarios, the states at the beginning of an episode have been visited many times, thus starting with exploit mode can be beneﬁcial; in other domains however, early actions may disproportionately determine the available future paths (e.g., build orders in StarCraft (Churchill & Buro, 2011)).
Revised: It is conceivable that the best choice among these is domain dependent (see Figure 6): in most scenarios, the states at the beginning of an episode have been visited many times, thus starting with exploit mode can be beneﬁcial; in other domains however, early actions may disproportionately determine the available future paths (e.g., build orders in StarCraft (Churchill & Buro, 2011)).

Original: 2. F LEXIBILITY WITHOUT ADDED BURDEN
Revised: 2. F LEXIBILITY WITHOUT ADDED BURDEN

Original: Our approach introduces additional ﬂexibility to the exploration process, even when holding the speciﬁcs of the learning algorithm and the exploration mode ﬁxed.
Revised: Our approach introduces additional ﬂexibility to the exploration process, even when holding the speciﬁcs of the learning algorithm and the exploration mode ﬁxed.

Original: To prevent this from turning into an undue hyper-parameter tuning burden, we recommend adding two additional mechanisms.
Revised: To prevent this from turning into an undue hyper-parameter tuning burden, we recommend adding two additional mechanisms.

Original: Bandit adaptation
Revised: Bandit adaptation

Original: The two main added degrees of freedom in our intra-episodic switching set-up are when (or how often) to enter explore mode, and when (or how quickly) to exit it.
Revised: The two main added degrees of freedom in our intra-episodic switching set-up are when (or how often) to enter explore mode, and when (or how quickly) to exit it.

Original: These can be parameterised by a duration, termination probability or target rate (see Section 3.1).
Revised: These can be parameterised by a duration, termination probability or target rate (see Section 3.1).

Original: In either case, we propose to follow (Schaul et al., 2019) and (Badia et al., 2020a), and delegate the adaptation of these settings to a meta-controller, which is implemented as a non-stationary multi-armed bandit that maximises episodic return.
Revised: In either case, we propose to follow Schaul et al. and Badia et al. (2020a), and delegate the adaptation of these settings to a meta-controller, which is implemented as a non-stationary multi-armed bandit that maximises episodic return.

Original: As an added beneﬁt, the ‘when’ of exploration can now become adaptive to both the task, and the stage of learning.
Revised: As an added beneﬁt, the ‘when’ of exploration can now become adaptive to both the task, and the stage of learning.

Original: Homeostasis In practice, the scales of the informed trigger signals may vary substantially across domains and across training time.
Revised: Homeostasis In practice, the scales of the informed trigger signals may vary substantially across domains and across training time.

Original: For example, the magnitude of D promise will depend on reward scales and density and can decrease over time as accuracy improves (the signals could also be noisy).
Revised: For example, the magnitude of D promise will depend on reward scales and density and can decrease over time as accuracy improves (the signals could also be noisy).

Original: This means that naively setting a threshold hyper-parameter is impractical.
Revised: This means that naively setting a threshold hyper-parameter is impractical.

Original: For a simple remedy, we have taken inspiration from neuroscience (Turrigiano & Nelson, 2004) to add homeostasis to the binary switching mechanism, which tracks recent values of the signal and adapts the threshold for switching so that a speciﬁc average target rate is obtained.
Revised: For a simple remedy, we have taken inspiration from neuroscience (Turrigiano & Nelson, 2004) to add homeostasis to the binary switching mechanism, which tracks recent values of the signal and adapts the threshold for switching so that a speciﬁc average target rate is obtained.

Original: This functions as an adaptive threshold, making tuning straightforward because the target rate of switching is conﬁgured independently of the scales of the trigger signal.
Revised: This functions as an adaptive threshold, making tuning straightforward because the target rate of switching is conﬁgured independently of the scales of the trigger signal.

Original: See Appendix A for the details of the implementation.
Revised: See Appendix A for the details of the implementation.

Original:  R ESULTS
Revised:  R ESULTS

Original: The design space we propose contains a number of atypical ideas for how to structure exploration.
Revised: The design space we propose contains a number of atypical ideas for how to structure exploration.

Original: For this reason, we opted to keep the rest of our experimental setup very conventional, and include multiple comparable baselines, ablations and variations.
Revised: For this reason, we opted to keep the rest of our experimental setup very conventional, and include multiple comparable baselines, ablations and variations.

Original: Setup: R2D2 on Atari
Revised: Setup: R2D2 on Atari

Original: We conduct our investigations on a subset of games of the Atari Learning Environment (Bellemare et al., 2013), a common benchmark for the study of exploration.
Revised: We conduct our investigations on a subset of games of the Atari Learning Environment (Bellemare et al., 2013), a common benchmark for the study of exploration.

Original: All experiments are conducted across 7 games (F ROSTBITE , G RAVITAR , H.E.R.O., M ONTEZUMA ’ S
Revised: All experiments are conducted across 7 games (F ROSTBITE , G RAVITAR , H.E.R.O., M ONTEZUMA ’ S

Original: R EVENGE , M S .
Revised: R EVENGE , M S .

Original: P AC -M
Revised: P AC -M

Original: AN , P HOENIX , S TAR G UNNER ), the ﬁrst 5 of which are classiﬁed as hard exploration games (Bellemare et al., 2016), using 3 seeds per game.
Revised: AN , P HOENIX , S TAR G UNNER ), the ﬁrst 5 of which are classiﬁed as hard exploration games (Bellemare et al., 2016), using 3 seeds per game.

Original: For our agent, we use the R2D architecture (Kapturowski et al., 2019), which is a modern, distributed version of DQN (Mnih et al., 2015) that employs a recurrent network to approximate its Q-value function.
Revised: For our agent, we use the R2D architecture (Kapturowski et al., 2019), which is a modern, distributed version of DQN (Mnih et al., 2015) that employs a recurrent network to approximate its Q-value function.

Original: This is a common basis used in exploration studies, e.g., (Dabney et al., 2020; Badia et al., 2020b;a).
Revised: This is a common basis used in exploration studies (Dabney et al., 2020; Badia et al., 2020b;a).

Original: The only major modiﬁcation to conventional R2D2 is its exploration mechanism, where instead we implement all the variants of mode-switching introduced in Section 2.
Revised: The only major modiﬁcation to conventional R2D2 is its exploration mechanism, where instead we implement all the variants of mode-switching introduced in Section 2.

Original: Separately from the experience collected for learning, we run an evaluator process that assesses the performance of the current greedy policy.
Revised: Separately from the experience collected for learning, we run an evaluator process that assesses the performance of the current greedy policy.

Original: This is what we report in all our performance curves (see Appendix A for more details).
Revised: This is what we report in all our performance curves (see Appendix A for more details).

Original: Baselines
Revised: Baselines

Original: There are a few simple baselines worth comparing to, namely the pure explore mode ( p X = 1 , Fig.1:A) and the pure exploit mode ( p X = 0 ), as well as the step-wise interleaved ε -greedy execution (Fig.1:C), where p X = 0 .
Revised: There are a few simple baselines worth comparing to, namely the pure explore mode ( p X = 1 , Fig.1:A) and the pure exploit mode ( p X = 0 ), as well as the step-wise interleaved ε -greedy execution (Fig.1:C), where p X = 0 .

Original: =
Revised: =

Original: ε (without additional episodic or intra-episodic structure).
Revised: ε (without additional episodic or intra-episodic structure).

Original: Given its wide adoption in well-tuned prior work, we expect the latter to perform well overall.
Revised: Given its wide adoption in well-tuned prior work, we expect the latter to perform well overall.

Original: The fourth baseline picks a mode for an entire episode at a time (Fig.1:B), with the probability of picking X being adapted by a bandit meta-controller.
Revised: The fourth baseline picks a mode for an entire episode at a time (Fig.1:B), with the probability of picking X being adapted by a bandit meta-controller.

Original: We denote these as experiment-level-X , experiment-level-G , step-level-0.01 and episode-level-* respectively.
Revised: We denote these as experiment-level-X , experiment-level-G , step-level-0.01 and episode-level-* respectively.

Original: For each of these, we have a version with uniform ( X U ) and intrinsic ( X I ) explore mode.
Revised: For each of these, we have a version with uniform ( X U ) and intrinsic ( X I ) explore mode.

Original: 3. V ARIANTS OF INTRA - EPISODIC EXPLORATION
Revised: 3. V ARIANTS OF INTRA - EPISODIC EXPLORATION

Original: As discussed in Section 2, there are multiple dimensions along which two-mode intra-episodic exploration can vary.
Revised: As discussed in Section 2, there are multiple dimensions along which two-mode intra-episodic exploration can vary.

Original: The concrete ones for our experiments are:
Revised: The concrete ones for our experiments are:

Original: • Explore mode: uniform random X U , or RND intrinsic reward
Revised: • Explore mode: uniform random X U , or RND intrinsic reward

Original: X I (denoted XU and XI ).
Revised: X I (denoted XU and XI ).

Original: • Explore duration ( n X ): this can be a ﬁxed number of steps (1 , 10 , 100) , or one of these is adaptively picked by a bandit (denoted by * ), or the switching is symmetric between entering and exiting explore mode (denoted by = ).
Revised: • Explore duration ( n X ): this can be a ﬁxed number of steps (1 , 10 , 100) , or one of these is adaptively picked by a bandit (denoted by * ), or the switching is symmetric between entering and exiting explore mode (denoted by = ).

Original: • Trigger type: either blind or informed (based on value promise, see Section 2.2).
Revised: • Trigger type: either blind or informed (based on value promise, see Section 2.2).

Original: • Exploit duration ( n G ): for blind triggers, the exploit duration can be parameterised by ﬁxed number of steps (10 , 100 , 1000 , 10000) , indirectly deﬁned by a probability of terminating (0 . 1 , 0 . 01 , 0 .
Revised: • Exploit duration ( n G ): for blind triggers, the exploit duration can be parameterised by ﬁxed number of steps (10 , 100 , 1000 , 10000) , indirectly deﬁned by a probability of terminating (0 . 1 , 0 . 01 , 0 .

Original: 001 , 0 . 0001) , or adaptively picked by a bandit over these choices (denoted by n* or p* , respectively).
Revised: 001 , 0 . 0001) , or adaptively picked by a bandit over these choices (denoted by n* or p* , respectively).

Original: For informed triggers, the exploit duration is indirectly parameterised by a target rate in (0 . 1 , 0 . 01 , 0 .
Revised: For informed triggers, the exploit duration is indirectly parameterised by a target rate in (0 . 1 , 0 . 01 , 0 .

Original: 001 , 0 . 0001) , or a bandit over them ( p* ), which is in turn transformed into an adaptive switching threshold by homeostasis (Section 2.2).
Revised: 001 , 0 . 0001) , or a bandit over them ( p* ), which is in turn transformed into an adaptive switching threshold by homeostasis (Section 2.2).

Original: • Starting mode: G greedy (default) or X explore (denoted by G or X ).
Revised: • Starting mode: G greedy (default) or X explore (denoted by G or X ).

Original: We can concisely refer to a particular instance by a tuple that lists these choices.
Revised: We can concisely refer to a particular instance by a tuple that lists these choices.

Original: For example,
Revised: For example,

Original: XU-intra(100,informed,p*,X) denotes uniform random exploration X U , with ﬁxed 100step explore periods, triggered by the value-promise signal at a bandit-determined rate, and starting in explore mode.
Revised: XU-intra(100,informed,p*,X) denotes uniform random exploration X U , with ﬁxed 100step explore periods, triggered by the value-promise signal at a bandit-determined rate, and starting in explore mode.

Original: See Figure 2 for an illustration.
Revised: See Figure 2 for an illustration.

Original: 3. P ERFORMANCE RESULTS
Revised: 3. P ERFORMANCE RESULTS

Original: We start by reporting overall performance results, to reassure the reader that our method is viable (and convince them to keep reading the more detailed and qualitative results in the following sections).
Revised: We start by reporting overall performance results, to reassure the reader that our method is viable (and convince them to keep reading the more detailed and qualitative results in the following sections).

Original: Figure 3 shows performance across 7 Atari games according to two human-normalised aggregation metrics (mean and median), comparing one form of intra-episodic exploration to all the baselines, separately for each explore mode ( X U and X I ).
Revised: Figure 3 shows performance across 7 Atari games according to two human-normalised aggregation metrics (mean and median), comparing one form of intra-episodic exploration to all the baselines, separately for each explore mode ( X U and X I ).

Original: The headline result is that intra-episodic exploration improves over both step-level and episode-level baselines (as well as the pure experimentlevel modes that we would not expect to be very competitive).
Revised: The headline result is that intra-episodic exploration improves over both step-level and episode-level baselines (as well as the pure experimentlevel modes that we would not expect to be very competitive).

Original: The full learning curves per game are found in the appendix, and show scores on hard exploration games like M ONTEZUMA ’ S R EVENGE or P HOENIX that are also competitive in absolute terms (at our compute budget of 1 B frames).
Revised: The full learning curves per game are found in the appendix, and show scores on hard exploration games like M ONTEZUMA ’ S R EVENGE or P HOENIX that are also competitive in absolute terms (at our compute budget of 2 B frames).

Original: Note that there is a subtle difference to the learning setups between X U and X I , as the latter requires training a separate head to estimate intrinsic reward values.
Revised: Note that there is a subtle difference to the learning setups between X U and X I , as the latter requires training a separate head to estimate intrinsic reward values.

Original: This is present even in pure exploit mode, where it acts as an auxiliary task only (Jaderberg et al., 2016), hence the differences in pure greedy curves in Figure 3.
Revised: This is present even in pure exploit mode, where it acts as an auxiliary task only (Jaderberg et al., 2016), hence the differences in pure greedy curves in Figure 3.

Original: For details, see Appendix A.
Revised: For details, see Appendix A.

Original: 3. D IVERSITY RESULTS
Revised: 3. D IVERSITY RESULTS

Original: In a study like ours, the emphasis is not on measuring raw performance, but rather on characterising the diversity of behaviours arising from the spectrum of proposed variants.
Revised: In a study like ours, the emphasis is not on measuring raw performance, but rather on characterising the diversity of behaviours arising from the spectrum of proposed variants.

Original: A starting point is to return to Figure 1 (right), and assess how much of the previously untouched space is now ﬁlled
Revised: A starting point is to return to Figure 1 (right), and assess how much of the previously untouched space is now ﬁlled

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d e r e t
Revised: d e r e t

Original: [Figure]
Revised: [Figure]

Original: by intra-episodic variants, and how the ‘when’ characteristics translate into performance.
Revised: by intra-episodic variants, and how the ‘when’ characteristics translate into performance.

Original: Figureanswers these questions, and raises some new ones.
Revised: Figureanswers these questions, and raises some new ones.

Original: First off, the raw amount of exploration p X is not a sufﬁcient predictor of performance, implying that the temporal structure matters.
Revised: First off, the raw amount of exploration p X is not a sufﬁcient predictor of performance, implying that the temporal structure matters.

Original: It also shows substantial bandit adaptation at work: compare the exploration statistics at the start (squares) and end-points of training (crosses), and how these trajectories differ per game; a common pattern is that reducing p X far below 0 .
Revised: It also shows substantial bandit adaptation at work: compare the exploration statistics at the start (squares) and end-points of training (crosses), and how these trajectories differ per game; a common pattern is that reducing p X far below 0 .

Original: 5 is needed for high performance.
Revised: 5 is needed for high performance.

Original: Interestingly, these adaptations are similar between X U and X I , despite very different explore modes (and differing performance
Revised: Interestingly, these adaptations are similar between X U and X I , despite very different explore modes (and differing performance

Original: [Figure]
Revised: [Figure]

Original: results).
Revised: results).

Original: We would expect prolonged intrinsic exploration periods to be more useful than prolonged random ones, and indeed, comparing the high- rmed X variant (purple) across X U and X
Revised: We would expect prolonged intrinsic exploration periods to be more useful than prolonged random ones, and indeed, comparing the high- rmed X variant (purple) across X U and X

Original: I , it appears more beneﬁcial for the latter.
Revised: I , it appears more beneﬁcial for the latter.

Original: Zooming in on speciﬁc games, a few results stand out: in X U mode, the only variant that escapes the inherent local optimum of P HOENIX is the blind, doubly adaptive one (purple), with the bandits radically shifting the exploration statistics over the course of training.
Revised: Zooming in on speciﬁc games, a few results stand out: in X U mode, the only variant that escapes the inherent local optimum of P HOENIX is the blind, doubly adaptive one (purple), with the bandits radically shifting the exploration statistics over the course of training.

Original: In contrast, the best results on M ONTEZUMA ’ S R EVENGE are produced by the symmetric, informed trigger variant (light green), which is forced to retain a high p X .
Revised: In contrast, the best results on M ONTEZUMA ’ S R EVENGE are produced by the symmetric, informed trigger variant (light green), which is forced to retain a high p X .

Original: Finally, F ROSTBITE is the one game where an informed trigger (red) clearly outperforms its blind equivalent (purple).
Revised: Finally, F ROSTBITE is the one game where an informed trigger (red) clearly outperforms its blind equivalent (purple).

Original: These insights are still limited to summary statistics, so Figure 5 looks in more depth at the detailed temporal structure within episodes (as in Figure 1, left).
Revised: These insights are still limited to summary statistics, so Figure 5 looks in more depth at the detailed temporal structure within episodes (as in Figure 1, left).

Original: Here the main comparison is between blind and informed triggers, illustrating that the characteristics of the ﬁne-grained within-episode structure can differ massively, despite attaining the same high-level statistics p X and med X .
Revised: Here the main comparison is between blind and informed triggers, illustrating that the characteristics of the ﬁne-grained within-episode structure can differ massively, despite attaining the same high-level statistics p X and med X .

Original: We can see quite a lot of variation in the trigger structure – the moments we enter exploration are not evenly spaced anymore.
Revised: We can see quite a lot of variation in the trigger structure – the moments we enter exploration are not evenly spaced anymore.

Original: As a bonus, the less rigid structure of the informed trigger (and possibly the more carefully chosen switch points) end up producing better performance too.
Revised: As a bonus, the less rigid structure of the informed trigger (and possibly the more carefully chosen switch points) end up producing better performance too.

Original: Figure 6 sheds light on a complementary dimension, differentiating the effects of starting in explore or exploit mode.
Revised: Figure 6 sheds light on a complementary dimension, differentiating the effects of starting in explore or exploit mode.

Original: In brief, each of these can be consistently beneﬁcial in some games, and consistently harmful in others.
Revised: In brief, each of these can be consistently beneﬁcial in some games, and consistently harmful in others.

Original: Another observation here is the dynamics of the bandit adaptation: when starting in exploit mode, it exhibits a preference for long initial exploit periods in many games (up to 10000 steps), but that effect vanishes when starting in explore mode (see also Appendix C).
Revised: Another observation here is the dynamics of the bandit adaptation: when starting in exploit mode, it exhibits a preference for long initial exploit periods in many games (up to 10000 steps), but that effect vanishes when starting in explore mode (see also Appendix C).

Original: More subtle effects arise from the choice of parameterisation of switching rates.
Revised: More subtle effects arise from the choice of parameterisation of switching rates.

Original: Figure 7 shows a stark qualitative difference on how probabilistic switching differs from step-count based switching, with the former spanning a much wider diversity of outcomes, which improves performance.
Revised: Figure 7 shows a stark qualitative difference on how probabilistic switching differs from step-count based switching, with the former spanning a much wider diversity of outcomes, which improves performance.

Original: 3. T AKE - AWAYS
Revised: 3. T AKE - AWAYS

Original: Summarising the empirical results in this section, two messages stand out.
Revised: Summarising the empirical results in this section, two messages stand out.

Original: First, there seems to be a sweet spot in terms of temporal granularity, and intra-episodic exploration is the right step towards ﬁnding it.
Revised: First, there seems to be a sweet spot in terms of temporal granularity, and intra-episodic exploration is the right step towards ﬁnding it.

Original: Second, the vastly increased design space of our proposed family of methods gives rise to a large diversity of behavioural characteristics; and this diversity is not superﬁcial, it also translates to meaningful performance differences, with different effects in different games, which cannot be reduced to simplistic metrics, such as p X .
Revised: Second, the vastly increased design space of our proposed family of methods gives rise to a large diversity of behavioural characteristics; and this diversity is not superﬁcial, it also translates to meaningful performance differences, with different effects in different games, which cannot be reduced to simplistic metrics, such as p X .

Original: In addition, we provide some sensible rules-of-thumb for practitioners willing to join us on the journey of intra-episodic exploration.
Revised: In addition, we provide some sensible rules-of-thumb for practitioners willing to join us on the journey of intra-episodic exploration.

Original: In general, it is useful to let a bandit ﬁgure out the precise settings, but it is worth curating its choices to at most a handful.
Revised: In general, it is useful to let a bandit ﬁgure out the precise settings, but it is worth curating its choices to at most a handful.

Original: Jointly using two bandits across factored dimensions is very adaptive, but can sometimes be harmful when they decrease the signal-to-noise ratio in each other’s learning signal.
Revised: Jointly using two bandits across factored dimensions is very adaptive, but can sometimes be harmful when they decrease the signal-to-noise ratio in each other’s learning signal.

Original: Finally, the choice of the uncertainty-based trigger should be informed by the switching modes (see Appendix B for details).
Revised: Finally, the choice of the uncertainty-based trigger should be informed by the switching modes (see Appendix B for details).

Original:  D ISCUSSION Time-based exploration control
Revised:  D ISCUSSION Time-based exploration control

Original: The emphasis of our paper was on the beneﬁts of heterogeneous temporal structure in mode-switching exploration.
Revised: The emphasis of our paper was on the beneﬁts of heterogeneous temporal structure in mode-switching exploration.

Original: Another potential advantage over monolithic approaches is that it may be easier to tune hyper-parameters related to an explicit exploration budget (e.g., via p X ) than to tune an intrinsic reward coefﬁcient, especially if extrinsic reward scales change across tasks or time, and if the non-stationarity of the intrinsic reward affects its overall scale.
Revised: Another potential advantage over monolithic approaches is that it may be easier to tune hyper-parameters related to an explicit exploration budget (e.g., via p X ) than to tune an intrinsic reward coefﬁcient, especially if extrinsic reward scales change across tasks or time, and if the non-stationarity of the intrinsic reward affects its overall scale.

Original: Diversity for diversity’s sake
Revised: Diversity for diversity’s sake

Original: One role of a general-purpose exploration method is to allow an agent to get off the ground in a wide variety of domains.
Revised: One role of a general-purpose exploration method is to allow an agent to get off the ground in a wide variety of domains.

Original: While this may clash with sample-efﬁcient learning on speciﬁc domains, we believe that the former objective will come to dominate in the long run.
Revised: While this may clash with sample-efﬁcient learning on speciﬁc domains, we believe that the former objective will come to dominate in the long run.

Original: In this light, methods that exhibit more diverse behaviour are preferable for that reason alone because they are more likely to escape local optima or misaligned priors.
Revised: In this light, methods that exhibit more diverse behaviour are preferable for that reason alone because they are more likely to escape local optima or misaligned priors.

Original: Related work
Revised: Related work

Original: While not the most common approach to exploration in RL, we are aware of some notable work investigating non-trivial temporal structure.
Revised: While not the most common approach to exploration in RL, we are aware of some notable work investigating non-trivial temporal structure.

Original: The (cid:15)z -greedy algorithm (Dabney et al., 2020) initiates contiguous chunks of directed behaviour (‘ﬂights’) with the length sampled from a heavy-tailed distribution.
Revised: The (cid:15)z -greedy algorithm (Dabney et al., 2020) initiates contiguous chunks of directed behaviour (‘ﬂights’) with the length sampled from a heavy-tailed distribution.

Original: In contrast to our proposal, these ﬂights act with a single constant action instead of invoking an explore mode.
Revised: In contrast to our proposal, these ﬂights act with a single constant action instead of invoking an explore mode.

Original: Campos et al.
Revised: Campos et al.

Original: (2021) pursue a similar idea, but with ﬂights along pre-trained coverage policies, while Ecoffet et al.
Revised: (2021) pursue a similar idea, but with ﬂights along pre-trained coverage policies, while Ecoffet et al.

Original: (2021) chain a ‘return-to-state’ policy to an explore mode.
Revised: (2021) chain a ‘return-to-state’ policy to an explore mode.

Original: Maybe closest to our X I setting is the work of Bagot et al.
Revised: Maybe closest to our X I setting is the work of Bagot et al.

Original: (2020), where periods of intrinsic reward pursuit are explicitly invoked by the agent.
Revised: (2020), where periods of intrinsic reward pursuit are explicitly invoked by the agent.

Original: Exploration with gradual change instead of abrupt mode switches generally appears at long time-scales, such as when pursuing intrinsic rewards (Schmidhuber, 2010; Oudeyer & Kaplan, 2009) but can also be effective at shorter timescales, e.g., Never-Give-Up (Badia et al., 2020b).
Revised: Exploration with gradual change instead of abrupt mode switches generally appears at long time-scales, such as when pursuing intrinsic rewards (Schmidhuber, 2010; Oudeyer & Kaplan, 2009) but can also be effective at shorter timescales, e.g., Never-Give-Up (Badia et al., 2020b).

Original: Related work on the question of which states to prefer for exploration decisions (Tokic, 2010) tends not to consider prolonged exploratory periods.
Revised: Related work on the question of which states to prefer for exploration decisions (Tokic, 2010) tends not to consider prolonged exploratory periods.

Original: Relation to options
Revised: Relation to options

Original: Ideas related to switching behaviours at intra-episodic time scales are wellknown outside of the context of exploration.
Revised: Ideas related to switching behaviours at intra-episodic time scales are wellknown outside of the context of exploration.

Original: In the options framework in hierarchical RL, the goal is to chain together a sequence of sub-behaviours into a reward-maximising policy (Sutton et al., 1999;
Revised: In the options framework in hierarchical RL, the goal is to chain together a sequence of sub-behaviours into a reward-maximising policy (Sutton et al., 1999;

Original: Mankowitz et al., 2016).
Revised: Mankowitz et al., 2016).

Original: Some work has looked at using options for exploration too (Jinnai et al., 2019a; Bougie & Ichise, 2021).
Revised: Some work has looked at using options for exploration too (Jinnai et al., 2019a; Bougie & Ichise, 2021).

Original: In its full generality, the options framework is a substantially more ambitious endeavour than our proposal, as it requires learning a full state-dependent hierarchical policy that picks which option to start (and when), as well as jointly learning the options themselves.
Revised: In its full generality, the options framework is a substantially more ambitious endeavour than our proposal, as it requires learning a full state-dependent hierarchical policy that picks which option to start (and when), as well as jointly learning the options themselves.

Original: Limitations
Revised: Limitations

Original: Our proposed approach inherits many typical challenges for exploration methods, such as sample efﬁciency or trading off risk.
Revised: Our proposed approach inherits many typical challenges for exploration methods, such as sample efﬁciency or trading off risk.

Original: An aspect that is particular to the intra-episode switching case is the different nature of the off-policy-ness.
Revised: An aspect that is particular to the intra-episode switching case is the different nature of the off-policy-ness.

Original: The resulting effective policy can produce state distributions that differ substantially from those of either of the two base mode behaviours that are being interleaved.
Revised: The resulting effective policy can produce state distributions that differ substantially from those of either of the two base mode behaviours that are being interleaved.

Original: It can potentially visit parts of the state space that neither base policy would reach if followed from the beginning of the episode.
Revised: It can potentially visit parts of the state space that neither base policy would reach if followed from the beginning of the episode.

Original: While a boon for exploration, this might pose a challenge to learning, as it could require off-policy corrections that treat those states differently and do not only correct for differences in action space.
Revised: While a boon for exploration, this might pose a challenge to learning, as it could require off-policy corrections that treat those states differently and do not only correct for differences in action space.

Original: Our paper does not use (non-trivial) off-policy correction (see Appendix A) as our initial experimentation showed it is not an essential component in the current setting (see Figure 14).
Revised: Our paper does not use (non-trivial) off-policy correction (see Appendix A) as our initial experimentation showed it is not an essential component in the current setting (see Figure 15).

Original: We leave this intriguing ﬁnding for future investigation.
Revised: We leave this intriguing ﬁnding for future investigation.

Original: Future work
Revised: Future work

Original: With the dimensions laid out in Section 2, it should be clear that this paper can but scratch the surface.
Revised: With the dimensions laid out in Section 2, it should be clear that this paper can but scratch the surface.

Original: We see numerous opportunities for future work, some of which we already carried out initial investigations (see Appendix B).
Revised: We see numerous opportunities for future work, some of which we already carried out initial investigations (see Appendix B).

Original: For starters, the mechanism could go beyond two-mode and switch between exploit, explore, novelty and mastery
Revised: For starters, the mechanism could go beyond two-mode and switch between exploit, explore, novelty and mastery

Original: (Thomaz & Breazeal, 2008), or between many diverse forms of exploration, such as levels of optimism (Derman et al., 2020;
Revised: (Thomaz & Breazeal, 2008), or between many diverse forms of exploration, such as levels of optimism (Derman et al., 2020;

Original: Moskovitz et al., 2021).
Revised: Moskovitz et al., 2021).

Original: Triggers could be expanded or reﬁned by using different estimations of uncertainty, such as ensemble discrepancy (Wiering & Van Hasselt, 2008; Buckman et al., 2018), amortised value errors (Flennerhag et al., 2020), or density models (Bellemare et al., 2016;
Revised: Triggers could be expanded or reﬁned by using different estimations of uncertainty, such as ensemble discrepancy (Wiering & Van Hasselt, 2008; Buckman et al., 2018), amortised value errors (Flennerhag et al., 2020), or density models (Bellemare et al., 2016;

Original: Ostrovski et al., 2017); or other signals, such as salience (Downar et al., 2002), minimal coverage (Jinnai et al., 2019a;b), or empowerment (Klyubin et al., 2005; Gregor et al., 2016; Houthooft et al., 2016).
Revised: Ostrovski et al., 2017); or other signals, such as salience (Downar et al., 2002), minimal coverage (Jinnai et al., 2019a;b), or empowerment (Klyubin et al., 2005; Gregor et al., 2016; Houthooft et al., 2016).

Original: Conclusion
Revised: Conclusion

Original: We have presented an initial study of intra-episodic exploration, centred on the scenario of switching between an explore and an exploit mode.
Revised: We have presented an initial study of intra-episodic exploration, centred on the scenario of switching between an explore and an exploit mode.

Original: We hope this has broadened the available forms of temporal structure in behaviour, leading to more diverse, adaptive and intentional forms of exploration, in turn enabling RL to scale to ever more complex domains.
Revised: We hope this has broadened the available forms of temporal structure in behaviour, leading to more diverse, adaptive and intentional forms of exploration, in turn enabling RL to scale to ever more complex domains.

Original: A D ETAILED EXPERIMENTAL SETUP A. A TARI ENVIRONMENT
Revised: A D ETAILED EXPERIMENTAL SETUP A. A TARI ENVIRONMENT

Original: We use a selection of games from the widely used Atari Learning Environment (ALE, (Bellemare et al., 2013)).
Revised: We use a selection of games from the widely used Atari Learning Environment (ALE, Bellemare et al.

Original: It is conﬁgured to not expose the ‘life-loss’ signal, and use the full action set (discrete actions) for all games (not the per-game reduced effective action spaces).
Revised: It is conﬁgured to not expose the ‘life-loss’ signal, and use the full action set (discrete actions) for all games (not the per-game reduced effective action spaces).

Original: We also use the sticky -action randomisation as in (Machado et al., 2018).
Revised: We also use the sticky -action randomisation as in (Machado et al., 2018).

Original: Episodes time-out after 108 k frames (i.e.
Revised: Episodes time-out after 108 k frames (i.e.

Original: 30 minutes of real-time game play).
Revised: 30 minutes of real-time game play).

Original: Differently from most past Atari RL agents following DQN (Mnih et al., 2015), our agent uses the raw 210 × 160 RGB frames as input to its value function (one at a time, without frame stacking), though it still applies a max-pool operation over the most recent 2 frames to mitigate ﬂickering inherent to the Atari simulator.
Revised: Differently from most past Atari RL agents following DQN (Mnih et al., 2015), our agent uses the raw 210 × 160 RGB frames as input to its value function (one at a time, without frame stacking), though it still applies a max-pool operation over the most recent 2 frames to mitigate ﬂickering inherent to the Atari simulator.

Original: As in most past work, an action-repeat of 4 is applied, over which rewards are summed.
Revised: As in most past work, an action-repeat of 4 is applied, over which rewards are summed.

Original: A. A GENT
Revised: A. A GENT

Original: The agent used in our Atari experiments is a distributed implementation of a value- and replay-based
Revised: The agent used in our Atari experiments is a distributed implementation of a value- and replay-based

Original: RL algorithm derived from the Recurrent Replay Distributed DQN (R2D2) architecture (Kapturowski et al., 2019).
Revised: RL algorithm derived from the Recurrent Replay Distributed DQN (R2D2) architecture (Kapturowski et al., 2019).

Original: This system comprises of a ﬂeet of 120 CPU-based actors (combined with a single TPU for batch inference) concurrently generating experience and feeding it to a distributed experience replay buffer, and a single TPU-based learner randomly sampling batches of experience sequences from replay and performing updates of the recurrent value function by gradient descent on a suitable RL loss.
Revised: This system comprises of a ﬂeet of 120 CPU-based actors (combined with a single TPU for batch inference) concurrently generating experience and feeding it to a distributed experience replay buffer, and a single TPU-based learner randomly sampling batches of experience sequences from replay and performing updates of the recurrent value function by gradient descent on a suitable RL loss.

Original: The value function is represented by a convolutional torso feeding into a linear layer, followed by a recurrent LSTM (Hochreiter & Schmidhuber, 1997) core, whose output is processed by a further linear layer before ﬁnally being output via a Dueling value head (Wang et al., 2016).
Revised: The value function is represented by a convolutional torso feeding into a linear layer, followed by a recurrent LSTM (Hochreiter & Schmidhuber, 1997) core, whose output is processed by a further linear layer before ﬁnally being output via a Dueling value head (Wang et al., 2016).

Original: The exact parameterisation follows the slightly modiﬁed R2D2 presented in (Dabney et al., 2020) and (Schaul et al., 2021), see Table 1 for a full list of hyper-parameters.
Revised: The exact parameterisation follows the slightly modiﬁed R2D2 presented by Dabney et al. (2020) and Schaul et al. Refer to Table 1 for a full list of hyper-parameters.

Original: It is trained via stochastic gradient descent on a multi-step TD loss (more precisely, a 5 -step Q-learning loss) with the use of a periodically updated target network (Mnih et al., 2015) for bootstrap target computation, using minibatches of sampled replay sequences.
Revised: It is trained via stochastic gradient descent on a multi-step TD loss (more precisely, a 5 -step Q-learning loss) with the use of a periodically updated target network (Mnih et al., 2015) for bootstrap target computation, using mini-batches of sampled replay sequences.

Original: Replay sampling is performed using prioritized experience replay (Schaul et al., 2016) with priorities computed from sequences’ TD errors following the scheme introduced in (Kapturowski et al., 2019).
Revised: Replay sampling is performed using prioritized experience replay (Schaul et al., 2016) with priorities computed from sequences’ TD errors following the scheme introduced by Kapturowski et al.

Original: As in R2D2, sequences of 80 observations are used for replay, with a preﬁx of 20 observations used for burn-in.
Revised: As in R2D2, sequences of 80 observations are used for replay, with a preﬁx of 20 observations used for burn-in.

Original: In a slight deviation from the original, our agent uses a ﬁxed replay ratio of 1 , i.e. the learner or actors get throttled dynamically if the average number of times a sample gets replayed exceeds or falls below this value; this makes experiments more reproducible and stable.
Revised: In a slight deviation from the original, our agent uses a ﬁxed replay ratio of 1 , i.e. the learner or actors get throttled dynamically if the average number of times a sample gets replayed exceeds or falls below this value; this makes experiments more reproducible and stable.

Original: Actors periodically pull the most recent network parameters from the learner to be used in their exploratory policy.
Revised: Actors periodically pull the most recent network parameters from the learner to be used in their exploratory policy.

Original: In addition to feeding the replay buffer, all actors periodically report their reward, discount and return histories to the learner, which then calculates running estimates of reward, discount and return statistics to perform return-based scaling (Schaul et al., 2021).
Revised: In addition to feeding the replay buffer, all actors periodically report their reward, discount and return histories to the learner, which then calculates running estimates of reward, discount and return statistics to perform return-based scaling (Schaul et al., 2021).

Original: If applicable, the episodic returns from the actors are also sent to the non-stationary bandit(s) that adapt the distribution over exploration parameters (e.g., target ratios ρ or period lengths n X ).
Revised: If applicable, the episodic returns from the actors are also sent to the non-stationary bandit(s) that adapt the distribution over exploration parameters (e.g., target ratios ρ or period lengths n X ).

Original: In return, the bandit(s) provide samples from that distribution to each actor at the start of a new episode, as in (Schaul et al., 2019).
Revised: In return, the bandit(s) provide samples from that distribution to each actor at the start of a new episode, just like Schaul et al.

Original: Our agent is implemented with JAX (Bradbury et al., 2018), uses the Haiku (Hennigan et al., 2020),
Revised: Our agent is implemented with JAX (Bradbury et al., 2018), uses the Haiku (Hennigan et al., 2020),

Original: Optax (Budden et al., 2020b), Chex (Budden et al., 2020a), and RLax (Hessel et al., 2020) libraries for neural networks, optimisation, testing, and RL losses, respectively, and Reverb (Cassirer et al., 2020) for distributed experience replay
Revised: Optax (Budden et al., 2020b), Chex (Budden et al., 2020a), and RLax (Hessel et al., 2020) libraries for neural networks, optimisation, testing, and RL losses, respectively, and Reverb (Cassirer et al., 2020) for distributed experience replay.

Original: A. T RAINING AND EVALUATION PROTOCOLS
Revised: A. T RAINING AND EVALUATION PROTOCOLS

Original: All our experiments ran for 200 k learner updates.
Revised: All our experiments ran for 200 k learner updates.

Original: With a replay ratio of 1 , sequence length of(adjacent sequences overlapping by 40 observations), a batch size of 64 , and an action-repeat ofthis corresponds to a training budget of 200000 × 64 × 40 × 1 × 4 ≈ 2 B environment frames (which
Revised: With a replay ratio of 1 , sequence length of(adjacent sequences overlapping by 40 observations), a batch size of 64 , and an action-repeat ofthis corresponds to a training budget of 200000 × 64 × 40 × 1 × 4 ≈ 2 B environment frames (which

Original: [Table]
Revised: [Table]

Original: Convolutional torso channels 32 , 64 , 128 ,
Revised: torso [Table]

Original: Convolutional torso kernel sizes 7 , 5 , 5 ,
Revised: Convolutional torso kernel sizes 7 , 5 , 5 ,

Original: Convolutional torso strides 4 , 2 , 2 ,
Revised: Convolutional torso [Table]

Original: Pre-LSTM linear layer
Revised: Pre-LSTM linear layer

Original: [Table]
Revised: [Table]

Original: LSTM hidden units
Revised: LSTM hidden units

Original: Post-LSTM linear layer units
Revised: Post-LSTM linear layer units

Original: Dueling value head units 2 × 256 (separate linear layer for each of value and advantage)
Revised: Dueling value head units 2 × 256 (separate linear layer for each of value and advantage)

Original: Acting
Revised: Acting

Original: Initial random No-Ops
Revised: Initial random No-Ops

Original: None
Revised: None

Original: Sticky
Revised: Sticky

Original: [Table]
Revised: [Table]

Original: Yes (prob 0 . 25 )
Revised: Yes (prob 0 . 25 )

Original: [Table]
Revised: [Table]

Original: Number of
Revised: Number of

Original: [Table] [Table]
Revised: [Table] [Table]

Original: environmentsteps
Revised: environment

Original: [Table]
Revised: [Table]

Original: 80 (+ preﬁx of 20 of burn-in)
Revised: 80 (+ preﬁx of 20 of burn-in)

Original: [Table]
Revised: [Table]

Original: size 4 × 10 6 observations ( 10 5 part-overlapping sequences)
Revised: size [Table] observations ( 10 5 part-overlapping sequences)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: sampling exponent 0 .

Original: [Table]
Revised: [Table]

Original: ratio 1 update per sample (on average)
Revised: ratio 1 update per sample (on average)

Original: [Table]
Revised: [Table]

Original: k =
Revised: k =

Original: [Table]
Revised: [Table]

Original: None
Revised: None

Original: Discount γ 0 .
Revised: Discount γ 0 .

Original: [Table]
Revised: [Table]

Original: clipping
Revised: clipping

Original: None
Revised: None

Original: Return-based scaling as in (Schaul et al., 2021)
Revised: Return [Table] based scaling as in (Schaul et al., 2021)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Adam (Kingma & Ba, 2014), learning rate η = 2 × 10 − 4 , (cid:15) =
Revised: Adam (Kingma & Ba, 2014), learning rate η = 2 × 10 − 4 , (cid:15) =

Original: 10 − 8 , momentum β 1 = 0 .
Revised: 10 − 8 , momentum β 1 = 0 .

Original: 9 , second moment β 2 = 0 .
Revised: 9 , second moment β 2 = 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 400 updates

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 2 × is less than 10% of the original R2D2 budget).
Revised: 2 × is less than 10% of the original R2D2 budget).

Original: In wall-clock-time, one such experiment takes about 12 hours (while 2 TPUs  and 120 CPUs).
Revised: In wall-clock-time, one such experiment takes about 12 hours using 2 TPUs (one for the batch inference, the other for the learner) and 120 CPUs.

Original: For evaluation, a separate actor (not feeding the replay buffer) is running alongside the agent using a greedy policy ( ε = 0 ), and pulling the most recent parameters at the beginning of each episode.
Revised: For evaluation, a separate actor (not feeding the replay buffer) is running alongside the agent using a greedy policy ( ε = 0 ), and pulling the most recent parameters at the beginning of each episode.

Original: We follow standard evaluation methodology for Atari, reporting mean and median ‘human-normalised’ scores as introduced in (Mnih et al., 2015) (i.e. the episode returns are normalised so that 0 corresponds to the score of a uniformly random policy while 1 corresponds to human performance), as well as the mean ‘human-capped’ score which caps the per-game performance at human level.
Revised: We follow standard evaluation methodology for Atari, reporting mean and median ‘human-normalised’ scores as introduced in (Mnih et al., 2015) (i.e. the episode returns are normalised so that 0 corresponds to the score of a uniformly random policy while 1 corresponds to human performance), as well as the mean ‘human-capped’ score which caps the per-game performance at human level.

Original: Error bars or shaded curves correspond to the minimum and maximum values across these seeds.
Revised: Error bars or shaded curves correspond to the minimum and maximum values across these seeds.

Original: A. R ANDOM NETWORK DISTILLATION
Revised: A. R ANDOM NETWORK DISTILLATION

Original: The agent setup for the X I experiments differs in a few ways from the default described above.
Revised: The agent setup for the X I experiments differs in a few ways from the default described above.

Original: First, a separate network is trained via Random Network Distillation (RND, (Burda et al., 2018)), which consists of a simple convnet with an MLP (no recurrence); for detailed settings, see RND section in Table 1.
Revised: First, a separate network is trained via Random Network Distillation (RND, (Burda et al., 2018)), which consists of a simple convnet with an MLP (no recurrence); for detailed settings, see RND section in Table 1.

Original: The RND prediction network is updated jointly with the Q-value network, on the same data.
Revised: The RND prediction network is updated jointly with the Q-value network, on the same data.

Original: The intrinsic reward derived from the RND loss is pursued at the same discount γ = 0 .
Revised: The intrinsic reward derived from the RND loss is pursued at the same discount γ = 0 .

Original: 997 as the external reward in G .
Revised: 997 as the external reward in G .

Original: The Q-value network is augmented with a second head that predicts the
Revised: The Q-value network is augmented with a second head that predicts the

Original: Q-values for the intrinsic reward; this branches off after the ‘Post-LSTM linear layer’ (with 256 ), and is the same type of dueling head, using the same scale normalisation method (Schaul et al., 2021).
Revised: Q-values for the intrinsic reward; this branches off after the ‘Post-LSTM linear layer’ (with 256 ), and is the same type of dueling head, using the same scale normalisation method (Schaul et al., 2021).

Original: In addition, the 5 -step Q-learning is adapted to use a simple off-policy correction, namely trace-cutting on non-greedy actions (akin to Watkins Q( λ ) with λ = 1 ), separately for each learning head.
Revised: In addition, the 5 -step Q-learning is adapted to use a simple off-policy correction, namely trace-cutting on non-greedy actions (akin to Watkins Q( λ ) with λ = 1 ), separately for each learning head.

Original: 3 The X I policy is the greedy policy according to the Q-values of the second head.
Revised: 3 The X I policy is the greedy policy according to the Q-values of the second head.

Original: Note that because of these differences in set-up, and especially because the second head can function as an auxiliary learning target, it may be misleading to compare X I and X U results head-to-head: we recommend looking at how things change within one of these settings (across variants of intraepisodic exploration or the baselines), rather than between them.
Revised: Note that because of these differences in set-up, and especially because the second head can function as an auxiliary learning target, it may be misleading to compare X I and X U results head-to-head: we recommend looking at how things change within one of these settings (across variants of intraepisodic exploration or the baselines), rather than between them.

Original: A. H OMEOSTASIS
Revised: A. H OMEOSTASIS

Original: The role of the homeostasis mechanism is to transform a sequence of scalar signals x t ∈ R (for 1 ≤ t ≤ T ) into a sequence of binary switching decisions y t ∈ { 0 , 1 } so that the average number of switches approximates a desired target rate ρ , that is , 1 T (cid:80) t y t ≈ ρ , and high values of x t correspond to a higher probability of y t = 1 .
Revised: The role of the homeostasis mechanism is to transform a sequence of scalar signals x t ∈ R (for 1 ≤ t ≤ T ) into a sequence of binary switching decisions y t ∈ { 0 , 1 } so that the average number of switches approximates a desired target rate ρ , that is , 1 T (cid:80) t y t ≈ ρ , and high values of x t correspond to a higher probability of y t = 1 .

Original: Furthermore, the decision at any point y t can only be based on the past signals x 1: t .
Revised: Furthermore, the decision at any point y t can only be based on the past signals x 1: t .

Original: One way to achieve this is to exponentiate x (to turn it into a positive number x + ) and then set an adaptive threshold to determine when to switch.
Revised: One way to achieve this is to exponentiate x (to turn it into a positive number x + ) and then set an adaptive threshold to determine when to switch.

Original: Algorithm 1 describes how this is done in pseudo-code.
Revised: Algorithm 1 describes how this is done in pseudo-code.

Original: The implementation deﬁnes a time-scale of interest τ := min( t, 100 /ρ ) , and uses it to track moving averages of three quantities, namely the mean and variance of x , as well as the mean of x + .
Revised: The implementation deﬁnes a time-scale of interest τ := min( t, 100 /ρ ) , and uses it to track moving averages of three quantities, namely the mean and variance of x , as well as the mean of x + .

Original: Require: target rate ρ
Revised: Require: target rate ρ

Original: 7: standardise x
Revised: 7: standardise x

Original: 8: update transformed moving average x + ← (1 − 1 τ ) x
Revised: 8: update transformed moving average x + ← (1 − 1 τ ) x

Original: + + 1 τ x + 9: sample y t ∼ Bernoulli (cid:16) min (cid:16) 1 , ρ x + x + (cid:17)(cid:17) 10: end for
Revised: + + 1 τ x + 9: sample y t ∼ Bernoulli (cid:16) min (cid:16) 1 , ρ x + x + (cid:17)(cid:17) 10: end for

Original: In our informed trigger experiments we use value promise as the particular choice of trigger signal x t =
Revised: In our informed trigger experiments we use value promise as the particular choice of trigger signal x t =

Original: D promise ( t − k, t ) .
Revised: D promise ( t − k, t ) .

Original: As discussed in Section3.1, when using a bandit, its choices for target rates are ρ ∈ { 0 . 1 , 0 . 01 , 0 .
Revised: As discussed in Section3.1, when using a bandit, its choices for target rates are ρ ∈ { 0 . 1 , 0 . 01 , 0 .

Original: B O THER VARIANTS
Revised: B O THER VARIANTS

Original: The results we report in the main paper are but a subset of the possible variants that could be tried in this rather large design space.
Revised: The results we report in the main paper are but a subset of the possible variants that could be tried in this rather large design space.

Original: In fact, we have done initial investigations on a few of these, which we report below.
Revised: In fact, we have done initial investigations on a few of these, which we report below.

Original: B. A DDITIONAL EXPLORE MODES
Revised: B. A DDITIONAL EXPLORE MODES

Original: Softer explore-exploit modes The all-or-nothing setting with a greedy exploit mode and a uniform random explore mode is clear and simple, but it is plausible that less extreme choices could work well too, such as an ε -greedy explore mode with ε = 0 .
Revised: Softer explore-exploit modes The all-or-nothing setting with a greedy exploit mode and a uniform random explore mode is clear and simple, but it is plausible that less extreme choices could work well too, such as an ε -greedy explore mode with ε = 0 .

Original: 4 and an ε -greedy exploit mode with
Revised: 4 and an ε -greedy exploit mode with

Original: [Figure]
Revised: [Figure]

Original: ε = 0 .
Revised: ε = 0 .

Original: We denote this pairing as X S .
Revised: We denote this pairing as X S .

Original: Preliminary results (see Figure 15) indicate that overall performance is mostly similar to X U , possibly less affected by the choice of granularity and triggers.
Revised: Preliminary results (see Figure 16) indicate that overall performance is mostly similar to X U , possibly less affected by the choice of granularity and triggers.

Original: Different discounts Another category of explore mode ( X γ ) is to pursue external reward but at a different time-scale (e.g., a much shorter discount like γ = 0 . 97 ).
Revised: Different discounts Another category of explore mode ( X γ ) is to pursue external reward but at a different time-scale (e.g., a much shorter discount like γ = 0 . 97 ).

Original: This results in less of a switch between explore and exploit modes, but rather in an alternation of long-term and short-term reward pursuits, producing a different kind of behavioural diversity.
Revised: This results in less of a switch between explore and exploit modes, but rather in an alternation of long-term and short-term reward pursuits, producing a different kind of behavioural diversity.

Original: So far, we do not have conclusive results to report with this mode.
Revised: So far, we do not have conclusive results to report with this mode.

Original: B. A DDITIONAL INFORMED TRIGGERS
Revised: B. A DDITIONAL INFORMED TRIGGERS

Original: Action-mismatch-based triggers Another type of informed trigger is to derive an uncertainty estimate from the discrepancies across an ensemble.
Revised: Action-mismatch-based triggers Another type of informed trigger is to derive an uncertainty estimate from the discrepancies across an ensemble.

Original: For example, we can train two heads that use an identical Q-learning update but are initialised differently.
Revised: For example, we can train two heads that use an identical Q-learning update but are initialised differently.

Original: From that, we can measure multiple forms of discrepancy, a nice and robust one is to rank the actions according to each head and compute how large the overlap among the top- k actions is.
Revised: From that, we can measure multiple forms of discrepancy, a nice and robust one is to rank the actions according to each head and compute how large the overlap among the top- k actions is.

Original: Variance-based triggers Another type of informed trigger is to measure the variance of the Qvalues themselves, taken across such an ensemble (of two heads) and use that as an alternative uncertainty-based trigger.
Revised: Variance-based triggers Another type of informed trigger is to measure the variance of the Qvalues themselves, taken across such an ensemble (of two heads) and use that as an alternative uncertainty-based trigger.

Original: Figure 8 shows preliminary results on how performance compares across these two new informed triggers, in relation to the value-promise one from Section 2.2.
Revised: Figure 8 shows preliminary results on how performance compares across these two new informed triggers, in relation to the value-promise one from Section 2.2.

Original: Overall, the action-mismatch trigger seems to have an edge, at least in this setting, and we plan to investigate this further in the future.
Revised: Overall, the action-mismatch trigger seems to have an edge, at least in this setting, and we plan to investigate this further in the future.

Original: From other probing experiments, it appears that for other explore modes, different trigger signals are more suitable.
Revised: From other probing experiments, it appears that for other explore modes, different trigger signals are more suitable.

Original: C A DDITIONAL RESULTS
Revised: C A DDITIONAL RESULTS

Original: This section includes additional results.
Revised: This section includes additional results.

Original: Wherever the main ﬁgures included a subset of games or variants (Figures 4, 5, 7) we show full results here (Figures10, 11, 12,  respectively), and the aggregated performances of Figure 3 are split out into individual games in Figure 9.
Revised: Wherever the main ﬁgures included a subset of games or variants (Figures 4, 5, 7) we show full results here (Figures 11, 12, 13, respectively), and the aggregated performances of Figure 3 are split out into individual games in Figure 10.

Original: Also, some of the learning curves from Figures 4 and 10 are shown in Figure 15.
Revised: Also, some of the learning curves from Figures 4 and 11 are shown in Figure 16.

Original: In addition, Figure 13 illustrates how the internal bandit probabilities evolve over time based on starting mode for the experiments shown in Figure 6.[Figure]
Revised: In addition, Figure 14 illustrates how the internal bandit probabilities evolve over time based on starting mode for the experiments shown in Figure 6.

Original: i s o d e r e t u r n [Figure] [Figure] i
Revised: Lastly, we provide some bonus illustrations for the intra-episodic design decisions included in the namings of our variants in Figure 9 with the hope of facilitating the interpretation and intuition

Original: [Figure]
Revised: [Figure]

Original: o d

Original: [Figure]
Revised: [Figure]

Original: r
Revised: We show the characteristic space of exploration (summarized by rmed X and p X on the X and Y axis, respectively) on rows 1, 2, 5, and 6, and how different explore-exploit proportions translate to performance (error bars spanning the min and max performance across 3 seeds) on rows 3, 4, 7, and 8, for X U mode (top) and X

Original: [Figure]
Revised: [Figure]

Original: t u r n

Original: [Figure][Figure] [Figure]
Revised: [Figure]