Original: E SCAPING S TOCHASTIC T RAPS WITH A LEATORIC M APPING A GENTS
Revised: E SCAPING S TOCHASTIC T RAPS WITH A LEATORIC M APPING A GENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When extrinsic rewards are sparse, artiﬁcial agents struggle to explore an environment.
Revised: When extrinsic rewards are sparse, artificial agents struggle to explore an environment.

Original: Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but fails when faced with action-dependent noise sources.
Revised: Curiosity, implemented as an intrinsic reward for prediction errors, can improve exploration but fails when faced with action-dependent noise sources.

Original: We present aleatoric mapping agents (AMAs), a neuroscience inspired novel form of curiosity modeled on the cholinergic system of the mammalian brain.
Revised: We present aleatoric mapping agents (AMAs), a neuroscience inspired novel form of curiosity modeled on the cholinergic system of the mammalian brain.

Original: AMAs aim to explicitly ascertain when dynamics of the environment are unpredictable, even if those dynamics are induced by the actions of the agent.
Revised: AMAs aim to explicitly ascertain when dynamics of the environment are unpredictable, even if those dynamics are induced by the actions of the agent.

Original: This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states, reducing intrinsic rewards for those transitions that are unpredictable.
Revised: This is achieved by generating separate forward predictions for the mean and aleatoric uncertainty of future states, reducing intrinsic rewards for those transitions that are unpredictable.

Original: We demonstrate that in a range of environments AMAs are able to circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents.
Revised: We demonstrate that in a range of environments AMAs are able to circumvent action-dependent stochastic traps that immobilise conventional curiosity driven agents.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Efﬁcient exploration is a central problem in reinforcement learning.
Revised: Efficient exploration is a central problem in reinforcement learning.

Original: Exploration is particular challenging in environments with sparse rewards—requiring agents to navigate with limited guidance (e.g. Sutton & Barto (2018); Pathak et al.
Revised: Exploration is particular challenging in environments with sparse rewards—requiring agents to navigate with limited guidance (e.g. Sutton & Barto (2018); Pathak et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: (2018b), see Weng (2020) for a review).
Revised: (2018b), see Weng (2020) for a review).

Original: A notable exploration method that effectively deals with sparse rewards is curiosity driven learning—where agents are equipped with a self-supervised forward prediction model that employs prediction errors as intrinsic rewards Schmidhuber (1991b); Pathak et al.
Revised: A notable exploration method that effectively deals with sparse rewards is curiosity driven learning—where agents are equipped with a self-supervised forward prediction model that employs prediction errors as intrinsic rewards Schmidhuber (1991b); Pathak et al.

Original: (2017); Schmidhuber (1991a).
Revised: (2017); Schmidhuber (1991a).

Original: Curiosity is built upon the intuition that in unexplored regions of the environment, the forward prediction error of the agent’s internal model will be large Schmidhuber (1991b); Pathak et al.
Revised: Curiosity is built upon the intuition that in unexplored regions of the environment, the forward prediction error of the agent’s internal model will be large Schmidhuber (1991b); Pathak et al.

Original: As a result, agents are rewarded for visiting regions of the state space that they have not previously occupied.
Revised: As a result, agents are rewarded for visiting regions of the state space that they have not previously occupied.

Original: If, however, a particular state transition is impossible to predict, it will trap a curious agent Burda et al.
Revised: If, however, a particular state transition is impossible to predict, it will trap a curious agent Burda et al.

Original: (2018b)
Revised: (2018b)

Original: ; Schmidhuber (1991a).
Revised: ; Schmidhuber (1991a).

Original: This is referred to as the noisy TV problem (e.g. Burda et al.
Revised: This is referred to as the noisy TV problem (e.g. Burda et al.

Original: (2018b); Schmidhuber (1991a)), the etymology being that a naively curious agent could dwell on the unpredictability of a noisy TV screen.
Revised: (2018b); Schmidhuber (1991a)), the etymology being that a naively curious agent could dwell on the unpredictability of a noisy TV screen.

Original: Several existing curiosity-like methods Burda et al.
Revised: Several existing curiosity-like methods Burda et al.

Original: (2018b)
Revised: (2018b)

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: (2017; 2019) aim to avoid noisy TVs or “stochastic traps” Shyam et al.
Revised: (2017; 2019) aim to avoid noisy TVs or “stochastic traps” Shyam et al.

Original: Nevertheless, employing dynamics based prediction errors as intrinsic rewards is difﬁcult as current methods either fail when stochastic traps are action-dependent, or require an ensemble of dynamics models Pathak et al.
Revised: Nevertheless, employing dynamics based prediction errors as intrinsic rewards is difficult as current methods either fail when stochastic traps are action-dependent, or require an ensemble of dynamics models Pathak et al.

Original: (2017; 2019); Shyam et al.
Revised: (2017; 2019); Shyam et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: (2018a).
Revised: (2018a).

Original: This work presents aleatoric mapping agents (AMAs), which use single network deterministic uncertainty estimation Kendall & Gal (2017) to escape stochastic traps.
Revised: This work presents aleatoric mapping agents (AMAs), which uses aleatoric uncertainty estimation Kendall & Gal (2017) to escape stochastic traps.

Original: AMAs are both inspired by and build upon proposals developed within neuroscience Yu & Dayan (2005), that suggest expected uncertainties in top down predictions are signalled by the modulation of cortical acetylcholine.
Revised: AMAs are both inspired by and build upon proposals developed within neuroscience Yu & Dayan (2005), that suggest expected uncertainties in top down predictions are signalled by the modulation of cortical acetylcholine.

Original: We build on this theory by algorithmically demonstrating a functional purpose for expected aleatoric uncertainties that is consistent with, but also further speciﬁes, Yu & Dayan (2005)’s predictions of cholinergic activity in the Posner task.
Revised: We build on this theory by algorithmically demonstrating a functional purpose for expected aleatoric uncertainties that is consistent with, but also further specifies, Yu & Dayan (2005)’s predictions of cholinergic activity in the Posner task.

Original: Our work adds further credence to Yu & Dayan (2005)’s call for experimental tests of whether acetylcholine signals epistemic or aleatoric uncertainties.
Revised: Our work adds further credence to Yu & Dayan (2005)’s call for experimental tests of whether acetylcholine signals epistemic or aleatoric uncertainties.

Original: Accordingly, a blueprint for a VR mouse task, along with theoretical predictions of cholinergic uncertainty signalling in such a task, is presented in Section 5.
Revised: Accordingly, a blueprint for a VR mouse task, along with theoretical predictions of cholinergic uncertainty signalling in such a task, is presented in Section 5.

Original:  B ACKGROUND 2. E PISTEMIC AND A LEATORIC U NCERTAINTIES
Revised:  B ACKGROUND 2. E PISTEMIC AND A LEATORIC U NCERTAINTIES

Original: The uncertainty of a statistical model can be described as the sum of two theoretically distinct types of uncertainty: epistemic uncertainty and aleatoric uncertainty  Hüllermeier & Waegeman (2021).
Revised: The uncertainty of a statistical model can be described as the sum of two theoretically distinct types of uncertainty: epistemic uncertainty and aleatoric uncertainty (e.g. Hora (1996), see Hüllermeier & Waegeman (2021) for a review).

Original: Epistemic uncertainty measures the errors of a model’s prediction that can be minimised with additional experience and learning Hüllermeier & Waegeman (2021).
Revised: Epistemic uncertainty measures the errors of a model’s prediction that can be minimised with additional experience and learning Hüllermeier & Waegeman (2021).

Original: As a result, an agent using epistemic uncertainties as intrinsic rewards tends to value dynamics it has not previously encountered, and hence cannot predict accurately, but could learn to predict in the future (e.g. Osband et al.
Revised: As a result, an agent using epistemic uncertainties as intrinsic rewards tends to value dynamics it has not previously encountered, and hence cannot predict accurately, but could learn to predict in the future (e.g. Osband et al.

Original: More concretely, epistemic uncertainty can be considered to be the expected information gain of observing the next predicted state Mukhoti et al.
Revised: More concretely, epistemic uncertainty can be considered to be the expected information gain of observing the next predicted state Mukhoti et al.

Original: On the other hand, prediction errors that are due to aleatoric uncertainties are, by deﬁnition, a result of unpredictable processes Hüllermeier & Waegeman (2021).
Revised: On the other hand, prediction errors that are due to aleatoric uncertainties are, by definition, a result of unpredictable processes Hüllermeier & Waegeman (2021).

Original: Therefore, any agents that receives intrinsic rewards for aleatoric dynamics risks being trapped, as exempliﬁed by the noisy TV problem Schmidhuber (1991a); Burda et al.
Revised: Therefore, any agents that receives intrinsic rewards for aleatoric dynamics risks being trapped, as exemplified by the noisy TV problem Schmidhuber (1991a); Burda et al.

Original: (2018a).
Revised: (2018a).

Original: Direct estimation of epistemic uncertainty surrounding future states would be an ideal basis for a curious agent but tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem Gal (2016).
Revised: Direct estimation of epistemic uncertainty surrounding future states would be an ideal basis for a curious agent but tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem Gal (2016).

Original: We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error.
Revised: We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error.

Original: This is similar to methods that separate epistemic and aleatoric uncertainties in return predictions Clements et al.
Revised: This is similar to methods that separate epistemic and aleatoric uncertainties in return predictions Clements et al.

Original: (2019), or within a latent variable model
Revised: (2019), or within a latent variable model

Original: Depeweg et al.
Revised: Depeweg et al.

Original: (2018)— allowing for the construction of policies that are rewarded for exploring their environments and punished for experiencing aleatoric uncertainty.
Revised: (2018)—allowing for the construction of policies that are rewarded for exploring their environments and punished for experiencing aleatoric uncertainty.

Original: However, as far as we are aware, we are the ﬁrst to compute aleatoric uncertainties within a scalable curiosity framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty.
Revised: However, as far as we are aware, we are the first to compute aleatoric uncertainties within a scalable curiosity framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty.

Original: 2. C URIOSITY AND I NTRINSIC M OTIVATION IN R EINFORCEMENT L EARNING
Revised: 2. C URIOSITY AND I NTRINSIC M OTIVATION IN R EINFORCEMENT L EARNING

Original: Curiosity-driven Pathak et al.
Revised: Curiosity-driven Pathak et al.

Original: (2017) agents assign value to states of the environment that they deem to be “interesting” Still & Precup (2012); Schmidhuber (1997).
Revised: (2017) agents assign value to states of the environment that they deem to be “interesting” Still & Precup (2012); Schmidhuber (1997).

Original: How a curiosity based method computes whether a state is “interesting” Still & Precup (2012); Schmidhuber (1997) is usually its deﬁning characteristic.
Revised: How a curiosity based method computes whether a state is “interesting” Still & Precup (2012); Schmidhuber (1997) is usually its defining characteristic.

Original: The original formulation of curiosity used prediction errors directly as intrinsic rewards Schmidhuber (1991b).
Revised: The original formulation of curiosity used prediction errors directly as intrinsic rewards Schmidhuber (1991b).

Original: The noisy TV problem quickly emerged when using this naïve approach in stochastic environments Schmidhuber (1991a).
Revised: The noisy TV problem quickly emerged when using this naïve approach in stochastic environments Schmidhuber (1991a).

Original: In order to evade the allure of stochastic traps, the ﬁrst proposed solution to the noisy TV problem implements “interesting” Still &
Revised: In order to evade the allure of stochastic traps, the first proposed solution to the noisy TV problem implements “interesting” Still &

Original: Precup (2012); Schmidhuber (1997) as prediction errors that reduce over time Schmidhuber (1991a);
Revised: Precup (2012); Schmidhuber (1997) as prediction errors that reduce over time Schmidhuber (1991a);

Original: Kaplan & Oudeyer (2007).
Revised: Kaplan & Oudeyer (2007).

Original: Others consider “interesting” Still & Precup (2012); Schmidhuber (1997) to mean a high dependency between present and future states and actions (i.e. “interesting” things are predictable Still & Precup (2012) or controllable Mohamed & Jimenez Rezende (2015)).
Revised: Others consider “interesting” Still & Precup (2012); Schmidhuber (1997) to mean a high dependency between present and future states and actions (i.e. “interesting” things are predictable Still & Precup (2012) or controllable Mohamed & Jimenez Rezende (2015)).

Original: Inverse dynamics feature (IDF) curiosity Pathak et al.
Revised: Inverse dynamics feature (IDF) curiosity Pathak et al.

Original: (2017) rejuvenated interest in using one step prediction errors as intrinsic rewards.
Revised: (2017) rejuvenated interest in using one step prediction errors as intrinsic rewards.

Original: IDF curiosity avoids stochastic traps by computing prediction errors with features that aim to only contain information concerning stimuli the agent can affect Pathak et al.
Revised: IDF curiosity avoids stochastic traps by computing prediction errors with features that aim to only contain information concerning stimuli the agent can affect Pathak et al.

Original: Further experiments Burda et al.
Revised: Further experiments Burda et al.

Original: (2018a) showed that simple one-step prediction errors also work effectively within a random representation space generated by feeding state observations through a randomly initialised network.
Revised: (2018a) showed that simple one-step prediction errors also work effectively within a random representation space generated by feeding state observations through a randomly initialised network.

Original: Burda et al.
Revised: Burda et al.

Original: (2018a) also showed the (IDF) approach is vulnerable to action-dependent noisy TVs—demonstrated by giving the agent a ‘remote control’ to a noisy TV in the environment that could induce unpredictable environment transitions.
Revised: (2018a) also showed the (IDF) approach is vulnerable to action-dependent noisy TVs—demonstrated by giving the agent a ‘remote control’ to a noisy TV in the environment that could induce unpredictable environment transitions.

Original: This motivated random network distillation (RND) Burda et al.
Revised: This motivated random network distillation (RND) Burda et al.

Original: (2018b), which removes dynamics from the prediction problem altogether—instructing a network to learn to predict the output of another ﬁxed randomly initialised network at each state, using the resulting error as intrinsic rewards.
Revised: (2018b), which removes dynamics from the prediction problem altogether—instructing a network to learn to predict the output of another fixed randomly initialised network at each state, using the resulting error as intrinsic rewards.

Original: Other exploration methods explicitly leverage uncertainty quantiﬁcation for exploration.
Revised: Other exploration methods explicitly leverage uncertainty quantification for exploration.

Original: The canonical approach is “optimism under uncertainty”, which in its most basic form means weighting the value of state-actions pairs inversely to the number of times they have been experienced (Sutton & Barto, 2018, p. 36).
Revised: The canonical approach is “optimism under uncertainty”, which in its most basic form means weighting the value of state-actions pairs inversely to the number of times they have been experienced (Sutton & Barto, 2018, p. 36).

Original: Known as count based methods Strehl & Littman (2008); Bellemare et al.
Revised: Known as count based methods Strehl & Littman (2008); Bellemare et al.

Original: (2016), this approach was shown to reliably evade noisy TVs in minigrid environments Raileanu &
Revised: (2016), this approach was shown to reliably evade noisy TVs in minigrid environments Raileanu &

Original: Rocktäschel (2020).
Revised: Rocktäschel (2020).

Original: However, it is not feasible to count state visitations in many environments where there is a large number of unique states Bellemare et al.
Revised: However, it is not feasible to count state visitations in many environments where there is a large number of unique states Bellemare et al.

Original: “Pseudo-count” methods exchange tabular look up tables for density models to estimate an analogous intrinsic reward to counts in large state spaces Bellemare et al.
Revised: “Pseudo-count” methods exchange tabular look up tables for density models to estimate an analogous intrinsic reward to counts in large state spaces Bellemare et al.

Original: Nevertheless, prediction error based methods for exploration remain a key component in state of the art algorithms deployed in high dimensional state spaces (e.g.
Revised: Nevertheless, prediction error based methods for exploration remain a key component in state of the art algorithms deployed in high dimensional state spaces (e.g.

Original: Badia et al.
Revised: Badia et al.

Original: Attempts have been made to reward Epistemic uncertainty directly.
Revised: Attempts have been made to reward Epistemic uncertainty directly.

Original: This typically requires a posterior distribution over model parameters, which is intractable without approximations such as ensembles or variational inference (e.g. Houthooft et al.
Revised: This typically requires a posterior distribution over model parameters, which is intractable without approximations such as ensembles or variational inference (e.g. Houthooft et al.

Original: Osband et al.
Revised: Osband et al.

Original: (2016) instantiated an ensemble
Revised: (2016) instantiated an ensemble

Original: Lakshminarayanan et al.
Revised: Lakshminarayanan et al.

Original: (2017) approach into the ﬁnal layer of a deep Q-network—rewarding its agents for epistemic value uncertainty.
Revised: (2017) approach into the final layer of a deep Q-network—rewarding its agents for epistemic value uncertainty.

Original: Pathak et al.
Revised: Pathak et al.

Original: (2019) use the variance of ensemble predictions being used as intrinsic rewards, while Shyam et al.
Revised: (2019) use the variance of ensemble predictions being used as intrinsic rewards, while Shyam et al.

Original: (2019) reward experience of epistemic uncertainty within an ensemble of environment models.
Revised: (2019) reward experience of epistemic uncertainty within an ensemble of environment models.

Revised: Lastly, some uncertainty estimation methods have recently been developed that enforce a smoothness constraint in the representation space Mukhoti et al.

Revised: (2021); van Amersfoort et al.

Revised: (2021)—allowing for sensible estimations of uncertainty to be made from learned representations—but these approaches have not yet been adopted in reinforcement learning.

Original: All this is to say, uncertainty often plays a role in different formulations of curiosity, which is why we looked at models acetylcholine—a neuromodulator associated with uncertainty—for inspiration in building curiosity models capable of avoiding distractions.
Revised: All in all, uncertainty often plays a role in different formulations of curiosity, which is why we looked at models acetylcholine—a neuromodulator associated with uncertainty—for inspiration in building curiosity models capable of avoiding distractions.

Original: 2. A CETYLCHOLINE
Revised: 2. A CETYLCHOLINE

Original: In the mammalian brain acetylcholine is implicated in a range of processes including learning and memory, fear, novelty detection, and attention Ranganath & Rainer (2003);
Revised: In the mammalian brain acetylcholine is implicated in a range of processes including learning and memory, fear, novelty detection, and attention Ranganath & Rainer (2003);

Original: Pepeu & Giovannini (2004); Acquas et al.
Revised: Pepeu & Giovannini (2004); Acquas et al.

Original: ; Barry et al.
Revised: ; Barry et al.

Original: ; Yu & Dayan (2005); Hasselmo (2006); Giovannini et al.
Revised: ; Yu & Dayan (2005); Hasselmo (2006); Giovannini et al.

Original: ; Parikh et al.
Revised: ; Parikh et al.

Original: Traditional views—supported by the rapid increase in cholinergic tone in response to environmental novelty and demonstrable effects on neural plasticity— emphasised its role as a learning signal, generating physiological changes that favour encoding of new information over retrieval Hasselmo (2006).
Revised: Traditional views—supported by the rapid increase in cholinergic tone in response to environmental novelty and demonstrable effects on neural plasticity— emphasised its role as a learning signal, generating physiological changes that favour encoding of new information over retrieval Hasselmo (2006).

Original: Notably, Yu & Dayan (2003) proposed an alternative perspective, suggesting that acetylcholine signals the expected uncertainty of top down predictions , while modulation of norepinephrine is a result of unexpected uncertainties .
Revised: Notably, Yu & Dayan (2003) proposed an alternative perspective, suggesting that acetylcholine signals the expected uncertainty of top down predictions , while modulation of norepinephrine is a result of unexpected uncertainties .

Original: More concretelty, Yu & Dayan (2003)’s model can be seen as favouring bottom up sensory input over top down predictions if predictions are believed to be inaccurate—consistent with evidence that shows acetylcholine inhibits feedback connections and strengthens sensory inputs Hasselmo (2006).
Revised: More concretelty, Yu & Dayan (2003)’s model can be seen as favouring bottom up sensory input over top down predictions if predictions are believed to be inaccurate—consistent with evidence that shows acetylcholine inhibits feedback connections and strengthens sensory inputs Hasselmo (2006).

Original: However this approach does not explicitly separate epistemic and aleatoric uncertainties Yu & Dayan (2005).
Revised: However this approach does not explicitly separate epistemic and aleatoric uncertainties Yu & Dayan (2005).

Original: In contrast, the utility of quantifying epistemic uncertainties for exploration has been widely recognised in the RL literature (e.g. Osband et al.
Revised: In contrast, the utility of quantifying epistemic uncertainties for exploration has been widely recognised in the RL literature (e.g. Osband et al.

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: Here we demonstrate a potential use of aleatoric uncertainties in exploring agents both biological and artiﬁcial.
Revised: Here we demonstrate a potential use of aleatoric uncertainties in exploring agents both biological and artificial.

Original: Namely, aleatoric uncertainties can be used to divert attention away from unpredictable dynamics when using prediction errors as intrinsic rewards.
Revised: Namely, aleatoric uncertainties can be used to divert attention away from unpredictable dynamics when using prediction errors as intrinsic rewards.

Original: This is similar to a model proposed by Parr & Friston (2017), suggesting acetylcholine may indicate expected uncertainties in top down predictions within an MDP.
Revised: This is similar to a model proposed by Parr & Friston (2017), suggesting acetylcholine may indicate expected uncertainties in top down predictions within an MDP.

Original: In this context we propose an extension to (Yu & Dayan, 2005)’s dichotomy.
Revised: In this context we propose an extension to Yu & Dayan (2005)’s dichotomy.

Original: Speciﬁcally, we suggest that in the mammalian brain, cortical acetylcholine signals expected aleatoric uncertainties, while norepinephrine is modulated by epistemic uncertainties both expected and unexpected.
Revised: Specifically, we suggest that in the mammalian brain, cortical acetylcholine signals expected aleatoric uncertainties, while norepinephrine is modulated by epistemic uncertainties both expected and unexpected.

Original: This formulation is attractive in an ML framework, providing a means to avoid stochastic traps, while also being consistent with empirical biological data Hasselmo (2006); Yu & Dayan (2003).
Revised: This formulation is attractive in an ML framework, providing a means to avoid stochastic traps, while also being consistent with empirical biological data Hasselmo (2006); Yu & Dayan (2003).

Original: Testing this hypothesis calls for experiments to clarify the nature of the uncertainty signalled by acetylcholine, which we present in Section 5.
Revised: Testing this hypothesis calls for experiments to clarify the nature of the uncertainty signalled by acetylcholine, which we present in Section 5.

Original:  M ETHOD3. I NTRINSIC R EWARDS
Revised:  M ETHOD

Original: AMAs operate in the arena of Markov decision processes that consist of states s ∈ S , actions a ∈ A , and rewards r ∈ R ⊂ R Sutton & Barto (2018).
Revised: AMAs operate in the arena of Markov decision processes that consist of states s ∈ S , actions a ∈ A , and rewards r ∈ R ⊂ R Sutton & Barto (2018).

Original: At each timestep t the agent selects an action via a stochastic policy a t ∼ π ( ·| s t ) Szepesvári (2010) and then receives a reward r t +1 and state s t + generated via the transition function p ( s t +1 , r t +1 | s t , a t ) of the environment Sutton & Barto (2018).
Revised: At each timestep t the agent selects an action via a stochastic policy a t ∼ π ( ·| s t ) Szepesvári (2010) and then receives a reward r t +1 and state s t + generated via the transition function p ( s t +1 , r t +1 | s t , a t ) of the environment Sutton & Barto (2018).

Original: The objective of the agent is to learn a stochastic policy π , parametrised by ξ , that generates an action as a function of the current state s t ,actions which aim to maximise the expectation of the sum of discounted future rewards (e.g. Mnih et al.
Revised: The objective of the agent is to learn a stochastic policy π , parametrised by ξ , that generates an action as a function of the current state s t , which aims to maximise the expectation of the sum of discounted future rewards (e.g. Mnih et al.

Original: [Equation]
Revised: [Equation]

Original: Where T is the episode length and γ is the discount factor.
Revised: Where T is the episode length and γ is the discount factor.

Original: Following other curiosity based methods, the total reward is the sum of the intrinsic reward provided by the intrinsic reward module of the agent and extrinsic rewards provided by the environment (e.g. Pathak et al.
Revised: Following other curiosity based methods, the total reward is the sum of the intrinsic reward provided by the intrinsic reward module of the agent and extrinsic rewards provided by the environment (e.g. Pathak et al.

Original: ; Badia et al.
Revised: ; Badia et al.

Original: Raileanu & Rocktäschel (2020); Burda et al. (2018b)).
Revised: Raileanu & Rocktäschel (2020); Burda et al. (2018b)).

Original: [Equation]
Revised: [Equation]

Original: Where the superscripts i and e indicate intrinsic and extrinsic reward, and β is a hyperparameter that regulates the inﬂuence of intrinsic rewards on the policy.
Revised: Where the superscripts i and e indicate intrinsic and extrinsic rewards, and β is a hyperparameter that regulates the influence of intrinsic rewards on the policy.

Original: Our intrinsic rewards are generated by a forward prediction module that we describe in the next section. 3. A VOIDING U NPREDICTABLE E NVIRONMENT D YNAMICS BY Q UANTIFYING A LEATORIC U NCERTAINTY The goal of a curious agent is to select actions that maximise prediction errors. However, it is demonstrably harmful to naively maximise all prediction errors Burda et al.
Revised: In previous works Burda et al.

Original: (2018a). So, what sorts of prediction errors are useful for exploration?
Revised: (2018a), the intrinsic reward r it is equal to the mean squared forward prediction error of a curiosity module.

Original: The active learning literature explains that the future prediction error of a model can be decomposed into 3 different terms: noise, bias and variance Settles (2009).
Revised: To avoid stochastic traps we subtract the aleatoric uncertainty—which is constrained to have a diagonal covariance Kendall & Gal (2017)—from the prediction error, so that agents are not surprised by transitions that were previously learnt to be unpredictable.

Original: [Equation]
Revised: [Equation]

Original: A curious agent should seek out environmental transitions that it could it learn, but has not yet (e.g. Burda et al. (2018b))—so that by experiencing and learning from them it will reduce the total future prediction error. In terms of Equation 3 above, it should seek large values of the variance term on the right hand side.
Revised: Where ˆ µ t +1 is the predicted mean of the next state, ˆ Σ t +1 is the predicted aleatoric uncertainty of the next state and η is a hyperparameter that regulates by how much the predicted uncertainty of the next state effects intrinsic rewards.

Original: The difﬁculty is that it observes the left hand side of the equation, and that can be large when the variance is small if the noise or bias is large.
Revised: The states being predicted are usually image observations but AMAs can also work with learned representations of state observations as we show in Section 4.4.

Original: To avoid stochastic transitions we explicitly model a quantity analogous to the noise term in Equation 3 as heteroscedastic aleatoric uncertainty. Heteroscedastic refers to the fact that the predicted uncertainty is a function of the model input. To compute heteroscedastic aleatoric uncertainty we follow Kendall & Gal (2017), who offer an objective for regressing both the meanand aleatoric uncertainty ofa deterministic deep learning model. While, Kendall & Gal (2017) use maximum a posteriori inference with a zero-mean Gaussian prior on the network parameters { θ, φ } , we found that the resulting regularisation terms in the cost function did not improve results—possible because regularisation terms could absorb some epistemic uncertainty.
Revised: To learn to predict the mean of the next state ˆ µ t

Original: Accordingly, we take a simpler approach and use maximum likelihood estimation.
Revised: +1 and its aleatoric uncertainty ˆ Σ t +1 , we follow

Original: Assuming that the data is Gaussian distributed we can deﬁne the following function for the likelihood Kendall & Gal (2017) ; Bishop [Equation] [Equation]
Revised: Kendall & Gal (2017)—fitting a diagonal covariance Gaussian distribution to the elements of the next state.

Original: Where f θ are g φ are the neural networks used for forward prediction, θ are the network parameters for the mean prediction  and φ are the parameters used for the aleatoric uncertainty prediction.
Revised: The predictions are made by a double-headed neural network—with a mean prediction head f parameterised by θ and a variance prediction head g parametrised by ϕ .

Original: Based on Kendall & Gal (2017)’s implementation, the two mean and variance prediction heads  share feature extracting parameters.
Revised: As employed in previous works, the separate heads of the double-headed deep network share feature extracting parameters Kendall & Gal (2017).

Original: The g φ term represents the aleatoric uncertainty  of future states, mapping onto the noise term in Equation 3.
Revised: The prediction network performs heteroscedastic aleatoric uncertainty estimation Kendall & Gal (2017), which in a reinforcement learning context means the prediction heads are conditioned on the current state and action. [Equation] Where N is the total number of states observed during training. While Kendall & Gal (2017) use maximum a posteriori inference with a zero-mean Gaussian prior on the network parameters { θ, ϕ } , we found empirically that the resulting regularisation terms in the cost function did not improve results.

Original: AMA’s predictive model is trained by maximising the likelihood  in
Revised: Accordingly, we simply perform maximum likelihood estimation with the likelihood presented in

Original: Equation 3 with respect to θ and φ , resulting in the following objective Kendall & Gal (2017).
Revised: Equation (4), resulting in the following cost function Kendall & Gal (2017).

Original: [Equation]
Revised: [Equation]

Original: To reiterate, both the estimated log variance log ˆ σ 2 s t +1 (predicted by g φ ) and the estimated mean (predicted by f θ ) of the labels ˆ µ 2 s t +1 are predicted by the two separate heads of the forward prediction curiosity network Kendall & Gal (2017).

Original: The ﬁrst term is the familiar mean squared error divided by thelog uncertainty .
Revised: The first term is the familiar mean squared error divided by the uncertainty .

Original: The second term blocks the explosion of predicted aleatoric uncertainties Kendall & Gal (2017).
Revised: The second term blocks the explosion of predicted aleatoric uncertainties Kendall & Gal (2017).

Original: We follow Kendall & Gal (2017)’s prescription of estimating log σ 2 instead of σ 2 to ensure stable optimisation.
Revised: We follow Kendall & Gal (2017)’s prescription of estimating log Σ instead of Σ to ensure stable optimisation.

Original: Furthermore, the hyperparameter λ was added tempering the model’s aleatoric uncertainty ‘budget’ (e.g. Depeweg et al.
Revised: Furthermore, the hyperparameter λ was added tempering the model’s aleatoric uncertainty budget (e.g. Depeweg et al.

Original: ; Clements et al.
Revised: ; Clements et al.

Original: ; Eriksson & Dimitrakakis (2019)).
Revised: ; Eriksson & Dimitrakakis (2019)).

Original: The forward prediction module should reward the agent for forward prediction errors unless the prediction error is impossible to reduce due to random dynamics Burda et al.

Original: (2018b). Relying on this intuition, we craft the following intrinsic reward function for a given ( s t +1 , ˆ s t +1 ) tuple—in a similar manner to Clements et al. [Equation] Where D is the dimensionality of the state observations andη is a hyperparameter controlling the weighting of aleatoric uncertainties.
Revised: We use the predicted mean and aleatoric uncertianty of the next state—which are being learned online with Equation (5)—to compute intrinsic rewards according to Equation (3).

Original: Note that theforward prediction model and the policy networkdo not share parameters. The reward function provided by Equation 6 is non-stationary as stochastic gradient descent of the forward prediction module is performed online according to Equation 5.
Revised: Lastly, we would like to highlight that the policy network is separate to the state prediction network as in other curiosity based methods Pathak et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section a range experiments with noisy TVs.
Revised: This section a range experiments with noisy TVs

Original: Extra details such as the hyperparameters and architectures used are in the appendix.

Original: Shaded regions are standard error of the mean.

Original: 4. N OISY MNIST
Revised: 4. N OISY MNIST

Original: First we completed a supervised learning task, similar to the noisy MNIST environment introduced by Pathak et al.
Revised: First we completed a supervised learning task, similar to the noisy MNIST environment introduced by Pathak et al.

Original: The environment does not elicit any actions from an agent.
Revised: The environment does not elicit any actions from an agent.

Original: Instead, the prediction network simply needs to learn one step mappings between pairs of MNIST handwritten digits.
Revised: Instead, the prediction network simply needs to learn one step mappings between pairs of MNIST handwritten digits.

Original: The ﬁrst images in the pairs are randomly selected 0s or 1s.
Revised: The first images in the pairs are randomly selected 0s or 1s.

Original: When the ﬁrst image is a 0 then the second image is the exact same image (these are the deterministic transitions).
Revised: When the first image is a 0 then the second image is the exact same image (these are the deterministic transitions).

Original: When the ﬁrst image is a 1, then the second image is a random digit from 2-9 (these are the stochastic transitions).
Revised: When the first image is a 1, then the second image is a random digit from 2-9 (these are the stochastic transitions).

Original: A prediction model capable of avoiding noisy TVs should eventually learn to compute equal intrinsic rewards for both types of transitions Pathak et al.
Revised: A prediction model capable of avoiding noisy TVs should eventually learn to compute equal intrinsic rewards for both types of transitions Pathak et al.

Original: We trained two different neural networks  on this task (adapted from Liao (2020)), one with a mean squared error (MSE) loss function—as a baseline— and the other with the AMA loss function Kendall & Gal (2017).
Revised: We trained two different neural networks AMA’s on this task (dapted from Liao (2020)), one with a mean squared error (MSE) loss function—as a baseline— and the other with the AMA loss function Kendall & Gal (2017).

Original: The networks are equivalent except that the AMA network has two prediction heads.
Revised: The networks are equivalent except that the AMA network has two prediction heads.

Original: Both networks contain skip connection from the input layer to the output layer and were optimised with Adam at a learning rate of 0.001 and a batch size of 32 Kingma & Ba (2015).
Revised: Both networks contain skip connection from the input layer to the output layer and were optimised with Adam at a learning rate of 0.001 and a batch size of 32 Kingma & Ba (2015).

Original: The uncertainty budget hyperparameter λ and the uncertainty weighting hyperparameter η were set to 1 for the AMA network.
Revised: The uncertainty budget hyperparameter λ and the uncertainty weighting hyperparameter η were set to 1 for the AMA network.

Original: The MSE prediction network is unable to reduce prediction errors for the stochastic transitions, causing it to produce much larger intrinsic rewards than the deterministic transitions, consistent with Pathak et al.
Revised: The MSE prediction network is unable to reduce prediction errors for the stochastic transitions, causing it to produce much larger intrinsic rewards than the deterministic transitions, consistent with Pathak et al.

Original: On the other hand, the AMA prediction network is able to cut its losses by attributing high variance to the stochastic transitions, making them just as rewarding as the deterministic transitions.
Revised: On the other hand, the AMA prediction network is able to cut its losses by attributing high variance to the stochastic transitions, making them just as rewarding as the deterministic transitions.

Original: [Figure] AMA and MSE have similar exploration noisy absent outperforming a no-intrinsic-reward a noisy TV is present only AMA curiosity is able to environment
Revised: 4. D EEP L EARNING M INIGRID

Original: Next we test AMA on the Gym MiniGrid environment Chevalier-Boisvert et al.
Revised: Next we test AMA on the Gym MiniGrid environment Chevalier-Boisvert et al.

Original: (2018), which allows for resource-limited deep reinforcement learning.
Revised: (2018), which allows for resource-limited deep reinforcement learning.

Original: The agent receives tensor observations describing its receptive ﬁeld at each timestep.
Revised: The agent receives tensor observations describing its receptive field at each timestep.

Original: The channels of the observations represent semantic features (e.g. blue door, grey wall, empty, etc.) of each grid tile.
Revised: The channels of the observations represent semantic features (e.g. blue door, grey wall, empty, etc.) of each grid tile.

Original: The action space is discrete (containing actions: turn left, turn right, move forward, pick up, drop, toggle objects and done) allowing the agent to move around the environment as wells well as open and close doors.
Revised: The action space is discrete (containing actions: turn left, turn right, move forward, pick up, drop, toggle objects and done) allowing the agent to move around the environment as wells well as open and close doors.

Original: We used singleton environments but with different seeds for each run—resulting in different room conﬁgurations—to generate the standard error regions for Figure 2.
Revised: We used singleton environments but with different seeds for each run—resulting in different room configurations—to generate the standard error regions for Figure 2.

Original: We measure exploration by counting the number of unique states visited throughout training.
Revised: We measure exploration by counting the number of unique states visited throughout training.

Original: An action-dependent noisy TV was added, inspired by other minigrid experiments with noisy TVs Raileanu & Rocktäschel (2020), by setting approximately half of the state observation to uniformly sampled integers within the range of possible minigrid values.
Revised: An action-dependent noisy TV was added, inspired by other minigrid experiments with noisy TVs Raileanu & Rocktäschel (2020), by setting approximately half of the state observation to uniformly sampled integers within the range of possible minigrid values.

Original: When the agent selects the ‘done’ action the noisy TV is activated in the next observation.
Revised: When the agent selects the ‘done’ action the noisy TV is activated in the next observation.

Original: This is the only effect of the ‘done’ action.
Revised: This is the only effect of the ‘done’ action.

Revised: AMA and MSE have similar exploration performance when the noisy TV is absent, outperforming a no-intrinsic-reward baseline—but when a noisy TV is present only the AMA curiosity approach is able to significantly explore the environment.

Revised: (e shows an example six room environment.

Original: We perform policy optimisation with a synchronous advantage actor critic (A2C) implementation recommended by the gym minigrid README Mnih et al.
Revised: We perform policy optimisation with a synchronous advantage actor critic (A2C) implementation recommended by the gym minigrid README Mnih et al.

Original: ; Willems (2020).
Revised: ; Willems (2020).

Original: For the minigrid experiments we train on intrinsic and extrinsic rewards with their relative weighting being equal.
Revised: For the minigrid experiments we train on intrinsic and extrinsic rewards with their relative weighting being equal.

Original: The actor critic weights were optimised with RMSProp Tieleman & Hinton (2014) at a learning rate of 0.001, while the intrinsic reward module was optimised with the Adam optimizer at a learning rate of 0.001 for the AMA agent and 0.0001 for the MSE agent Kingma & Ba (2015).
Revised: The actor critic weights were optimised with RMSProp Tieleman & Hinton (2014) at a learning rate of 0.001, while the intrinsic reward module was optimised with the Adam optimizer at a learning rate of 0.001 for the AMA agent and 0.0001 for the MSE agent Kingma & Ba (2015).

Original: Hyperparameters were optimised for overall performance with and without the noisy TV, see appendix for details.
Revised: Hyperparameters were optimised for overall performance with and without the noisy TV, see appendix for details.

Original: The forward prediction module works in the observation space as opposed to a learned feature space as is implemented in other curiosity driven methods Pathak et al.
Revised: The forward prediction module works in the observation space as opposed to a learned feature space as is implemented in other curiosity driven methods Pathak et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: (2018b)
Revised: (2018b)

Original: ; Raileanu &
Revised: ; Raileanu &

Original: Rocktäschel (2020).
Revised: Rocktäschel (2020).

Original: Pixel based curiosity was chosen due to its simplicity.
Revised: Pixel based curiosity was chosen due to its simplicity.

Original: The forward prediction model is a double headed CNN, which is inspired by previous intrinsic motivation works on minigrid Raileanu & Rocktäschel (2020).
Revised: The forward prediction model is a double headed CNN, which builds upon a previous intrinsic motivation implementation on minigrid Raileanu & Rocktäschel (2020).

Original: The uncertainty budget λ of the AMA network was set to 0.1, the uncertainty weighting η was set to 1 and intrinsic rewards were clipped below zero.
Revised: The uncertainty budget λ of the AMA network was set to 0.1, the uncertainty weighting η was set to 1 and intrinsic rewards were clipped below zero.

Original: We perform experiments in four and six room conﬁgurations of the environment.
Revised: We perform experiments in four and six room configurations of the environment.

Original: Without a noisy
Revised: Without a noisy

Original: TV both AMA and MSE reward functions generate visit more states compared to the no intrinsic reward baseline.
Revised: TV both AMA and MSE reward functions generate visit more states compared to the no intrinsic reward baseline.

Original: On the other hand, the presence of a noisy TV profoundly affects the performance of the MSE curiosity agent, greatly reducing the number of states visited.
Revised: On the other hand, the presence of a noisy TV profoundly affects the performance of the MSE curiosity agent, greatly reducing the number of states visited.

Original: In contrast, AMA agents are almost unaffected by the presence of an action dependent noisy TV.
Revised: In contrast, AMA agents are almost unaffected by the presence of an action dependent noisy TV.

Original: 4. M ARIO AND S PACE I NVADERS
Revised: 4. M ARIO AND S PACE I NVADERS

Original: Next we examine whether our ﬁndings from the MNIST and minigrid experiments scale to games from Atari Bellemare et al.
Revised: Next we examine whether our findings from the MNIST and minigrid experiments scale to games from Atari Bellemare et al.

Original: and Gym Retro Nichol et al.
Revised: and Gym Retro Nichol et al.

Original: ; Kauten (2018).
Revised: ; Kauten (2018).

Original: We adapt the proximal policy optimisation (PPO)
Revised: We adapt the proximal policy optimisation (PPO)

Original: Schulman et al.
Revised: Schulman et al.

Original: (2017) curiosity implementation from Burda et al.
Revised: (2017) curiosity implementation from Burda et al.

Original: (2018a), using pixel based in our AMA system.
Revised: (2018a), using pixel based in our AMA system.

Original: We extend Burda et al.
Revised: We extend Burda et al.

Original: (2018a)’s
Revised: (2018a)’s

Original: U-Net Ronneberger et al.
Revised: U-Net Ronneberger et al.

Original: (2015) to use two output heads to predict the mean and variance of future states.
Revised: (2015) to use two output heads to predict the mean and variance of future states.

Revised: For our baselines we use official implementations from Burda et al.

Revised: (2018b) and Burda et al.

Revised: (2018a).

Original: We leave all PPO hyperparameters equal to their values from Burda et al.
Revised: We leave all PPO hyperparameters equal to their values from Burda et al.

Original: (2018a).
Revised: (2018a).

Original: We set the uncertainty budget hyperparameter λ to 1 and the uncertainty weighting hyperparameter η to 2.
Revised: We set the uncertainty budget hyperparameter λ to 1 and the uncertainty weighting hyperparameter η to 2.

Original: For the these experiments we do not clip intrinsic rewards.
Revised: For the these experiments we do not clip intrinsic rewards.

Original: To simulate the noisy TV thought experiment, we extend the environment’s action space with an action that induces grayscale tiled CIFAR-10 Krizhevsky et al.
Revised: To simulate the noisy TV thought experiment, we extend the environment’s action space with an action that induces grayscale tiled CIFAR-10 Krizhevsky et al.

Original: (2014) images (examples in appendix) in place ofgame frames for thenext observation and sends the ultimate action of the action space to the game emulator.
Revised: (2014) images (examples in appendix) in place of for the and sends the ultimate action of the action space to the game emulator.

Original: The choice of the ultimate action is arbitrary.
Revised: The choice of the ultimate action is arbitrary.

Original: In the appendix we also demonstrate robustness (with the same hyperparameters) to uniform noise.
Revised: In the appendix we also demonstrate robustness (with the same hyperparameters) to uniform noise.

Original: We compare AMA to three alternative intrinsic reward methods: random network distillation (RND)
Revised: We compare AMA to three alternative intrinsic reward methods: random network distillation (RND)

Original: Burda et al.
Revised: Burda et al.

Original: (2018b), inverse dynamics feature (IDF) curiosity Pathak et al.
Revised: (2018b), inverse dynamics feature (IDF) curiosity Pathak et al.

Original: and MSE pixel based curiosity Burda et al.
Revised: and MSE pixel based curiosity Burda et al.

Original: (2018a).
Revised: (2018a).

Original: Policy optimisation is done with intrinsic rewards only but we measure performance with extrinsic rewards following Burda et al.
Revised: Policy optimisation is done with intrinsic rewards only but we measure performance with extrinsic rewards following Burda et al.

Original: (2018a).
Revised: (2018a).

Original: Compared to the relatively weak baseline of pixel based curiosity, the Space Invaders and Mario experiments show similar results to the minigrid experiments—MSE and AMA pixel based curiosity have comparable performance when no noisy TV is present (Figure 3(b) and 3 (d)), while with the noisy TV AMA greatly outperforms MSE pixel based curiosity (Figure 3(a) and Figure 3(c)).
Revised: Compared to the relatively weak baseline of pixel based curiosity, the Space Invaders and Mario experiments show similar results to the minigrid experiments—MSE and AMA pixel based curiosity have comparable performance when no noisy TV is present (Figure 4(b) and 4 (d)), while with the noisy TV AMA greatly outperforms MSE pixel based curiosity (Figure 4(a) and Figure 4(c)).

Original: Unsurprisingly, random network distillation and inverse dynamics curiosity maintain their superiority over pixel based methods without a noisy TV (Figure 3(b) and 3(d)).
Revised: Unsurprisingly, random network distillation and inverse dynamics curiosity maintain their superiority over pixel based methods without a noisy TV (Figure 4(b) and 4(d)).

Original: However, unlike AMA curiosity, both these baselines are vulnerable to action dependent noisy TVs (Figure 3(a) and Figure 3(c)).
Revised: However, unlike AMA curiosity, both these baselines are vulnerable to action dependent noisy TVs (Figure 4(a) and Figure 4(c)).

Original: While RND has previously been shown to be able to evade stochastic traps Burda et al.
Revised: While RND has previously been shown to be able to evade stochastic traps Burda et al.

Original: (2018b), it seems that if the number of novel states in the trap is large then RND remains vulnerable to trapping.
Revised: (2018b), it seems that if the number of novel states in the trap is large then RND remains vulnerable to trapping.

Original: 4. B ANK H EIST : A N A TARI G AME WITH A N ATURAL S TOCHASTIC T RAP
Revised: 4. B ANK H EIST : A N A TARI G AME WITH A N ATURAL S TOCHASTIC T RAP

Original: Although Atari games are famously deterministic Machado et al.
Revised: Although Atari games are famously deterministic Machado et al.

Original: , we identiﬁed a naturally ocurring stochastic trap in the Bank Heist gameplay videos of the original IDF curiosity paper.
Revised: (2018), we identified a naturally ocurring stochastic trap in the Bank Heist gameplay videos of the original IDF curiosity paper.

Original: The objective of Bank Heist is to simultaneously avoid police cars and navigate to banks distributed across four 2D mazes.
Revised: The objective of Bank Heist is to simultaneously avoid police cars and navigate to banks distributed across four 2D mazes.

Original: The mazes can be entered and exited through the sides of the screen.
Revised: The mazes can be entered and exited through the sides of the screen.

Original: With each enter/exit the bank locations reset stochastically.
Revised: With each enter/exit the bank locations reset stochastically.

Original: Additionally, the getaway car has the option to drop dynamite in order to destroy police cars.
Revised: Additionally, the getaway car has the option to drop dynamite in order to destroy police cars.

Original: avoids the natural trap in Bank Heist but actually explores more of the maze on average than the IDF MSE.

Original: The stochastic trap coincidentally generates large extrinsic rewards—even though the agent only uses intrinsic rewards.

Original: When trained on purely intrinsic rewards, IDF curiosity will perpetually enter and exit the maze while also dropping dynamite.
Revised: When trained on purely intrinsic rewards, IDF curiosity will perpetually enter and exit the maze while also dropping dynamite.

Original: This creates high prediction error as it is impossible to predict when the dynamite will explode and where the banks will regenerate.
Revised: This creates high prediction error as it is impossible to predict when the dynamite will explode and where the banks will regenerate.

Original: Coincidentally, an agent stuck in this stochastic trap achieves a high extrinsic reward—resetting the bank positions causes banks to occasionally regenerate on top of the getaway car increasing game score but is it actually exploring?
Revised: Coincidentally, an agent stuck in this stochastic trap achieves a high extrinsic reward—resetting the bank positions causes banks to occasionally regenerate on top of the getaway car increasing game score but is it actually exploring?

Original: We measured the average number of maze pixels crossed by the agent’s car in an episode.
Revised: We measured the average number of maze pixels crossed by the agent’s car in an episode.

Original: In doing so, we see that as extrinsic rewards rise—i.e the point at which the agent begins to regenerate the banks—the average coverage of the maze sharply decreases.
Revised: In doing so, we see that as extrinsic rewards rise—i.e the point at which the agent begins to regenerate the banks—the average coverage of the maze sharply decreases.

Original: Next we integrated the AMA prediction paradigm into the IDF approach, predicting the mean and aleatoric variance of future state representations and computing rewards with Equation 6 with AMA hyperparameters λ and η set to 1.
Revised: Next we integrated the AMA prediction paradigm into the IDF approach, predicting the mean and aleatoric variance of future state representations and computing rewards with Equation (3) with AMA hyperparameters λ and η set to 1.

Original: With the AMA approach the car visited more positions in the maze as it not susceptible to the stochastic trap but performed poorly in terms of extrinsic rewards as more exploration does not necessarily correlate with better extrinsic reward.
Revised: With the AMA approach the car visited more positions in the maze as it not susceptible to the stochastic trap but performed poorly in terms of extrinsic rewards as more exploration does not necessarily correlate with better extrinsic reward.

Original: This section demonstrates that noisy TVs can be an inherent issue even in environments with simple dynamics.
Revised: This section demonstrates that noisy TVs can be an inherent issue even in environments with simple dynamics.

Original: Further, the results highlight the pitfalls of measuring exploration performance with extrinsic reward (which the agent is not trained to optimise)—although we believe extrinsic reward correlates well with coverage for Space Invaders and Mario.
Revised: Further, the results highlight the pitfalls of measuring exploration performance with extrinsic reward (which the agent is not trained to optimise)—although we believe extrinsic reward correlates well with coverage for Space Invaders and Mario.

Original:  A P ROPOSED T EST FOR THE A LEATORIC M ODEL OF A CETYLCHOLINE
Revised:  A P ROPOSED T EST FOR THE A LEATORIC M ODEL OF A CETYLCHOLINE

Original: Inspired by Yu & Dayan (2005), we propose that in the mammalian brain acetylcholine signals aleatoric uncertainty surrounding future states.
Revised: Inspired by Yu & Dayan (2005), we propose that in the mammalian brain acetylcholine signals aleatoric uncertainty surrounding future states.

Revised: However, we are not aware of any experimental neuroscience data that elucidates the specific nature of the uncertainty signalled by acetylcholine.

Original: To make these predictions concrete for experimental neuroscientists, this section proposes a 1D rodent VR experiment blueprint designed to test the speciﬁc nature of cholinergic uncertainty signalling in the mammalian brain.
Revised: As a result, this section proposes a 1D rodent VR task designed to test the specific nature of cholinergic uncertainty signalling in the mammalian brain which we hope will be picked up by experimental neuroscientists.

Original: To supplement our experimental plan, we compute theoretical predictions of cholinergic activity within either an aleatoric or epistemic acetylcholine model—two competing interpretations of Yu & Dayan (2005)’s model.
Revised: To supplement our experimental proposal, we compute theoretical predictions of cholinergic activity within either an aleatoric or epistemic acetylcholine model—two competing interpretations of Yu & Dayan (2005)’s work.

Revised: The aleatoric model uses aleatoric uncertainties as a theoretical acetylcholine signal Kendall & Gal (2017), while the epistemic model uses ensemble variance as an acetylcholine signal Pathak et al.

Original: The proposed task places an animal in a VR corridor containing a series of spatial landmarks and two reward zones in which it must respond in order to have a chance of receiving a reward.
Revised: The proposed task places an animal in a VR corridor containing a series of spatial landmarks and two reward zones in which it must respond in order to have a chance of receiving a reward.

Original: Responding in reward zone A causes the animal to teleport to a random position along the track.
Revised: Responding in reward zone A causes the animal to teleport to a random position along the track.

Original: Responding in reward zone B causes the animal to teleport to a ﬁxed position on the track.
Revised: Responding in reward zone B causes the animal to teleport to a fixed position on the track.

Original: To compute how both models predict the cholinergic signal should respond in the proposed rodent VR experiment, we simulate the task with a simple multi-armed bandit environment.
Revised: To compute how both models predict the cholinergic signal should respond in the proposed rodent VR experiment, we simulate the task with a simple multi-armed bandit environment.

Original: In our bandit model of the task an agent predicts a 1D function by sampling minibatches from different regions of the input.
Revised: In our bandit model of the task an agent predicts a 1D function by sampling minibatches from different regions of the input.

Original: In one region of sample space the function takes a simple sinuisoidal form, analogous to zone A of the VR track, in a second region the function consists of points randomly sampled from a standard normal distribution at each timestep, analogous to zone B (Figure 5(c)).
Revised: In one region of sample space the function takes a simple sinuisoidal form, analogous to zone A of the VR track, in a second region the function consists of points randomly sampled from a standard normal distribution at each timestep, analogous to zone B (Figure 5(c)).

Original: We applied two models to this task, in the ﬁrst acetycholine was identiﬁed with aleatoric uncertainty, while in the second—as a control—acetylcholine tracks epistemic uncertainty.
Revised: As described previously, we applied two models to this task, in the first acetycholine was identified with aleatoric uncertainty, while in the second—as a control—acetylcholine tracks epistemic uncertainty.

Original: We train an action value based multi-armed bandit to maximise intrinsic rewards for two kinds of forward prediction models: a double headed network trained to optimise the AMA objective and an ensemble of networks where each member is minimising their own MSE (e.g. Pathak et al.
Revised: We trained an action value based multi-armed bandit to maximise intrinsic rewards for two kinds of forward prediction models: a double headed network trained to optimise the AMA objective and an ensemble of networks where each member is minimising their own MSE (e.g. Pathak et al.

Revised: The aleatoric model uses the AMA reward function whereas the epistemic model is (intrinsically) rewarded for variance in ensemble predictions.

Revised: We plot both models’ uncertainties in each reward zone over time—recovering a clear prediction of cholinergic activity in both cases.

Revised: The aleatoric uncertainty of AMA remains high in reward zone B but decreases in reward zone A.

Revised: On the other hand, the epistemic model shows a decrease in acetylcholine in both reward zones over time.

Revised: We hope these clear and distinct predictions on the nature of cholinergic uncertainty signalling will be tested by the experimental neuroscience community in a task similar to the one we propose.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: The AMA reward function implicitly rewards epistemic uncertainty by assuming the total uncertainty can be decomposed into epistemic and aleatoric uncertainties.
Revised: The AMA reward function implicitly rewards epistemic uncertainty by assuming the total uncertainty can be decomposed into epistemic and aleatoric uncertainties.

Original: While theoretically true Kendall & Gal (2017), there is no guarantee that AMAs are able to surgically subtract those errors due to aleatoric dynamics from the total prediction error.
Revised: While theoretically true Kendall & Gal (2017), there is no guarantee that AMAs are able to surgically subtract those errors due to aleatoric dynamics from the total prediction error.

Original: Additionally, aleatoric uncertainty estimates are not guaranteed to be reliable for out of distribution data, meaning intrinsic rewards could become less reliable the further the agent travels into novel territory Mukhoti et al.
Revised: Additionally, aleatoric uncertainty estimates are not guaranteed to be reliable for out of distribution data, meaning intrinsic rewards could become less reliable the further the agent travels into novel territory Mukhoti et al.

Original: In practice we ﬁnd that a stochastic policy—and clipping in the case of minigrid—offsets potentially deceptive intrinsic rewards.
Revised: In practice we find that a stochastic policy—and clipping in the case of minigrid—offsets potentially deceptive intrinsic rewards.

Original: Furthermore, trust region methods in the retro games (i e. using PPO Schulman et al.
Revised: Furthermore, trust region methods in the retro games (i.e. using PPO Schulman et al.

Original: (2017) instead of A2C Mnih et al.
Revised: (2017) instead of A2C Mnih et al.

Original: (2016)) may also compensate for ocassionally deceptive rewards—suggested by the fact that intrinsic reward clipping was not necessary for the retro game experiments.
Revised: (2016)) may also compensate for ocassionally deceptive rewards—suggested by the fact that intrinsic reward clipping was not necessary for the retro game experiments.

Original: Finally, we note that like all curiosity approaches (e.g. Burda et al.
Revised: Finally, we note that like all curiosity approaches (e.g. Burda et al.

Original: (2018a))
Revised: (2018a))

Original: , our method generates non-stationary rewards, which is known to make learning difﬁcult for RL agents.
Revised: , our method generates non-stationary rewards, which is known to make learning difficult for RL agents.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown AMAs are able to avoid action-dependent stochastic traps that destroy the exploration capabilities of conventional curiosity driven agents Burda et al.
Revised: We have shown AMAs are able to avoid action-dependent stochastic traps that destroy the exploration capabilities of conventional curiosity driven agents Burda et al.

Original: (2018a).
Revised: (2018a).

Original: AMAs tractably avoid stochastic traps by decreasing intrinsic rewards in regions with high estimated aleatory.
Revised: AMAs tractably avoid stochastic traps by decreasing intrinsic rewards in regions with high estimated aleatory.

Original: The success of AMAs further cements a proposal from theoretical neuroscience Yu & Dayan (2005) that experimentalists should complete experiments to clarify the nature of uncertainty signalled by acetylcholine.
Revised: The success of AMAs further cements a proposal from theoretical neuroscience Yu & Dayan (2005) that experimentalists should complete experiments to clarify the nature of uncertainty signalled by acetylcholine.

Original: Accordingly, we present an experimental design along with theoretical predictions of cholinergic activities from competing models.
Revised: Accordingly, we present an experimental design along with theoretical predictions of cholinergic activities from competing models.

Original: Completing the animal experiment proposed has the potential to beneﬁt neuroscience and reinforcement learning alike Hassabis et al.
Revised: Completing the animal experiment proposed has the potential to benefit neuroscience and reinforcement learning alike Hassabis et al.

Original: Future work could also analyse an alternative hypothesis that novelty based methods such as random network distillation could be an accurate model of acetylcholine Barry et al.
Revised: Future work could also analyse an alternative hypothesis that novelty based methods such as random network distillation could be an accurate model of acetylcholine Barry et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: (2018b).
Revised: (2018b).

Original: Future RL research should aim to integrate the AMA approach into curiosity methods that operate in feature spaces besides pixels or even within those methods that circumvent dynamics altogether (e.g. Burda et al.
Revised: Future RL research should aim to integrate the AMA approach into curiosity methods that operate in feature spaces besides pixels or even within those methods that circumvent dynamics altogether (e.g. Burda et al.

Original: (2018b))
Revised: (2018b))

Original: , with the aim of achieving state of the exploration even when noisy TVs are present.
Revised: , with the aim of achieving state of the exploration even when noisy TVs are present.

Original:  A PPENDIX 8. N OISY MNIST
Revised:  A PPENDIX 8. N OISY MNIST

Original: We use three random seeds for the repeats of the MNIST experiments.
Revised: We use three random seeds for the repeats of the MNIST experiments.

Original: The results in the graph show test set performance.
Revised: The results in the graph show test set performance.

Original: The hyperparameters used are listed below.
Revised: The hyperparameters used are listed below.

Original: The learning rate was manually tuned so that the identity transformation was learned for the deterministic transitions (hence very low loss for the MSE and AMA network) and the AMA network produced sensible uncertainty estimates for the stochastic transitions.
Revised: The learning rate was manually tuned so that the identity transformation was learned for the deterministic transitions (hence very low loss for the MSE and AMA network) and the AMA network produced sensible uncertainty estimates for the stochastic transitions.

Original: [Table]
Revised: [Table]

Original: 8. M INIGRID
Revised: 8. M INIGRID

Original: We used 5 seeds for the minigrid experiments.
Revised: We used 5 seeds for the minigrid experiments.

Original: We tuned 2 different hyperparameter via grid search: curiosity learning rate ∈ [0 . 01 , 0 .
Revised: We tuned 2 different hyperparameter via grid search: curiosity learning rate ∈ [0 . 01 , 0 .

Original: 001 , 0 . 0001] and intrinsic reward scaling (before normalisation) ∈ [1 , 10 , 100] for the AMA and MSE modules on the six room environment (by summing up the novel states visited with and without a noisy TV).
Revised: 001 , 0 . 0001] and intrinsic reward scaling (before normalisation) ∈ [1 , 10 , 100] for the AMA and MSE modules on the six room environment (by summing up the novel states visited with and without a noisy TV).

Original: We used different seeds for the grid search and the ﬁnal results.
Revised: We used different seeds for the grid search and the final results.

Original: We also adapt an implementation of the Welford algorithm from stack overﬂow for normalising rewards 1 .
Revised: We also adapt an implementation of the Welford algorithm from stack overflow for normalising rewards 1 .

Original: The architecture for forward prediction is adapted from the implementation from [22] but in preliminary experimentation we ended up changing their prediction architecture dramatically. 
Revised: The architecture for forward prediction is adapted from the implementation from Raileanu & Rocktäschel (2020) but in preliminary experimentation we ended up changing their prediction architecture dramatically. [Table] [Figure] [Table]

Original: 8. ATARI
Revised: 8. A

Revised: [Figure]

Original: For each run in the Space Invaders and Mario experiments we used ﬁve seeds per method.
Revised: For each run in the Space Invaders and Mario experiments we used five seeds per method.

Original: For the
Revised: For the

Original: Bank Heist experiments we used three seeds per method.
Revised: Bank Heist experiments we used three seeds per method.

Original: We use the ofﬁcial implementations [3, 9] for the baselines we compare to (with their default hyperparameters).
Revised: We use the official implementations Burda et al. (2018b;a) for the baselines we compare to (with their default hyperparameters).

Original: For AMA and Pixel MSE we adapt from [9].
Revised: For AMA and Pixel MSE we adapt from Burda et al. (2018a).

Original: The hyperparameters used for our AMA experiments can be found in Table 3.
Revised: The hyperparameters used for our AMA experiments can be found in Table 3.

Original: We did not change the PPO/vanilla curiosity hyperparameters from the original implementation we adapted and only changed the AMA hyperparameters.
Revised: We did not change the PPO/vanilla curiosity hyperparameters from the original implementation we adapted and only changed the AMA hyperparameters.

Original: The hyperparameters were chosen by ﬁrst exploring different conﬁgurations on smaller minigrid environments and evaluating promising conﬁgurations on theAtari environments. [Table]
Revised: The hyperparameters were chosen by first exploring different configurations on smaller minigrid environments and evaluating promising configurations on the

Original: [Table] [Table] [Table]
Revised: Atari environments.

Original: The UNet architecture used for the forward predictions is described below.
Revised: The UNet architecture used for the forward predictions is described below.

Original: We duplicate the decoder head to create a two headed output but we only describe the encoder and decoder here.
Revised: We duplicate the decoder head to create a two headed output but we only describe the encoder and decoder here.

Original: For the
Revised: For the

Original: Pixel MSE baselines we use identical architectures but don’t use the uncertainty predictions and train on MSE only.
Revised: Pixel MSE baselines we use identical architectures but don’t use the uncertainty predictions and train on MSE only.

Original: There are UNet style residual connections between the corresponding encoder and  decoder layers.
Revised: There are UNet style residual connections between the corresponding encoder and [Table] [Table] decoder layers.

Original: Leaky ReLU activations are used in the encoder layers and Tanh activations are used in the decoder layers.
Revised: Leaky ReLU activations are used in the encoder layers and Tanh activations are used in the decoder layers.

Original: Batch normalisation is used throughout the hidden layers.
Revised: Batch normalisation is used throughout the hidden layers.

Original: Action information is concatenated at each layer.
Revised: Action information is concatenated at each layer.

Original: See supplementary code for further details.
Revised: See supplementary code for further details.

Original: To integrate AMA into the IDF approach, we did not share any representations between mean and variance prediction heads, instead we used two prediction MLPs for the mean and variance.
Revised: To integrate AMA into the IDF approach, we did not share any representations between mean and variance prediction heads, instead we used two prediction MLPs for the mean and variance.

Original: Leaky
Revised: Leaky

Original: ReLU is used throughout hidden layers and action information is concatenated at each layer.
Revised: ReLU is used throughout hidden layers and action information is concatenated at each layer.

Original: We used ﬁve layers with 512 units each and UNet style residual connections.
Revised: We used five layers with 512 units each and UNet style residual connections.

Original: 8. B ANDIT
Revised: 8. B ANDIT

Original: We performed 3 repeats to produce the standard error regions show in the graph.
Revised: We performed 3 repeats to produce the standard error regions show in the graph.

Original: Learning rate was tuned by hand, observing how well the network performed in making predictions as the bandit sampled different regions of the environment.
Revised: Learning rate was tuned by hand, observing how well the network performed in making predictions as the bandit sampled different regions of the environment.

Original: The intrinsic reward method for the epistemic bandit is based on [7, 24].
Revised: The intrinsic reward method for the epistemic bandit is based on Pathak et al.

Revised: ; Lakshminarayanan et al.

Original: We use an action value based bandit algorithm with (cid:15) -greedy exploration [1, p.32].
Revised: We use an action value based bandit algorithm with ϵ -greedy exploration (Sutton & Barto, 2018, p. 31).

Original: 8. P LOT F ORMATTING
Revised: 8. P LOT F ORMATTING

Original: For the plots we use Weight and Biases 2 including their built in smoothing function.
Revised: For the plots we use Weight and Biases 2 including their built in smoothing function.

Original: For all results we use the exponential moving average (with a smoothing parameter of 0.59) except in the minigrid plots where we used the moving average smoother (with a smoothing parameter of 0.1).
Revised: For all results we use the exponential moving average (with a smoothing parameter of 0.59) except in the minigrid plots where we used the [Table]

Original: 8. D IFFERENT N OISE D ISTRIBUTION FOR R ETRO G AMES
Revised: 8. D IFFERENT N OISE D ISTRIBUTION FOR R ETRO G AMES

Original: For the CIFAR noisy TV we tiled a random CIFAR image (from the training set) for each frame observed on the noisy TV.
Revised: For the CIFAR noisy TV we tiled a random CIFAR image (from the training set) for each frame observed on the noisy TV.

Original: This required around around 2.5 tiles to ﬁll the 84 × 84 pixels of the retro game frames.
Revised: This required around around 2.5 tiles to fill the 84 × 84 pixels of the retro game frames.

Original: An example frame can be seen in Figure 11.
Revised: An example frame can be seen in Figure 11.

Original: 8. H YPERPARAMETER A NALYSIS
Revised: 8. H YPERPARAMETER A NALYSIS

Original: The AMA curiosity method contains two hyperparameters λ and η .
Revised: The AMA curiosity method contains two hyperparameters λ and η .

Original: We analysed on Mario, Space
Revised: We analysed on Mario, Space

Original: Invaders and Minigrid how sensitive performance is to the values of these hyperaparameters.
Revised: Invaders and Minigrid how sensitive performance is to the values of these hyperaparameters.

Original: We ﬁnd that decent results can be achieved with no tuning on the retro game environment (Figure 14 and
Revised: We find that decent results can be achieved with no tuning on the retro game environment (Figure 14 and

Original: Figure 15), while in the six room minigrid environment it is important to tune λ
Revised: Figure 15), while in the six room minigrid environment it is important to tune λ

Original: (Figure 16).
Revised: (Figure 16).

Original: 8. E VENTUAL D ECREASE IN E XPLORATION D URING M ARIO T RAINING
Revised: 8. E VENTUAL D ECREASE IN E XPLORATION D URING M ARIO T RAINING

Original: In initial experiments we noticed the AMA agent lost motivation to explore its environment after reaching it peak extrinsic reward.
Revised: In initial experiments we noticed the AMA agent lost motivation to explore its environment after reaching it peak extrinsic reward.

Original: To ensure that this was not an inherent problem with AMA, we ran the other curiosity methods for further frames and found similar eventual decreases in extrinsic reward (Figure 16).
Revised: To ensure that this was not an inherent problem with AMA, we ran the other curiosity methods for further frames and found similar eventual decreases in extrinsic reward (Figure 16).

Original: Presumably the cause of this is that once a signiﬁcant portion of the environment has been explored the agent is no longer motivated to return to those regions (as prediction error decreases)—this issue has been noted by previous authors Pathak et al.
Revised: Presumably the cause of this is that once a significant portion of the environment has been explored the agent is no longer motivated to return to those regions (as prediction error decreases)—this issue has been noted by previous authors Pathak et al.

Original: 8. P OTENTIAL N EGATIVE S OCIAL I MPACTS
Revised: 8. P OTENTIAL N EGATIVE S OCIAL I MPACTS

Original: The work presented is very far from any real world deployment.
Revised: The work presented is very far from any real world deployment.

Original: If it were to be deployed in any real world context then extensive testing would need to be done to understand how the curiosity agents would behave in novel environments as erratic behaviours could be dangerous in, for example, a robotic control context.
Revised: If it were to be deployed in any real world context then extensive testing would need to be done to understand how the curiosity agents would behave in novel environments as erratic behaviours could be dangerous in, for example, a robotic control context.

Original: The AMA objective is overarching (like other curiosity methods) and so care should be taken if deploying in something like a recommender system as the agent could ﬁnd certain behaviours intrinsically rewarding that you might not have intended it to (like the noisy TV problem).
Revised: The AMA objective is overarching (like other curiosity methods) and so care should be taken if deploying in something like a recommender system as the agent could find certain behaviours intrinsically rewarding that you might not have intended it to (like the noisy TV problem).

Original: Lastly, although the AMA system contains notions of uncertainty quantiﬁcation, that does not mean it is able to completely understand the limits of its predictions and so one should not be overconﬁdent in its abilities to do so.
Revised: Lastly, although the AMA system contains notions of uncertainty quantification, that does not mean it is able to completely understand the limits of its predictions and so one should not be overconfident in its abilities to do so.

Original: 8. H ARDWARE
Revised: 8. H ARDWARE

Original: The experiments were performed on three different machines depending on their availability: Acore CPU with one GeForce GTX TITAN X, a 12 core CPU with two GeForce GTX TITAN Xs and one 8 core CPU with two GeForce 2080Ti GPUs.
Revised: The experiments were performed on three different machines depending on their availability: Acore CPU with one GeForce GTX TITAN X, a 12 core CPU with two GeForce GTX TITAN Xs and one 8 core CPU with two GeForce 2080Ti GPUs.

Original: We list times here on the 2080Ti machine, the other machines were as much as 2 × slower.
Revised: We list times here on the 2080Ti machine, the other machines were as much as 2 × slower.

Original: The minigrid experiments took around 40 minutes per run, the Space Invader experiments took aroundhours per run, the Mario experiments took around 1 hour and 20 minutes per run.
Revised: The minigrid experiments took around 40 minutes per run, the Space Invader experiments took aroundhours per run, the Mario experiments took around 1 hour and 20 minutes per run.

Original: 8. L ICENSING
Revised: 8. L ICENSING

Original: The repository from [44] has an MIT license.
Revised: The repository from Willems (2020) has an MIT license.

Original: The code used from [43] has an Apache License 2.0.[22] has a creative commons license.
Revised: The code used from Chevalier-Boisvert et al. (2018) has an Apache License 2.0.

Original: Besides those listed we are not of code adapted a free rat bandit
Revised: Raileanu & Rocktäschel (2020)

Original: 8. F URTHER C ODE A CKNOWLEDGEMENTS
Revised: 8. F URTHER C ODE A CKNOWLEDGEMENTS

Original: Although we did not directly use their code we would like to acknowledge the following open source contributions that provided a useful reference when implementing Kendall and Gal’s [10] aleatoric uncertainty estimation algorithms: https://github.com/ShellingFord221/My-implementation-of-What-UncertaintiesDo-We-Need-in-Bayesian-Deep-Learning-for-Computer-Vision https://github.com/pmorerio/dl-uncertainty https://github.com/hmi88/what
Revised: Although we did not directly use their code we would like to acknowledge the following open source contributions that provided a useful reference when implementing Kendall and Gal’s Kendall & Gal (2017) aleatoric uncertainty estimation algorithms: https://github.com/ShellingFord221/My-implementation-of-What-UncertaintiesDo-We-Need-in-Bayesian-Deep-Learning-for-Computer-Vision https://github.com/pmorerio/dl-uncertainty https://github.com/hmi88/what