Original: Exploring through Random Curiosity with General Value Functions
Revised: Exploring through Random Curiosity with General Value Functions

Original: Abstract
Revised: Abstract

Original: Efﬁcient exploration in reinforcement learning is a challenging problem commonly addressed through intrinsic rewards.
Revised: Efficient exploration in reinforcement learning is a challenging problem commonly addressed through intrinsic rewards.

Original: Recent prominent approaches are based on state novelty or variants of artiﬁcial curiosity .
Revised: Recent prominent approaches are based on state novelty or variants of artificial curiosity .

Original: However, directly applying them to partially observable environments can be ineffective and lead to premature dissipation of intrinsic rewards.
Revised: However, directly applying them to partially observable environments can be ineffective and lead to premature dissipation of intrinsic rewards.

Original: Here we propose random curiosity with general value functions (RC-GVF), a novel intrinsic reward function that draws upon connections between these distinct approaches.
Revised: Here we propose random curiosity with general value functions (RC-GVF), a novel intrinsic reward function that draws upon connections between these distinct approaches.

Original: Instead of using only the current observation’s novelty or a curiosity bonus for failing to predict precise environment dynamics, RC-GVF derives intrinsic rewards through predicting temporally extended general value functions.
Revised: Instead of using only the current observation’s novelty or a curiosity bonus for failing to predict precise environment dynamics, RC-GVF derives intrinsic rewards through predicting temporally extended general value functions.

Original: We demonstrate that this improves exploration in a hard-exploration diabolical lock problem.
Revised: We demonstrate that this improves exploration in a hard-exploration diabolical lock problem.

Original: Furthermore, RC-GVF signiﬁcantly outperforms previous methods in the absence of ground-truth episodic counts in the partially observable MiniGrid environments.
Revised: Furthermore, RC-GVF significantly outperforms previous methods in the absence of ground-truth episodic counts in the partially observable MiniGrid environments.

Original: Panoramic observations on MiniGrid further boost RC-GVF’s performance such that it is competitive to baselines exploiting privileged information in form of episodic counts.
Revised: Panoramic observations on MiniGrid further boost RC-GVF’s performance such that it is competitive to baselines exploiting privileged information in form of episodic counts.

Original:  Introduction
Revised:  Introduction

Original: How should an agent efﬁciently explore environments with high-dimensional state spaces and sparse rewards [70, 2]?
Revised: How should an agent efficiently explore environments with high-dimensional state spaces and sparse rewards [70, 2]?

Original: An extrinsic reward-maximising agent will make little progress until it stumbles upon a rewarding sequence of actions.
Revised: An extrinsic reward-maximising agent will make little progress until it stumbles upon a rewarding sequence of actions.

Original: In the literature, this issue is commonly addressed by providing additional intrinsic reward to guide the agent’s exploratory behaviour [49, 12, 43].
Revised: In the literature, this issue is commonly addressed by providing additional intrinsic reward to guide the agent’s exploratory behaviour [49, 12, 43].

Original: One class of prominent approaches rely on state novelty , where an intrinsic reward in the form of a ‘novelty bonus’ is awarded based on how often a state has been visited [65, 6].
Revised: One class of prominent approaches rely on state novelty , where an intrinsic reward in the form of a ‘novelty bonus’ is awarded based on how often a state has been visited [65, 6].

Original: More recent works have extended these approaches to high dimensional state spaces where tabular counts are inapplicable [7, 41, 10].
Revised: More recent works have extended these approaches to high dimensional state spaces where tabular counts are inapplicable [7, 41, 10].

Original: Another class of approaches is based on artiﬁcial curiosity , where agents are rewarded in proportion to the prediction errors or information gains of a predictive world model [52, 60].
Revised: Another class of approaches is based on artificial curiosity , where agents are rewarded in proportion to the prediction errors or information gains of a predictive world model [52, 60].

Original: Curiosity-based techniques have also been scaled up to handle larger state spaces [22, 43].
Revised: Curiosity-based techniques have also been scaled up to handle larger state spaces [22, 43].

Original: Expecting an agent to have access to the complete environment state is unrealistic.
Revised: Expecting an agent to have access to the complete environment state is unrealistic.

Original: In such partially observable settings [3, 26], observations may look alike in different states, and the beneﬁt of approximate state novelty bonuses as intrinsic rewards is unclear.
Revised: In such partially observable settings [3, 26], observations may look alike in different states, and the benefit of approximate state novelty bonuses as intrinsic rewards is unclear.

Original: Consider the environment in
Revised: Consider the environment in

Original: Figure 1 where an agent is required to traverse a series of blue and white tiles, yet it is only able to observe the current tile it is standing on.
Revised: Figure 1 where an agent is required to traverse a series of blue and white tiles, yet it is only able to observe the current tile it is standing on.

Original: The colours of the tiles alternate in a regular fashion until the end of the corridor, where there is a surprising sequence of consecutive blue tiles.
Revised: The colours of the tiles alternate in a regular fashion until the end of the corridor, where there is a surprising sequence of consecutive blue tiles.

Original: Directly applying state novelty approaches may not be particularly meaningful in this case, as can be seen when using random network distillation (RND; [10]).
Revised: Directly applying state novelty approaches may not be particularly meaningful in this case, as can be seen when using random network distillation (RND; [10]).

Original: In RND, the agent receives a novelty bonus based on the error of its predictions about the output of a ﬁxed randomly initialised neural network applied to the current observation.
Revised: In RND, the agent receives a novelty bonus based on the error of its predictions about the output of a fixed randomly initialised neural network applied to the current observation.

Original: As shown in the top panel of Figure 1, RND is unable to ascribe novelty when the state pattern changes later in the sequence and the intrinsic reward has vanished.
Revised: As shown in the top panel of Figure 1, RND is unable to ascribe novelty when the state pattern changes later in the sequence and the intrinsic reward has vanished.

Original: Similarly, intrinsic rewards may also vanish prematurely for certain curiosity-based approaches when single-step dynamics are simple [42], while longer horizon predictions (as needed in the partially observable case) are susceptible to rapidly compounding errors during sequential rollouts [69]. 
Revised: Similarly, intrinsic rewards may also vanish prematurely for certain curiosity-based approaches when single-step dynamics are simple [42], while longer horizon predictions (as needed in the partially observable case) are susceptible to rapidly compounding errors during sequential rollouts [69]. [Figure]

Original: The agent (red triangle) observes the colour of the tile that it currently occupies (white or blue).
Revised: The agent (red triangle) observes the colour of the tile that it currently occupies (white or blue).

Original: The extrinsic reward is always zero.
Revised: The extrinsic reward is always zero.

Original: At each timestep (frame), the agent moves one tile forward, until it reaches the last tile.
Revised: At each timestep (frame), the agent moves one tile forward, until it reaches the last tile.

Original: Unlike our approach (RC-GVF), RND does not generate an intrinsic reward when encountering the surprising last few tiles (step 1000).
Revised: Unlike our approach (RC-GVF), RND does not generate an intrinsic reward when encountering the surprising last few tiles (step 1000).

Original: In this paper, we explore a connection between state novelty and artiﬁcial curiosity to address these limitations.
Revised: In this paper, we explore a connection between state novelty and artificial curiosity to address these limitations.

Original: We propose random curiosity with general value functions (RC-GVF) as a novel approach to generating intrinsic rewards.
Revised: We propose random curiosity with general value functions (RC-GVF) as a novel approach to generating intrinsic rewards.

Original: It takes inspiration from exploration through temporally extended experiments that summarise long observation sequences [53].
Revised: It takes inspiration from exploration through temporally extended experiments that summarise long observation sequences [53].

Original: Our approach uses random general value functions (GVFs)
Revised: Our approach uses random general value functions (GVFs)

Original: [67] to pose questions about the cumulative future value of observation-dependent features.
Revised: [67] to pose questions about the cumulative future value of observation-dependent features.

Original: Speciﬁcally, we train an ensemble of predictors to minimise the temporal difference (TD) errors of the general value functions and derive an intrinsic reward based on the TDerrors and disagreements in the long-term predictions.
Revised: Specifically, we train an ensemble of predictors to minimise the temporal difference (TD) errors of the general value functions and derive an intrinsic reward based on the TDerrors and disagreements in the long-term predictions.

Original: The effective horizon of the prediction task can be controlled via the discount factor, where a zero discount can be related to state novelty with random network distillation (RND) [10].
Revised: The effective horizon of the prediction task can be controlled via the discount factor, where a zero discount can be related to state novelty with random network distillation (RND) [10].

Original: We hypothesise that predicting information from an temporally extended horizon improves exploration in POMDPs and guards against premature vanishing of intrinsic rewards due to the increase in difﬁculty of the auxiliary task.
Revised: We hypothesise that predicting information from an temporally extended horizon improves exploration in POMDPs and guards against premature vanishing of intrinsic rewards due to the increase in difficulty of the auxiliary task.

Original: Indeed, in our toy experiment of Figure 1 it can be seen how RC-GVF manages to generate a spike of intrinsic reward at the end of the sequence.
Revised: Indeed, in our toy experiment of Figure 1 it can be seen how RC-GVF manages to generate a spike of intrinsic reward at the end of the sequence.

Original: We evaluate our approach on sparse-reward and partially observable RL problems.
Revised: We evaluate our approach on sparse-reward and partially observable RL problems.

Original: Our results on a hard exploration diabolical lock problem [36] and the Minigrid suite of environments [13] reveal the beneﬁts of considering extended horizon predictions in comparison to approaches that rely on immediate partial observations to generate exploration bonuses.
Revised: Our results on a hard exploration diabolical lock problem [36] and the Minigrid suite of environments [13] reveal the benefits of considering extended horizon predictions in comparison to approaches that rely on immediate partial observations to generate exploration bonuses.

Original: Furthermore, existing baselines in
Revised: Furthermore, existing baselines in

Original: Minigrid are heavily reliant on privileged information about episodic state-visitation counts from the simulator, which is commonly used to scale the exploration bonus.
Revised: Minigrid are heavily reliant on privileged information about episodic state-visitation counts from the simulator, which is commonly used to scale the exploration bonus.

Original: We demonstrate that RC-GVF succeeds in many environments without the use of such episodic counts.
Revised: We demonstrate that RC-GVF succeeds in many environments without the use of such episodic counts.

Original:  Preliminaries
Revised:  Preliminaries

Original: Reinforcement learning in POMDPs We consider the scenario where an agent interacts with a
Revised: Reinforcement learning in POMDPs We consider the scenario where an agent interacts with a

Original: Partially Observable Markov Decision Process (POMDP) with time steps t 2 N , with environment states S t 2 S , observations 
Revised: Partially Observable Markov Decision Process (POMDP) with time steps t ∈ N , with environment states S t ∈ S , observations O t ∈

Original: Ot 2 O , actions A t 2 A , extrinsic rewards R e : S ! [0 , 1] , and policies that map histories of different lengths to distributions over actions ⇡ : 
Revised: O , actions A t ∈ A , extrinsic rewards R e : S → [0 , 1] , and policies that map histories of different lengths to distributions over actions π : H → ∆ ( A ) where

Original: H!  ( A ) where H t =
Revised: H t =

Original: O 1: t 2 H [26].
Revised: O 1: t ∈ H [26].

Original: The agent only observes O t at time step t , which may not fully describe the
Revised: The agent only observes O t at time step t , which may not fully describe the

Original: MDP state, and thus it becomes necessary to condition the agent on the history H t .
Revised: MDP state, and thus it becomes necessary to condition the agent on the history H t .

Original: The learning objective is to ﬁnd the optimal policy ⇡ ⇤ that maximizes the expected discounted return
Revised: The learning objective is to find the optimal policy π ∗ that maximizes the expected discounted return

Original: [Equation]
Revised: [Equation]

Original: (1) where  2 [0 , 1) is the discount factor and the expectation is over stochastic components in the environment and the policy.
Revised: where γ ∈ [0 , 1) is the discount factor and the expectation is over stochastic components in the environment and the policy.

Original: Random Network Distillation In random network distillation (RND) [10] the agent receives a state novelty reward proportional to the error of making predictions ˆ z ( o t ) where the targets are generated by a randomly initialized neural network Z  : O ! R d .
Revised: Random Network Distillation In random network distillation (RND) [10] the agent receives a state novelty reward proportional to the error of making predictions ˆ z ( o t ) where the targets are generated by a randomly initialized neural network Z ϕ : O → R d .

Original: The RND intrinsic reward for an observation is given by
Revised: The RND intrinsic reward for an observation is given by

Original: [Equation]
Revised: [Equation]

Original: RND-like error signals can be a simple way to obtain effective uncertainty estimates about targets at a given input, which could subsequently be used for exploration in RL or detecting out of distribution samples [14].
Revised: RND-like error signals can be a simple way to obtain effective uncertainty estimates about targets at a given input, which could subsequently be used for exploration in RL or detecting out of distribution samples [14].

Original: Beyond its simplicity, an appealing aspect of using RND is that it allows us to implicitly incorporate prior beliefs about useful rewards through the neural network architecture [38].
Revised: Beyond its simplicity, an appealing aspect of using RND is that it allows us to implicitly incorporate prior beliefs about useful rewards through the neural network architecture [38].

Original: General Value Functions
Revised: General Value Functions

Original: A general value function (GVF) [67] is deﬁned by a policy ⇡ , a cumulant or pseudo-reward function Z : O ! R , and a discount factor  z 2 [0 , 1) .
Revised: A general value function (GVF) [67] is defined by a policy π , a cumulant or pseudo-reward function Z : O → R , and a discount factor γ z ∈ [0 , 1) .

Original: It can be expressed as
Revised: It can be expressed as

Original: [Equation]
Revised: [Equation]

Original: General value functions extend the concept of predicting expected cumulative values to arbitrary signals beyond the reward.
Revised: General value functions extend the concept of predicting expected cumulative values to arbitrary signals beyond the reward.

Original: They can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy ⇡ and discount factor  z .
Revised: They can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy π and discount factor γ z .

Original: Predictions from GVFs have previously been used as features for state representation [47] or to specify auxiliary tasks that assist in shaping representations for the main task [24, 73].
Revised: Predictions from GVFs have previously been used as features for state representation [47] or to specify auxiliary tasks that assist in shaping representations for the main task [24, 73].

Original:  Random Curiosity with General Value Functions
Revised:  Random Curiosity with General Value Functions

Original: In the spirit of artiﬁcial curiosity [53], we are interested in predicting long-term outcomes in the environment under a particular policy as a means for exploration.
Revised: In the spirit of artificial curiosity [53], we are interested in predicting long-term outcomes in the environment under a particular policy as a means for exploration.

Original: Our approach, RC-GVF , rewards an agent for taking actions that lead to higher uncertainty about the future.
Revised: Our approach, RC-GVF , rewards an agent for taking actions that lead to higher uncertainty about the future.

Original: However, instead of predicting the entire sequence of future states, we consider a set of random numeric questions about the environment and capture uncertainty about their outcome using general value functions (GVFs).
Revised: However, instead of predicting the entire sequence of future states, we consider a set of random numeric questions about the environment and capture uncertainty about their outcome using general value functions (GVFs).

Original: 3.1 General value functions to predict the future
Revised: 3.1 General value functions to predict the future

Original: The central piece of RC-GVF is the prediction of temporally extended outcomes in the environment.
Revised: The central piece of RC-GVF is the prediction of temporally extended outcomes in the environment.

Original: This is different from only predicting a feature of the current state (as in state novelty exploration [65]) or dynamics models that predict the entire (sequence of) next observation(s) (as in early versions of artiﬁcial curiosity [52]).
Revised: This is different from only predicting a feature of the current state (as in state novelty exploration [65]) or dynamics models that predict the entire (sequence of) next observation(s) (as in early versions of artificial curiosity [52]).

Original: At time step t , the current observation o t is mapped to a collection of pseudo-rewards z t +1 2 R d which together with the policy and discount factor deﬁne a question that can be asked about the future: “what is the expected discounted cumulative sum of these pseudorewards under a given policy?”.
Revised: At time step t , the current observation o t is mapped to a collection of pseudo-rewards z t +1 ∈ R d which together with the policy and discount factor define a question that can be asked about the future: “what is the expected discounted cumulative sum of these pseudorewards under a given policy?”.

Original: The answer to this question (i.e. the prediction target) is based on outcomes in the environment, as in Equation 3.
Revised: The answer to this question (i.e. the prediction target) is based on outcomes in the environment, as in Equation 3.

Original: In this paper, we investigate whether a set of random pseudo rewards are sufﬁcient to drive exploration in this way.
Revised: In this paper, we investigate whether a set of random pseudo rewards are sufficient to drive exploration in this way.

Original: Previously it was found that random features extracted by a neural network are often sufﬁcient to capture useful properties of the input [45].
Revised: Previously it was found that random features extracted by a neural network are often sufficient to capture useful properties of the input [45].

Original: Similarly, neural network architectures can be used to express prior knowledge about useful features [71, 14].
Revised: Similarly, neural network architectures can be used to express prior knowledge about useful features [71, 14].

Original: To this end, we generate pseudo-rewards from a ﬁxed and randomly initialised neural network Z  : O ! R d with parameters  that maps observations to pseudo-rewards.
Revised: To this end, we generate pseudo-rewards from a fixed and randomly initialised neural network Z ϕ : O → R d with parameters ϕ that maps observations to pseudo-rewards.

Original: This choice also bypasses the difﬁcult problem of discovering meaningful general value functions [73].
Revised: This choice also bypasses the difficult problem of discovering meaningful general value functions [73].

Original: Training the GVF predictors Our exploration mechanism is to reward the agent for taking actions that generate previously unknown outcomes.
Revised: Training the GVF predictors Our exploration mechanism is to reward the agent for taking actions that generate previously unknown outcomes.

Original: To facilitate this, we train a separate (recurrent) neural network–which we call the predictor –to predict these values.
Revised: To facilitate this, we train a separate (recurrent) neural network–which we call the predictor –to predict these values.

Original: Concretely, a predictor ˆ v ⇡ ,z : H ! R d maps histories (of observations, actions, and pseudo rewards) to pseudo-values.
Revised: Concretely, a predictor ˆ v π,z : H → R d maps histories (of observations, actions, and pseudo rewards) to pseudo-values.

Original: The predictor is trained on-policy, implying that the GVFs are evaluated under the current policy.
Revised: The predictor is trained on-policy, implying that the GVFs are evaluated under the current policy.

Original: One motivating factor for this choice is that it couples the prediction task to the current policy, thus creating an incentive to vary behaviour for additional exploration [18].
Revised: One motivating factor for this choice is that it couples the prediction task to the current policy, thus creating an incentive to vary behaviour for additional exploration [18].

Original: We use the (truncated)  -return as the target for the predictor, which can be recursively expressed as
Revised: We use the (truncated) λ -return as the target for the predictor, which can be recursively expressed as

Original: [Equation]
Revised: [Equation]

Original: Here,  z 2 [0 , 1] is the usual parameter that allows balancing the bias-variance trade off by interpolating between TD(0) and Monte Carlo estimates of the pseudo-return [64].
Revised: Here, λ z ∈ [0 , 1] is the usual parameter that allows balancing the bias-variance trade off by interpolating between TD(0) and Monte Carlo estimates of the pseudo-return [64].

Original: The predictor is trained to minimise the mean squared TD-error with the  -return target.
Revised: The predictor is trained to minimise the mean squared TD-error with the λ -return target.

Original: For convenience in notation, we will denote G zt (  z ) as G zt , despite its dependence on  z .
Revised: For convenience in notation, we will denote G zt ( λ z ) as G zt , despite its dependence on λ z .

Original: 3.2 Disagreement and prediction error as intrinsic reward
Revised: 3.2 Disagreement and prediction error as intrinsic reward

Original: To generate an intrinsic reward, a straightforward choice could be to consider the error between the temporal difference target (from Equation 4) and the predictor’s output at the current observation:
Revised: To generate an intrinsic reward, a straightforward choice could be to consider the error between the temporal difference target (from Equation 4) and the predictor’s output at the current observation:

Original: [Equation]
Revised: [Equation]

Original: However, the presence of aleatoric uncertainty is a problem that arises as a consequence of extending the horizon of the predictive task.
Revised: However, the presence of aleatoric uncertainty is a problem that arises as a consequence of extending the horizon of the predictive task.

Original: From an exploration perspective, the agent should focus on the reducible epistemic uncertainty, and not the irreducible aleatoric uncertainty.
Revised: From an exploration perspective, the agent should focus on the reducible epistemic uncertainty, and not the irreducible aleatoric uncertainty.

Original: Directly minimising
Revised: Directly minimising

Original: Equation 5 with the predictor would ignore the inherent variance in the TD-target (due to stochasticity in the policy and environment), and thus using the prediction error of the GVF target as an intrinsic reward does not distinguish between aleatoric and epistemic uncertainty [15, 23].
Revised: Equation 5 with the predictor would ignore the inherent variance in the TD-target (due to stochasticity in the policy and environment), and thus using the prediction error of the GVF target as an intrinsic reward does not distinguish between aleatoric and epistemic uncertainty [15, 23].

Original: Another way to recognize this is by decomposing the expected prediction error as described by
Revised: Another way to recognize this is by decomposing the expected prediction error as described by

Original: Jain et al.
Revised: Jain et al.

Original: The expected loss of a predictor ˆ f ( · ) at an input o , E [ L ( t, ˆ f ( o ))] , can be split into epistemic ( E ) and aleatoric ( A ) components: 
Revised: The expected loss of a predictor ˆ f ( · ) at an input o , E [ L ( t, ˆ f ( o ))] , can be split into epistemic ( E ) and aleatoric ( A ) components: (cid:82) L ( t, ˆ f ( o ))

Original: [Equation]
Revised: dP ( t | o ) =

Original:  where P ( t | o ) is the conditional distribution over targets for input o and L ( t, · ) is a chosen loss function.
Revised: E ( ˆ f, o ) + A ( o ) , where P ( t | o ) is the conditional distribution over targets for input o and L ( t, · ) is a chosen loss function.

Original: The aleatoric uncertainty at input o , A ( o ) , is deﬁned as the expected prediction error of a Bayes optimal predictor function.
Revised: The aleatoric uncertainty at input o , A ( o ) , is defined as the expected prediction error of a Bayes optimal predictor function.

Original: E ( ˆ f, o ) denotes the epistemic uncertainty of a predictor at o .
Revised: E ( ˆ f, o ) denotes the epistemic uncertainty of a predictor at o .

Original: To overcome this issue with aleatoric uncertainty in RC-GVF, we propose to train an ensemble of predictors and utilise the variance across their predictions as a multiplicative factor on the prediction error.
Revised: To overcome this issue with aleatoric uncertainty in RC-GVF, we propose to train an ensemble of predictors and utilise the variance across their predictions as a multiplicative factor on the prediction error.

Original: Concretely, we train K predictors ˆ v k ⇡ ,z , k 2 { 1 , 2 . . .
Revised: Concretely, we train K predictors ˆ v kπ,z , k ∈ { 1 , 2 . . .

Original: K } .
Revised: K } .

Original: The prediction target for each member of the ensemble is the same  -pseudo-return (Equation 4) by using bootstrapped values from that member’s predictions.
Revised: The prediction target for each member of the ensemble is the same λ -pseudo-return (Equation 4) by using bootstrapped values from that member’s predictions.

Original: Using the ensemble of predictors, the intrinsic reward is given by
Revised: Using the ensemble of predictors, the intrinsic reward is given by

Original: [Equation]
Revised: [Equation]

Original: where  corresponds to element-wise multiplication.
Revised: where ⊙ corresponds to element-wise multiplication.

Original: In this formulation, even when prediction error remains, the exploration bonus will vanish as the predictors converge to the same expectation.
Revised: In this formulation, even when prediction error remains, the exploration bonus will vanish as the predictors converge to the same expectation.

Original: 3.3 Effective horizon and relation to random network distillation
Revised: 3.3 Effective horizon and relation to random network distillation

Original: The effective horizon over which predictions are considered depends on the choice of the discount factor  z .
Revised: The effective horizon over which predictions are considered depends on the choice of the discount factor γ z .

Original: It can be shown that for a horizon H  z , " 2 R satisfying
Revised: It can be shown that for a horizon H γ z ,ε ∈ R satisfying

Original: [Equation]
Revised: [Equation]

Original: (8) the discounted sum of (pseudo) rewards beyond this horizon is bounded by " [28, 30].
Revised: the discounted sum of (pseudo) rewards beyond this horizon is bounded by ε [28, 30].

Original: From Equation 4 we can see that for the special case of  z = 0 and any choice of  z , we have
Revised: From Equation 4 we can see that for the special case of γ z = 0 and any choice of λ z , we have

Original: G zt
Revised: G zt

Original: =
Revised: =

Original: Z t +1 =
Revised: Z t +1 =

Original: Z  ( o t ) .
Revised: Z ϕ ( o t ) .

Original: Note that in this case the TD-error between prediction and target is equivalent to the intrinsic reward provided by random network distillation (RND; Equation 2).
Revised: Note that in this case the TD-error between prediction and target is equivalent to the intrinsic reward provided by random network distillation (RND; Equation 2).

Original: Since the prediction targets are now deterministic, there is no problem with aleatoric uncertainty, and a single predictor sufﬁces (as opposed to an ensemble).
Revised: Since the prediction targets are now deterministic, there is no problem with aleatoric uncertainty, and a single predictor suffices (as opposed to an ensemble).

Original:  Related work
Revised:  Related work

Original: Exploration In tabular RL, explicit counts for states or state-action pairs can be maintained to enable exploration and achieve efﬁcient learning [61, 4].
Revised: Exploration In tabular RL, explicit counts for states or state-action pairs can be maintained to enable exploration and achieve efficient learning [61, 4].

Original: Count-based bonuses have been scaled to larger state spaces through the use of density models, which provide pseudo-counts [7, 41].
Revised: Count-based bonuses have been scaled to larger state spaces through the use of density models, which provide pseudo-counts [7, 41].

Original: State novelty bonuses are similarly inspired heuristics that provide a bonus based on estimated novelty of the visited state [10].
Revised: State novelty bonuses are similarly inspired heuristics that provide a bonus based on estimated novelty of the visited state [10].

Original: Perhaps the simplest way of implementing exploration through ‘curiosity’ is to reward the policy for encountering transitions to states that surprise a learning predictor [52].
Revised: Perhaps the simplest way of implementing exploration through ‘curiosity’ is to reward the policy for encountering transitions to states that surprise a learning predictor [52].

Original: This is suitable for deterministic environments but it suffers from the ‘noisy TV problem’ in stochastic environments [56, 9].
Revised: This is suitable for deterministic environments but it suffers from the ‘noisy TV problem’ in stochastic environments [56, 9].

Original: To overcome this limitation, intrinsic rewards have been deﬁned by learning progress (the ﬁrst derivative of the prediction error)
Revised: To overcome this limitation, intrinsic rewards have been defined by learning progress (the first derivative of the prediction error)

Original: [51], information gain [60, 22], or compression progress [55].
Revised: [51], information gain [60, 22], or compression progress [55].

Original: Also compare such bonuses to pseudo-counts [7].
Revised: Also compare such bonuses to pseudo-counts [7].

Original: Alternatively, it is possible to mitigate sensitivity to noise by measuring prediction errors in a latent space [53, 43].
Revised: Alternatively, it is possible to mitigate sensitivity to noise by measuring prediction errors in a latent space [53, 43].

Original: Our method is different from state count and novelty approaches in that it models temporally extended values beyond the current observation.
Revised: Our method is different from state count and novelty approaches in that it models temporally extended values beyond the current observation.

Original: It includes a state-novelty measure, RND [10], as a special case when the GVF discount factors are zero.
Revised: It includes a state-novelty measure, RND [10], as a special case when the GVF discount factors are zero.

Original: Unlike most artiﬁcial curiosity approaches, RC-GVF does not model the entire environment dynamics, in line with the idea of conducting temporally extended experiments whose outcomes are abstract summaries (represented in latent variables) of long observation sequences [54].
Revised: Unlike most artificial curiosity approaches, RC-GVF does not model the entire environment dynamics, in line with the idea of conducting temporally extended experiments whose outcomes are abstract summaries (represented in latent variables) of long observation sequences [54].

Original: We mitigate sensitivity to aleatoric uncertainty by measuring the disagreement of an ensemble of predictors, inspired by prior work (for single-step prediction models) [59, 44].
Revised: We mitigate sensitivity to aleatoric uncertainty by measuring the disagreement of an ensemble of predictors, inspired by prior work (for single-step prediction models) [59, 44].

Original: Ensembles for exploration have also been explored in the context of posterior sampling [62].
Revised: Ensembles for exploration have also been explored in the context of posterior sampling [62].

Original: Bootstrapped-DQN [38] uses an ensemble of Q-value functions for a model-free interpretation of posterior sampling [39].
Revised: Bootstrapped-DQN [38] uses an ensemble of Q-value functions for a model-free interpretation of posterior sampling [39].

Original: Unlike in our approach, these value functions are restricted to the original task rewards, and can not capture arbitrary pseudo-rewards.
Revised: Unlike in our approach, these value functions are restricted to the original task rewards, and can not capture arbitrary pseudo-rewards.

Original: Recent works suggest incorporating model-based planning to directly optimise for long-term novelty [63, 58, 31].
Revised: Recent works suggest incorporating model-based planning to directly optimise for long-term novelty [63, 58, 31].

Original: Our intrinsic reward–which is generated by considering multiple future steps–could motivate approaches for optimising long-term novelty without explicit rollout-based planning.
Revised: Our intrinsic reward–which is generated by considering multiple future steps–could motivate approaches for optimising long-term novelty without explicit rollout-based planning.

Original: General Value Functions Sutton et al.
Revised: General Value Functions Sutton et al.

Original: [67] proposed general value functions (GVFs) as an approach to represent predictive knowledge about the world.
Revised: [67] proposed general value functions (GVFs) as an approach to represent predictive knowledge about the world.

Original: Closely related is the vector-valued adaptive critic [50] which predicts and controls cumulative values of special input vectors.
Revised: Closely related is the vector-valued adaptive critic [50] which predicts and controls cumulative values of special input vectors.

Original: Several works have studied the impact of using auxiliary value functions to improve representation learning in RL [24, 8, 35].
Revised: Several works have studied the impact of using auxiliary value functions to improve representation learning in RL [24, 8, 35].

Original: Further, it was previously found that GVFs of randomly generated pseudo-rewards can also be useful for shaping and learning representations [34, 78] (see also related work on forecasts [47], TD-networks [66], and predictive state representations [32]).
Revised: Further, it was previously found that GVFs of randomly generated pseudo-rewards can also be useful for shaping and learning representations [34, 78] (see also related work on forecasts [47], TD-networks [66], and predictive state representations [32]).

Original: This can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy ⇡ and discount factor  z .
Revised: This can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy π and discount factor γ z .

Original: In our implementation, the predictor is an entirely separate network and thus
Revised: In our implementation, the predictor is an entirely separate network and thus

Original: GVF prediction can not be used to improve the agent’s internal representation.
Revised: GVF prediction can not be used to improve the agent’s internal representation.

Original: Instead, we use the
Revised: Instead, we use the

Original: GVF prediction to derive a useful signal to guide exploration.
Revised: GVF prediction to derive a useful signal to guide exploration.

Original: The discovery/selection of useful GVF ‘questions’ is an open problem [48].
Revised: The discovery/selection of useful GVF ‘questions’ is an open problem [48].

Original: Existing approaches learn questions related to optimizing performance on the main task [73, 27].
Revised: Existing approaches learn questions related to optimizing performance on the main task [73, 27].

Original: However, such an approach might not be suitable for sparse-reward environments.
Revised: However, such an approach might not be suitable for sparse-reward environments.

Original: In our approach, we side-step the GVF discovery problem through the use of randomly generated pseudo-rewards and on-policy predictions.
Revised: In our approach, we side-step the GVF discovery problem through the use of randomly generated pseudo-rewards and on-policy predictions.

Original:  Experiments
Revised:  Experiments

Original: In this section we present an empirical evaluation of RC-GVF.
Revised: In this section we present an empirical evaluation of RC-GVF.

Original: First, we consider a hard exploration diabolical lock problem from the literature [36] and demonstrate the beneﬁts of predicting beyond the immediate observation.
Revised: First, we consider a hard exploration diabolical lock problem from the literature [36] and demonstrate the benefits of predicting beyond the immediate observation.

Original: We then proceed to experiments with the standard Minigrid suite of partially observable and procedurally generated environments, demonstrating how RC-GVF is able to outperform several strong baselines [18, 77].
Revised: We then proceed to experiments with the standard Minigrid suite of partially observable and procedurally generated environments, demonstrating how RC-GVF is able to outperform several strong baselines [18, 77].

Original: 5.1 Diabolical locks
Revised: 5.1 Diabolical locks

Original: The diabolical lock problem [36] is considered a difﬁcult environment for exploration due to noisy high-dimensional observations, stochastic dynamics and misleading rewards that deter the agent from ﬁnding the sparse optimal reward at the end of the lock.
Revised: The diabolical lock problem [36] is considered a difficult environment for exploration due to noisy high-dimensional observations, stochastic dynamics and misleading rewards that deter the agent from finding the sparse optimal reward at the end of the lock.

Original: In the literature, RND is often shown to fail on these types of environments due to only considering the immediate future [36].
Revised: In the literature, RND is often shown to fail on these types of environments due to only considering the immediate future [36].

Original: Due to
Revised: Due to

Original: RC-GVF’s policy-conditioned predictions of future quantities
Revised: RC-GVF’s policy-conditioned predictions of future quantities

Original: , this task is instructive to demonstrate the qualitative advantages of our method and how this results in markedly different performance.
Revised: , this task is instructive to demonstrate the qualitative advantages of our method and how this results in markedly different performance.

Original: [Table]
Revised: [Table]

Original: Environment details The environment consists of states organised in 3 rows { a, b, c } and H columns from { 1 , 2 , . . .
Revised: Environment details The environment consists of states organised in 3 rows { a, b, c } and H columns from { 1 , 2 , . . .

Original: H } (see Figure 6 in
Revised: H } (see Figure 6 in

Original: Appendix A.1).
Revised: Appendix A.1).

Original: In each episode, the agent is initialised in one of two possible starting states, either at a 1 or b 1 .
Revised: In each episode, the agent is initialised in one of two possible starting states, either at a 1 or b 1 .

Original: At each state the agent has
Revised: At each state the agent has

Original: L available actions, only one of which is ‘good’ and transitions to a ‘good’ state in the next column with equal probability.
Revised: L available actions, only one of which is ‘good’ and transitions to a ‘good’ state in the next column with equal probability.

Original: Taking the good action gives a negative (anti-shaped) reward of  1 /H , except at the end of the lock (states a H or b H ), where the sparse optimal reward of 10 is received.
Revised: Taking the good action gives a negative (anti-shaped) reward of − 1 /H , except at the end of the lock (states a H or b H ), where the sparse optimal reward of 10 is received.

Original: The remaining L  1 actions at any state are ‘bad’, causing a deterministic transition to the ‘dead’ row c at the next column (zero reward).
Revised: The remaining L − 1 actions at any state are ‘bad’, causing a deterministic transition to the ‘dead’ row c at the next column (zero reward).

Original: All actions from a dead state lead to the dead state in the next row, and hence the optimal policy is to take the respective good actions in each state.
Revised: All actions from a dead state lead to the dead state in the next row, and hence the optimal policy is to take the respective good actions in each state.

Original: The good action at each state is assigned randomly as part of the MDP speciﬁcation.
Revised: The good action at each state is assigned randomly as part of the MDP specification.

Original: The agent never observes the state directly and only has access to a high-dimensional noisy observation (details in Appendix A.1).
Revised: The agent never observes the state directly and only has access to a high-dimensional noisy observation (details in Appendix A.1).

Original: In our experiment, we consider a problem with horizon H = 100 , L = 10 , and observation noise  o = 0 .
Revised: In our experiment, we consider a problem with horizon H = 100 , L = 10 , and observation noise σ o = 0 .

Original: As discussed by Misra et al.
Revised: As discussed by Misra et al.

Original: [36], this exploration problem is hard because the probability of reaching the optimal reward through a uniformly random policy is L  H ( 10  100 in our instance).
Revised: [36], this exploration problem is hard because the probability of reaching the optimal reward through a uniformly random policy is L − H ( 10 − 100 in our instance).

Original: Further, the stochastic transitions from good states prevent a solution that relies on memorising a state-independent successful sequence of H actions independent of the state.
Revised: Further, the stochastic transitions from good states prevent a solution that relies on memorising a state-independent successful sequence of H actions independent of the state.

Original: Implementation We use Proximal Policy Optimization (PPO)
Revised: Implementation We use Proximal Policy Optimization (PPO)

Original: [57] in an actor-critic framework as the base agent and use 128 pseudo-rewards for RND and RC-GVF.
Revised: [57] in an actor-critic framework as the base agent and use 128 pseudo-rewards for RND and RC-GVF.

Original: For RC-GVF we set  z = 0 .and
Revised: For RC-GVF we set γ z = 0 .and

Original: use two prediction heads in the ensemble.
Revised: use two prediction heads in the ensemble.

Original: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.
Revised: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.

Original: Further details are in Appendix A.1.
Revised: Further details are in Appendix A.1.

Original: Results Table 1 compares RND to RC-GVF in this environment.
Revised: Results Table 1 compares RND to RC-GVF in this environment.

Original: It can be seen how RC-GVF succeeds in completing the lock on all runs, while the best run of RND only reaches 60% of the lock (and does not complete it).
Revised: It can be seen how RC-GVF succeeds in completing the lock on all runs, while the best run of RND only reaches 60% of the lock (and does not complete it).

Original: Indeed, once the agent takes an incorrect action and ‘falls’ into the bottom row, the future consequences (and thus, the GVFs) become highly predictable.
Revised: Indeed, once the agent takes an incorrect action and ‘falls’ into the bottom row, the future consequences (and thus, the GVFs) become highly predictable.

Original: By staying alive, the agent will encounter new observations and unpredictable transitions which appear interesting.
Revised: By staying alive, the agent will encounter new observations and unpredictable transitions which appear interesting.

Original: Further experiments with additional baselines are presented in Appendix D.
Revised: Further experiments with additional baselines are presented in Appendix D.

Original: This result is illustrative of RC-GVFs exploration capabilities as we will see next.
Revised: This result is illustrative of RC-GVFs exploration capabilities as we will see next.

Original: 5.2 MiniGrid
Revised: 5.2 MiniGrid

Original: We evaluate RC-GVF on procedurally generated environments from MiniGrid [13], which is a standard benchmark in the deep reinforcement learning literature for exploration [46, 11, 18, 77, 76, 42].
Revised: We evaluate RC-GVF on procedurally generated environments from MiniGrid [13], which is a standard benchmark in the deep reinforcement learning literature for exploration [46, 11, 18, 77, 76, 42].

Original: Exploration in these environments is challenging due to partial observability, extremely sparse rewards, and the procedural generation of mazes and objects.
Revised: Exploration in these environments is challenging due to partial observability, extremely sparse rewards, and the procedural generation of mazes and objects.

Original: Environment Details Broadly speaking, we will consider three classes of environments organised according to several difﬁculty levels.
Revised: Environment Details Broadly speaking, we will consider three classes of environments organised according to several difficulty levels.

Original: First, we study MultiRoom-N7-S8 and MultiRoom-N12-S10 , which are navigation tasks in a maze with seven and twelve rooms respectively.
Revised: First, we study MultiRoom-N7-S8 and MultiRoom-N12-S10 , which are navigation tasks in a maze with seven and twelve rooms respectively.

Original: Next, we consider two levels of difﬁculty in KeyCorridor-S4R3 and KeyCorridor-S5R3 .
Revised: Next, we consider two levels of difficulty in KeyCorridor-S4R3 and KeyCorridor-S5R3 .

Original: Here, the agent needs to pick up a ball behind a locked door.
Revised: Here, the agent needs to pick up a ball behind a locked door.

Original: Finally, we examine the ObstructedMaze set of environments where the agent has a similar task of picking up a ball behind a locked door, but keys are hidden in boxes ( ObstructedMaze-2Dlh ), and doors can be obstructed ( ObstructedMaze-2Dlhb ).
Revised: Finally, we examine the ObstructedMaze set of environments where the agent has a similar task of picking up a ball behind a locked door, but keys are hidden in boxes ( ObstructedMaze-2Dlh ), and doors can be obstructed ( ObstructedMaze-2Dlhb ).

Original: The observations
Revised: The observations

Original: [Figure]
Revised: [Figure]

Revised: v

Revised: [Figure]

Revised: t u

Revised: [Figure]

Revised: v

Revised: [Figure]

Revised: g

Original:  corresponds to an  view of  cells 
Revised: [Figure] t u r [Figure] corresponds to an egocentric view of the cells in the front of the agent.

Original:  MiniGrid  are available in Appendix A.2.
Revised: Further details of the MiniGrid environments are available in Appendix A.2.

Original: Implementation We compare RC-GVF to RND [10], AGAC [18], and NovelD [77].
Revised: Implementation We compare RC-GVF to RND [10], AGAC [18], and NovelD [77].

Original: We use
Revised: We use

Original: Proximal Policy Optimization (PPO) [57] as our base agent for all approaches.
Revised: Proximal Policy Optimization (PPO) [57] as our base agent for all approaches.

Original: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.
Revised: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.

Original: At each time step t , the agent receives a reward R t =
Revised: At each time step t , the agent receives a reward R t =

Original: R e ( s t ) +  R i ( o t ) , where  2 R + balances the contribution of the extrisic R e ( s t ) and
Revised: R e ( s t ) + βR i ( o t ) , where β ∈ R + balances the contribution of the extrinsic R e ( s t ) and

Original: intrinsic R i ( o t ) rewards.
Revised: intrinsic R i ( o t ) rewards.

Original: For RC-GVF we set  z = 0 . 6 , use two prediction heads in the ensemble of predictors and use 128 pseudo-rewards (same as for RND).
Revised: For RC-GVF we set γ z = 0 . 6 , use two prediction heads in the ensemble of predictors and use 128 pseudo-rewards (same as for RND).

Original: Other important hyper-parameters, such as the intrinsic reward coefﬁcient (  ), entropy coefﬁcient, and learning rate of the predictor are obtained via an extensive hyper-parameter search for all baselines (see Appendices C.2 and C.3 for details, including on our implementation of baselines.).
Revised: Other important hyper-parameters, such as the intrinsic reward coefficient ( β ), entropy coefficient, and learning rate of the predictor are obtained via an extensive hyper-parameter search for all baselines (see Appendices C.2 and C.3 for details, including on our implementation of baselines.).

Original: We present results averaged over 10 independent runs for each approach in every ﬁgure (solid line).
Revised: We present results averaged over 10 independent runs for each approach in every figure (solid line).

Original: Unless mentioned otherwise, the shading indicates 95% bootstrapped conﬁdence intervals.
Revised: Unless mentioned otherwise, the shading indicates 95% bootstrapped confidence intervals.

Original: 5.2.1 Egocentric observations
Revised: 5.2.1 Egocentric observations

Original: We will ﬁrst consider the usual setting where the agent receives access to the egocentric observations from the environment.
Revised: We will first consider the usual setting where the agent receives access to the egocentric observations from the environment.

Original: Importantly, we do not provide agents with access to privileged information about the environment in the form of so-called ‘episodic counts’.
Revised: Importantly, we do not provide agents with access to privileged information about the environment in the form of so-called ‘episodic counts’.

Original: This is different from several recent approaches that have been applied to MiniGrid, which incorporate these values as part of their intrinsic reward [46, 18, 77].
Revised: This is different from several recent approaches that have been applied to MiniGrid, which incorporate these values as part of their intrinsic reward [46, 18, 77].

Original: The use of episodic counts obtained through the simulator is problematic for solving exploration problems in a partially observable setting, since it tells the agent precisely how frequently it has been in each state.
Revised: The use of episodic counts obtained through the simulator is problematic for solving exploration problems in a partially observable setting, since it tells the agent precisely how frequently it has been in each state.

Original: Indeed, the use of episodic counts alone can be sufﬁcient for exploration in MiniGrid (Figure 7 in Appendix E.1).
Revised: Indeed, the use of episodic counts alone can be sufficient for exploration in MiniGrid (Figure 7 in Appendix E.1).

Original: Meanwhile, there exists no good method for obtaining episodic counts in the absence of a simulator, as it is difﬁcult to accurately estimate pseudo-counts from partial observations (eg.
Revised: Meanwhile, there exists no good method for obtaining episodic counts in the absence of a simulator, as it is difficult to accurately estimate pseudo-counts from partial observations (eg.

Original: as in Figure 1).
Revised: as in Figure 1).

Original: Details about the exact usage of episodic counts by baselines in their intrinsic reward formulation are provided in Appendix C.2.
Revised: Details about the exact usage of episodic counts by baselines in their intrinsic reward formulation are provided in Appendix C.2.

Original: In Figure 2 we compare RC-GVF, RND, AGAC, and NovelD in the absence of episodic counts, while using egocentric observations.
Revised: In Figure 2 we compare RC-GVF, RND, AGAC, and NovelD in the absence of episodic counts, while using egocentric observations.

Original: It can be seen that RC-GVF explores faster and succeeds more often than RND and the other baselines in the KeyCorridor-S4R3 environment and MultiRoomN7-S8 environment.
Revised: It can be seen that RC-GVF explores faster and succeeds more often than RND and the other baselines in the KeyCorridor-S4R3 environment and MultiRoomN7-S8 environment.

Original: In comparison to RND and NovelD that utilise prediction errors in immediate random embeddings, it appears that RC-GVF beneﬁts from extended horizon predictions of random pseudo-rewards.
Revised: In comparison to RND and NovelD that utilise prediction errors in immediate random embeddings, it appears that RC-GVF benefits from extended horizon predictions of random pseudo-rewards.

Original: Meanwhile, AGAC only succeeds in ObstructedMaze-2Dlh, where all baselines perform similarly well.
Revised: Meanwhile, AGAC only succeeds in ObstructedMaze-2Dlh, where all baselines perform similarly well.

Original: Finally, we observe that all approaches struggle with the most difﬁcult level of considered environments (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).
Revised: Finally, we observe that all approaches struggle with the most difficult level of considered environments (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).

Original: [Figure]
Revised: [Figure]

Revised: -GVF (using panoramic observations) achieves similar performance without.

Original: 5.2.2 Panoramic observations
Revised: 5.2.2 Panoramic observations

Original: We now demonstrate how using episodic counts from the simulator is crucial to the success of the baselines.
Revised: We now demonstrate how using episodic counts from the simulator is crucial to the success of the baselines.

Original: In Figure 3 it can be seen how AGAC and NovelD improve with episodic counts, and also solve the more difﬁcult problems (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).
Revised: In Figure 3 it can be seen how AGAC and NovelD improve with episodic counts, and also solve the more difficult problems (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).

Original: Panoramic observations
Revised: Panoramic observations

Original: In order to increase performance of RC-GVF further, without introducing episodic counts, we investigate the use of augmented panoramic views as previously explored by
Revised: In order to increase performance of RC-GVF further, without introducing episodic counts, we investigate the use of augmented panoramic views as previously explored by

Original: Parisi et al.
Revised: Parisi et al.

Original: In the MiniGrid environments, the observation changes almost in its entirety when the agent changes the direction it faces.
Revised: In the MiniGrid environments, the observation changes almost in its entirety when the agent changes the direction it faces.

Original: In contrast, moving to the next cell leads to fewer sudden changes in the observation.
Revised: In contrast, moving to the next cell leads to fewer sudden changes in the observation.

Original: The lack of gradual changes of egocentric observations through rotations is a consequence of having four discrete angles for orientation (0  , 90  , 180  , 270  ) , which may be an unrealistic depiction of how rotations affect observations for agents situated in the real world.
Revised: The lack of gradual changes of egocentric observations through rotations is a consequence of having four discrete angles for orientation (0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) , which may be an unrealistic depiction of how rotations affect observations for agents situated in the real world.

Original: As a consequence, prediction errors are dominated by predictions of the outcomes of turning (rather than other actions).
Revised: As a consequence, prediction errors are dominated by predictions of the outcomes of turning (rather than other actions).

Original: To address this, prior work proposed to make the agent’s observations invariant to rotation by augmenting the observation with all directional observations [42].
Revised: To address this, prior work proposed to make the agent’s observations invariant to rotation by augmenting the observation with all directional observations [42].

Original: In effect, this assumes that the agent rotates 360  after moving to a cell, or alternatively that it is equipped with additional sensors on its sides and back.
Revised: In effect, this assumes that the agent rotates 360 ◦ after moving to a cell, or alternatively that it is equipped with additional sensors on its sides and back.

Original: Similar to Parisi et al.
Revised: Similar to Parisi et al.

Original: [42], we will consider panoramic views only for generating intrinsic rewards, and egocentric observations for the base PPO agent.
Revised: [42], we will consider panoramic views only for generating intrinsic rewards, and egocentric observations for the base PPO agent.

Original: In Figure 3, we demonstrate how RC-GVF with panoramic views and without privileged information about the underlying state of the MDP can solve harder problems and is competitive with the baselines that use episodic counts in ﬁve out of the six settings.
Revised: In Figure 3, we demonstrate how RC-GVF with panoramic views and without privileged information about the underlying state of the MDP can solve harder problems and is competitive with the baselines that use episodic counts in five out of the six settings.

Original: We see that RC-GVF with panoramic observations comfortably succeeds in the KeyCorridor-S5R3 and MultiRoom-N12-S10 environments where it was previously unsuccessful.
Revised: We see that RC-GVF with panoramic observations comfortably succeeds in the KeyCorridor-S5R3 and MultiRoom-N12-S10 environments where it was previously unsuccessful.

Original: RC-GVF did not succeed in solving the harder ObstructedMaze2Dlhb within the given frames despite the inclusion of panoramic views.
Revised: RC-GVF did not succeed in solving the harder ObstructedMaze2Dlhb within the given frames despite the inclusion of panoramic views.

Original: Figure 3 shows that the use of panoramic views does not make the exploration problem trivial; RND also uses panoramic views (but no episodic counts).
Revised: Figure 3 shows that the use of panoramic views does not make the exploration problem trivial; RND also uses panoramic views (but no episodic counts).

Original: It improves with panoramic views on KeyCorridor-S4R3 and MultiRoom-N7S but struggles with the harder instances.
Revised: It improves with panoramic views on KeyCorridor-S4R3 and MultiRoom-N7S but struggles with the harder instances.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: Changing the temporal prediction horizon We study the effect of changing the temporal prediction horizon by evaluating RC-GVF with different discount factors.
Revised: Changing the temporal prediction horizon We study the effect of changing the temporal prediction horizon by evaluating RC-GVF with different discount factors.

Original: Recall from Equation
Revised: Recall from Equation

Revised: 18 Frames (millions) 0.

Revised: A v e r a g e r e t u r n (a) KeyCorridor-S5R       6 Frames (millions) 0.

Revised: A v e r a g e r e t u r n z =0.99 z =0.

Revised: z =0.7 z =0.

Revised: z =0.3 z

Original:  [Figure]
Revised: =0. z =0.05 z [Figure] [Figure]

Original: how the discount factor can me mapped to the prediction horizon, which lets us estimate a lower bound on the horizon of about 11 time steps for  z = 0 . 6 with ✏ = 0 .
Revised: how the discount factor can me mapped to the prediction horizon, which lets us estimate a lower bound on the horizon of about 11 time steps for γ z = 0 . 6 with ϵ = 0 .

Original: In Figure 4 it can be seen how a larger prediction horizon is desirable for KeyCorridorS5R3.
Revised: In Figure 4 it can be seen how a larger prediction horizon is desirable for KeyCorridorS5R3.

Original: Among the higher discount factors, 0 .
Revised: Among the higher discount factors, 0 .

Original: 95 is a good choice for this environment, but not 0 . 99 .
Revised: 95 is a good choice for this environment, but not 0 . 99 .

Original: Compared to our default value of  z = 0 .
Revised: Compared to our default value of γ z = 0 .

Original: 6 , a short horizon using 0 .
Revised: 6 , a short horizon using 0 .

Original: 3 reduces performance further, while discount factors closest to 0 yield worse performance.
Revised: 3 reduces performance further, while discount factors closest to 0 yield worse performance.

Original: In contrast, for MultiRoom-N12-S10, we ﬁnd that low and intermediate values of  z perform well.
Revised: In contrast, for MultiRoom-N12-S10, we find that low and intermediate values of γ z perform well.

Original: The high GVF discount factors of 0 . 99 and 0 .
Revised: The high GVF discount factors of 0 . 99 and 0 .

Original: 95 perform worse.
Revised: 95 perform worse.

Original: Comparing  z =to RND shows how the use of recurrent networks and the variance term derived from the ensemble of predictors contributes most to the performance of RC-GVF in this domain.
Revised: Comparing γ z =to RND shows how the use of recurrent networks and the variance term derived from the ensemble of predictors contributes most to the performance of RC-GVF in this domain.

Original: Ablation To better understand which of these components are contributing most to the success of RC-GVF, we consider the following additional ablations: (1) RC-GVF (  z = 0 ) without the variance term in Equation 6, and (2) RC-GVF (  z = 0 ) without the recurrent predictor.
Revised: Ablation To better understand which of these components are contributing most to the success of RC-GVF, we consider the following additional ablations: (1) RC-GVF ( γ z = 0 ) without the variance term in Equation 6, and (2) RC-GVF ( γ z = 0 ) without the recurrent predictor.

Original: Figure 5a compares these variations to RC-GVF (  z = 0 ), RC-GVF (  z = 0 . 6 ), and RND.
Revised: Figure 5a compares these variations to RC-GVF ( γ z = 0 ), RC-GVF ( γ z = 0 . 6 ), and RND.

Original: It can be seen that the inclusion of the history-conditioned recurrent predictor and the variance term individually contribute to the improved performance of RC-GVF.
Revised: It can be seen that the inclusion of the history-conditioned recurrent predictor and the variance term individually contribute to the improved performance of RC-GVF.

Original: Using the history conditioned recurrent predictor sharply improves the performance of RC-GVF with  z = 0 .
Revised: Using the history conditioned recurrent predictor sharply improves the performance of RC-GVF with γ z = 0 .

Original: We hypothesise that this effect arises from the previous pseudo-rewards available to the predictor, which might enable better predictions on input observations it was not explicitly trained on (which is often the case in procedurally generated environments).
Revised: We hypothesise that this effect arises from the previous pseudo-rewards available to the predictor, which might enable better predictions on input observations it was not explicitly trained on (which is often the case in procedurally generated environments).

Original: In support of this hypothesis, we observed that using a recurrent predictor with solely observations as inputs does not produce such an improvement (Figure 9 in Appendix E.3).
Revised: In support of this hypothesis, we observed that using a recurrent predictor with solely observations as inputs does not produce such an improvement (Figure 9 in Appendix E.3).

Original: In Figure 5b, we present ablations for  z = 0 .
Revised: In Figure 5b, we present ablations for γ z = 0 .

Original: We consider RC-GVF without the variance term in Equation 6, and without the recurrent predictor.
Revised: We consider RC-GVF without the variance term in Equation 6, and without the recurrent predictor.

Original: Removing either the disagreement term or the recurrent predictor worsens the performance of RC-GVF with  z = 0 . 
Revised: Removing either the disagreement term or the recurrent predictor worsens the performance of RC-GVF with γ z = 0 . 6 .

Original: Interestingly, unlike as was observed for  z = 0 , here we ﬁnd that RC-GVF with only disagreement (no recurrent predictor) performs better than RC-GVF with only recurrent predictor (no disagreement).
Revised: Interestingly, unlike as was observed for γ z = 0 , here we find that RC-GVF with only disagreement (no recurrent predictor) performs better than RC-GVF with only recurrent predictor (no disagreement).

Original: This further emphasises the importance of handling the aleatoric uncertainty for non-zero GVF discounts.
Revised: This further emphasises the importance of handling the aleatoric uncertainty for non-zero GVF discounts.

Original: Increasing the ensemble size We conduct experiments with larger ensemble sizes in two MiniGrid environments.
Revised: Increasing the ensemble size We conduct experiments with larger ensemble sizes in two MiniGrid environments.

Original: We obtain comparable results for ensemble sizes of 2, 4, 6 and 8 predictors in Figurein Appendix E.4, indicating that two member ensembles usually sufﬁce for these problems.
Revised: We obtain comparable results for ensemble sizes of 2, 4, 6 and 8 predictors in Figurein Appendix E.4, indicating that two member ensembles usually suffice for these problems.

Original:  Limitations and Societal Impact
Revised:  Limitations and Societal Impact

Original: Limitations The incorporation of the current policy into the prediction task of GVF prediction can have beneﬁts with regards to behavioural exploration.
Revised: Limitations The incorporation of the current policy into the prediction task of GVF prediction can have benefits with regards to behavioural exploration.

Original: However, it may also encourage over-exploration due to the same state appearing ‘interesting’ under different policies.
Revised: However, it may also encourage over-exploration due to the same state appearing ‘interesting’ under different policies.

Original: Possible mitigation strategies include off-policy variants or explicitly policy-conditioned (general) value functions [17, 20].
Revised: Possible mitigation strategies include off-policy variants or explicitly policy-conditioned (general) value functions [17, 20].

Original: In some situations it may be difﬁcult to ascertain what is a good choice of the GVF discount factor  z , which effectively determines the prediction horizon length, and can be inﬂuenced by optimisation effects under function approximation [72].
Revised: In some situations it may be difficult to ascertain what is a good choice of the GVF discount factor γ z , which effectively determines the prediction horizon length, and can be influenced by optimisation effects under function approximation [72].

Original: The best choice may be problem-dependent and could
Revised: The best choice may be problem-dependent and could

Revised: [Figure]

Revised: [Figure]

Revised: A v e r a g e r e t u r n

Revised: RC

Original: [Figure]
Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: ) w/o disagreement RC

Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: A

Revised: [Figure]

Revised: e r e t u r n

Revised: [Figure]

Revised: [Figure]

Revised: disagreement RC

Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: [Figure]

Revised: RC

Revised: [Figure]

Original:  further beneﬁt from dynamic adjustment during learning [5, 37].
Revised: z [Figure] [Figure] further benefit from dynamic adjustment during learning [5, 37].

Original: Nonetheless, our results in Figures 2 and 3 indicate that a default of  z = 0 .
Revised: Nonetheless, our results in Figures 2 and 3 indicate that a default of γ z = 0 .

Original: 6 works well in many of the considered settings.
Revised: 6 works well in many of the considered settings.

Original: Several other intrinsic reward formulations using RC-GVF are possible (eg.
Revised: Several other intrinsic reward formulations using RC-GVF are possible (eg.

Original: based on information/prediction gain, learning progress, or only the disagreement term), which we did not investigate.
Revised: based on information/prediction gain, learning progress, or only the disagreement term), which we did not investigate.

Original: Our speciﬁc formulation in Equation 6 was motivated by the prediction error in RND, which does not separate the aleatoric uncertainty when  z > 0 .
Revised: Our specific formulation in Equation 6 was motivated by the prediction error in RND, which does not separate the aleatoric uncertainty when γ z > 0 .

Original: Starting with the prediction error term also allowed us to closely study the connection between RND and RC-GVF.
Revised: Starting with the prediction error term also allowed us to closely study the connection between RND and RC-GVF.

Original: A general limitation of curiosity-based exploration for task-speciﬁc RL is that more information is gathered than is necessary to solve the given task [40, 33].
Revised: A general limitation of curiosity-based exploration for task-specific RL is that more information is gathered than is necessary to solve the given task [40, 33].

Original: Indeed, the space of potential GVFs is enormous.
Revised: Indeed, the space of potential GVFs is enormous.

Original: In the case of RC-GVF one could aim at focusing it by integrating task-speciﬁc information [68] or by considering predictions using a smaller set of policies that adequately cover the state space [1].
Revised: In the case of RC-GVF one could aim at focusing it by integrating task-specific information [68] or by considering predictions using a smaller set of policies that adequately cover the state space [1].

Original:  impact This paper does not focus on real-world applications of exploration in RL and thus does not have a direct societal impact.
Revised: Societal impact This paper does not focus on real-world applications of exploration in RL and thus does not have a direct societal impact.

Original: In applications, advanced exploration methods may lead to unexpected policy behaviour, which could be mitigated by incorporating safety constraints [19].
Revised: In applications, advanced exploration methods may lead to unexpected policy behaviour, which could be mitigated by incorporating safety constraints [19].

Original:  Conclusion
Revised:  Conclusion

Original: We introduced random curiosity with general value functions (RC-GVF) , an exploration method that intrinsically rewards a reinforcement learning agent based on errors and uncertainty in predicting random transformations of observation sequences generated through the agent’s actions.
Revised: We introduced random curiosity with general value functions (RC-GVF) , an exploration method that intrinsically rewards a reinforcement learning agent based on errors and uncertainty in predicting random transformations of observation sequences generated through the agent’s actions.

Original: Unlike statenovelty approaches, ours takes into account multi-step future behaviours of policy and environment.
Revised: Unlike statenovelty approaches, ours takes into account multi-step future behaviours of policy and environment.

Original: Unlike commonly used artiﬁcial curiosity approaches, RC-GVF does not rely on model rollouts and does not need to predict all details of future observations.
Revised: Unlike commonly used artificial curiosity approaches, RC-GVF does not rely on model rollouts and does not need to predict all details of future observations.

Original: On the diabolical lock problem [36] and the MiniGrid environments [13] we demonstrate that predicting abstract quantities of extended time intervals can improve exploration in POMDPs.
Revised: On the diabolical lock problem [36] and the MiniGrid environments [13] we demonstrate that predicting abstract quantities of extended time intervals can improve exploration in POMDPs.

Original: Compared to recent methods such as AGAC [18] and NovelD [77] that rely on privileged episodic state-visitation counts in MiniGrid, our RC-GVF achieves competitive results even when using only panoramic observations.
Revised: Compared to recent methods such as AGAC [18] and NovelD [77] that rely on privileged episodic state-visitation counts in MiniGrid, our RC-GVF achieves competitive results even when using only panoramic observations.

Original: Importantly, in the natural setting where episodic counts are not available, RC-GVF signiﬁcantly outperforms all baseline methods.
Revised: Importantly, in the natural setting where episodic counts are not available, RC-GVF significantly outperforms all baseline methods.

Original: Our approach can be generalized by moving beyond random pseudo-rewards, considering general value functions under a set of different policies [1], and introducing time or state-dependent discounting.
Revised: Our approach can be generalized by moving beyond random pseudo-rewards, considering general value functions under a set of different policies [1], and introducing time or state-dependent discounting.

Original: This offers exciting avenues for future research.
Revised: This offers exciting avenues for future research.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank Kenny Young, Francesco Faccio, Anand Gopalakrishnan, and Dylan Ashley for valuable comments.
Revised: We would like to thank Kenny Young, Francesco Faccio, Anand Gopalakrishnan, and Dylan Ashley for valuable comments.

Original: This research was supported by the ERC Advanced Grant (742870), the Swiss
Revised: This research was supported by the ERC Advanced Grant (742870), the Swiss

Original: National Science Foundation grant (200021_192356), and by the Swiss National Supercomputing
Revised: National Science Foundation grant (200021_192356), and by the Swiss National Supercomputing

Original: Centre (CSCS projects s1090 and s1127).
Revised: Centre (CSCS projects s1090 and s1127).