Original: Exploring through Random Curiosity with General Value Functions
Revised: Exploring through Random Curiosity with General Value Functions

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Efficient exploration in reinforcement learning is a challenging problem commonlyaddressed through intrinsic rewards.
Revised: Efficient exploration in reinforcement learning is a challenging problem commonly addressed through intrinsic rewards.

Original: Recent prominent approaches are based onstate novelty or variants of artificial curiosity .
Revised: Recent prominent approaches are based on state novelty or variants of artificial curiosity .

Original: However, directly applying themto partially observable environments can be ineffective and lead to prematuredissipation of intrinsic rewards.
Revised: However, directly applying them to partially observable environments can be ineffective and lead to premature dissipation of intrinsic rewards.

Original: Here we propose random curiosity with generalvalue functions (RC-GVF), a novel intrinsic reward function that draws uponconnections between these distinct approaches.
Revised: Here we propose random curiosity with general value functions (RC-GVF), a novel intrinsic reward function that draws upon connections between these distinct approaches.

Original: Instead of only using only thecurrent observation’s novelty or a curiosity bonus for failing to predict preciseenvironment dynamics, RC-GVF derives intrinsic rewards through the task ofpredicting temporally extended general value functions.
Revised: Instead of only using only the current observation’s novelty or a curiosity bonus for failing to predict precise environment dynamics, RC-GVF derives intrinsic rewards through the task of predicting temporally extended general value functions.

Original: We demonstrate that thisimproves exploration in a hard-exploration diabolical lock problem.
Revised: We demonstrate that this improves exploration in a hard-exploration diabolical lock problem.

Original: Further, RCGVF significantly outperforms previous methods in the absence of ground-truthepisodic counts in the partially observable MiniGrid environments.
Revised: Further, RCGVF significantly outperforms previous methods in the absence of ground-truth episodic counts in the partially observable MiniGrid environments.

Original: Panoramicobservations on MiniGrid further boost RC-GVF’s performance such that it iscompetitive to baselines exploiting episodic counts.
Revised: Panoramic observations on MiniGrid further boost RC-GVF’s performance such that it is competitive to baselines exploiting episodic counts.

Original:  Introduction
Revised:  Introduction

Original: Efficient exploration in reinforcement learning (RL) is a challenging problem, especially in highdimensional state spaces and sparse-reward environments [64, 2].
Revised: How should an agent efficiently explore environments with high-dimensional state spaces and sparse rewards [70, 2]?

Original: When reward feedback from theenvironment is sparse, an extrinsic reward-maximizing agent makes little progress until it stumblesupon a rewarding sequence of actions.
Revised: An extrinsic reward-maximising agent will make little progress until it stumbles upon a rewarding sequence of actions.

Original: This problem is commonly addressed by providing additionalintrinsic reward to guide the agent’s exploratory behaviour [45, 12, 39].
Revised: In the literature, this issue is commonly addressed by providing additional intrinsic reward to guide the agent’s exploratory behaviour [49, 12, 43].

Original: One class of prominent approaches rely on state novelty , where an intrinsic reward in the formof a ‘novelty bonus’ is awarded based on how often a state has been visited [59, 6].
Revised: One class of prominent approaches rely on state novelty , where an intrinsic reward in the form of a ‘novelty bonus’ is awarded based on how often a state has been visited [65, 6].

Original: More recentworks have extended these approaches to high dimensional state spaces where tabular counts areinapplicable [7, 37, 10].
Revised: More recent works have extended these approaches to high dimensional state spaces where tabular counts are inapplicable [7, 41, 10].

Original: Another class of approaches is based on artificial curiosity  where agents arerewarded in proportion to the prediction errors or information gains of a predictive world model [48,55].
Revised: Another class of approaches is based on artificial curiosity , where agents are rewarded in proportion to the prediction errors or information gains of a predictive world model [52, 60].

Original: Curiosity-based techniques have also been scaled up to handle larger state spaces [21,
Revised: Curiosity-based techniques have also been scaled up to handle larger state spaces [22, 43].

Original: In general, expecting an agent to have access to the complete environment state is often an unrealistic assumption.
Revised: Expecting an agent to have access to the complete environment state is unrealistic.

Original: In such partially observable settings [3, 25], observations may look alike in different states, and the benefit of approximate state novelty bonuses as intrinsic rewards is unclear.
Revised: In such partially observable settings [3, 26], observations may look alike in different states, and the benefit of approximate state novelty bonuses as intrinsic rewards is unclear.

Original: Consider the environment in Figure 1 where an agent is required to traverse a series of blue and white tiles, yet it is only able to observe the current tile it is standing on.
Revised: Consider the environment in Figure 1 where an agent is required to traverse a series of blue and white tiles, yet it is only able to observe the current tile it is standing on.

Original: The colours of the tiles alternate in a regular fashion until the end of the corridor, where there is a surprising sequence of consecutive blue tiles
Revised: The colours of the tiles alternate in a regular fashion until the end of the corridor, where there is a surprising sequence of consecutive blue tiles.

Original: Directly applying state novelty approaches may not be particularly meaningful in this case as can be seen when using random network distillation (RND; [10]).
Revised: Directly applying state novelty approaches may not be particularly meaningful in this case, as can be seen when using random network distillation (RND; [10]).

Original: Here, the agent receives a novelty bonus based on the error of its predictions about the output of a fixed randomly initialised neural network applied to the current observation.
Revised: In RND, the agent receives a novelty bonus based on the error of its predictions about the output of a fixed randomly initialised neural network applied to the current observation.

Original: As shown in the top panel of Figure 1, RND is unable to ascribenovelty when the state pattern changes later in the sequence and the intrinsic reward has vanished.
Revised: As shown in the top panel of Figure 1, RND is unable to ascribe novelty when the state pattern changes later in the sequence and the intrinsic reward has vanished.

Original: Similarly, intrinsic rewards may also vanish prematurely for certain curiosity-based approaches whensingle-step dynamics are simple [38], while longer horizon predictions (as needed in the partiallyobservable case) are susceptible to rapidly compounding errors during sequential rollouts [63].
Revised: Similarly, intrinsic rewards may also vanish prematurely for certain curiosity-based approaches when single-step dynamics are simple [42], while longer horizon predictions (as needed in the partially observable case) are susceptible to rapidly compounding errors during sequential rollouts [69].

Original: [Figure]
Revised: [Figure]

Original: The agent (red triangle) observes the colour of the tile that it currently occupies (white or blue).
Revised: The agent (red triangle) observes the colour of the tile that it currently occupies (white or blue).

Revised: The extrinsic reward is always zero.

Original: At each time-step (frame), the agent moves one tile forward.
Revised: At each timestep (frame), the agent moves one tile forward, until it reaches the last tile.

Original: Unlike our approach (RC-GVF), RND does not generate an intrinsic reward when encountering the surprising last few tiles (step 1000).
Revised: Unlike our approach (RC-GVF), RND does not generate an intrinsic reward when encountering the surprising last few tiles (step 1000).

Original: Here we explore a connection between statenovelty and artificial curiosity to address theselimitations.
Revised: In this paper, we explore a connection between state novelty and artificial curiosity to address these limitations.

Original: We propose random curiosity withgeneral value functions (RC-GVF) as a novel approach to generating intrinsic rewards.
Revised: We propose random curiosity with general value functions (RC-GVF) as a novel approach to generating intrinsic rewards.

Original: It takesinspiration from exploration through temporallyextended experiments that summarise long observation sequences [49].
Revised: It takes inspiration from exploration through temporally extended experiments that summarise long observation sequences [53].

Original: Our approach usesrandom general value functions (GVFs) [61] topose questions about the cumulative future valueof observation-dependent features.
Revised: Our approach uses random general value functions (GVFs) [67] to pose questions about the cumulative future value of observation-dependent features.

Original: Specifically,we train an ensemble of predictors to minimisethe temporal difference (TD) errors of the general value functions and derive an intrinsic reward based on the TD-errors and disagreementsin the long-term predictions.
Revised: Specifically, we train an ensemble of predictors to minimise the temporal difference (TD) errors of the general value functions and derive an intrinsic reward based on the TDerrors and disagreements in the long-term predictions.

Original: The effective horizon of the prediction task can be controlled viathe discount factor, where a zero discount canbe related to state novelty with random networkdistillation (RND) [10].
Revised: The effective horizon of the prediction task can be controlled via the discount factor, where a zero discount can be related to state novelty with random network distillation (RND) [10].

Original: We hypothesise thatpredicting information from an temporally extended horizon improves exploration in POMDPs andguards against premature vanishing of intrinsic rewards due to the increase in difficulty of the auxiliarytask.
Revised: We hypothesise that predicting information from an temporally extended horizon improves exploration in POMDPs and guards against premature vanishing of intrinsic rewards due to the increase in difficulty of the auxiliary task.

Original: Indeed, in our toy experiment of Figure 1 it can be see how RC-GVF manages to generate aspike of intrinsic reward at the end of the sequence.
Revised: Indeed, in our toy experiment of Figure 1 it can be seen how RC-GVF manages to generate a spike of intrinsic reward at the end of the sequence.

Original: We evaluate our approach on sparse-reward and partially observable RL problems.
Revised: We evaluate our approach on sparse-reward and partially observable RL problems.

Original: Our results ona hard exploration diabolical lock problem [32] and the Minigrid suite of environments [13] revealthe benefits of considering extended horizon predictions in comparison to approaches that rely onimmediate partial observations to generate exploration bonuses.
Revised: Our results on a hard exploration diabolical lock problem [36] and the Minigrid suite of environments [13] reveal the benefits of considering extended horizon predictions in comparison to approaches that rely on immediate partial observations to generate exploration bonuses.

Original: Furthermore, existing baselines areheavily reliant on privileged information about episodic state-visitation counts from the simulator,which is commonly used to scale the exploration bonus.
Revised: Furthermore, existing baselines in Minigrid are heavily reliant on privileged information about episodic state-visitation counts from the simulator, which is commonly used to scale the exploration bonus.

Original: We demonstrate that RC-GVF succeeds inmany environments without the use of such episodic counts.
Revised: We demonstrate that RC-GVF succeeds in many environments without the use of such episodic counts.

Original:  Preliminaries
Revised:  Preliminaries

Original: Reinforcement learning in POMDPs We consider the scenario where an agent interacts with a
Revised: Reinforcement learning in POMDPs We consider the scenario where an agent interacts with a

Original: Partially Observable Markov Decision Process (POMDP) with time steps t ∈ N , with environmentstates S t ∈ S , observations O t ∈
Revised: Partially Observable Markov Decision Process (POMDP) with time steps t ∈ N , with environment states S t ∈ S , observations O t ∈

Original: O , actions A t ∈ A , extrinsic rewards R e : S → [0 , 1] , andpolicies that map histories of different lengths to distributions over actions π : H → ∆ ( A ) where
Revised: O , actions A t ∈ A , extrinsic rewards R e : S → [0 , 1] , and policies that map histories of different lengths to distributions over actions π : H → ∆ ( A ) where

Original: H t =
Revised: H t =

Original: O 1: t ∈ H [25].
Revised: O 1: t ∈ H [26].

Original: The agent only observes O t at time step t , which may not fully describe the
Revised: The agent only observes O t at time step t , which may not fully describe the

Original: MDP state, and thus it becomes necessary to condition the agent on the history H t .
Revised: MDP state, and thus it becomes necessary to condition the agent on the history H t .

Original: The learning objective is to find the optimal policy π ∗ that maximizes the expected discounted return
Revised: The learning objective is to find the optimal policy π ∗ that maximizes the expected discounted return

Original: [Equation]
Revised: [Equation]

Original: [Equation] where γ ∈ [0 , 1) is the discount factor and the expectation is over stochastic components in theenvironment and the policy.
Revised: where γ ∈ [0 , 1) is the discount factor and the expectation is over stochastic components in the environment and the policy.

Original: Random Network Distillation In random network distillation (RND) [10] the agent receives astate novelty reward proportional to the error of making predictions ˆ z ( o t ) where the targets aregenerated by a randomly initialized neural network Z ϕ : O → R d .
Revised: Random Network Distillation In random network distillation (RND) [10] the agent receives a state novelty reward proportional to the error of making predictions ˆ z ( o t ) where the targets are generated by a randomly initialized neural network Z ϕ : O → R d .

Original: The RND intrinsic reward for anobservation is given by
Revised: The RND intrinsic reward for an observation is given by

Original: [Equation]
Revised: [Equation]

Original: RND-like error signals can be a simple way to obtain effective uncertainty estimates about targets ata given input, which could subsequently be used for exploration in RL or detecting out of distributionsamples [14].
Revised: RND-like error signals can be a simple way to obtain effective uncertainty estimates about targets at a given input, which could subsequently be used for exploration in RL or detecting out of distribution samples [14].

Original: Beyond its simplicity, an appealing aspect of using RND is that it allows us to implicitlyincorporate prior beliefs about useful rewards through the neural network architecture [34].
Revised: Beyond its simplicity, an appealing aspect of using RND is that it allows us to implicitly incorporate prior beliefs about useful rewards through the neural network architecture [38].

Original: General Value Functions
Revised: General Value Functions

Original: A general value function (GVF) [61] is defined by a policy π , a cumulantor pseudo-reward function Z : O → R , and a discount factor γ z ∈ [0 , 1) .
Revised: A general value function (GVF) [67] is defined by a policy π , a cumulant or pseudo-reward function Z : O → R , and a discount factor γ z ∈ [0 , 1) .

Original: It can be expressed as
Revised: It can be expressed as

Original: [Equation]

Original: π

Original: [Equation]
Revised: [Equation]

Original: General value functions extend the concept of predicting expected cumulative values to arbitrarysignals beyond the reward.
Revised: General value functions extend the concept of predicting expected cumulative values to arbitrary signals beyond the reward.

Original: This can be viewed as ‘answering’ questions about cumulative quantitiesof interest under a particular policy π and discount factor γ z .
Revised: They can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy π and discount factor γ z .

Original: Predictions from GVFs have previouslybeen used as features for state representation [43] or to specify auxiliary tasks that assist in shapingrepresentations for the main task [23, 66].
Revised: Predictions from GVFs have previously been used as features for state representation [47] or to specify auxiliary tasks that assist in shaping representations for the main task [24, 73].

Original:  Random Curiosity with General Value Functions
Revised:  Random Curiosity with General Value Functions

Original: In the spirit of artificial curiosity [49], we are interested in predicting long-term outcomes in theenvironment under a particular policy as a means for exploration.
Revised: In the spirit of artificial curiosity [53], we are interested in predicting long-term outcomes in the environment under a particular policy as a means for exploration.

Original: Our approach, RC-GVF , rewardsan agent for taking actions that lead to higher uncertainty about the future.
Revised: Our approach, RC-GVF , rewards an agent for taking actions that lead to higher uncertainty about the future.

Original: However, instead ofpredicting the entire sequence of future states, we consider a set of random numeric questions aboutthe environment and capture uncertainty about their outcome using general value functions (GVFs).
Revised: However, instead of predicting the entire sequence of future states, we consider a set of random numeric questions about the environment and capture uncertainty about their outcome using general value functions (GVFs).

Original: 3.1 General value functions to predict the future
Revised: 3.1 General value functions to predict the future

Original: The central piece of RC-GVF is the prediction of temporally extended outcomes in the environment.
Revised: The central piece of RC-GVF is the prediction of temporally extended outcomes in the environment.

Original: This is different from only predicting a feature of the current state (as in state novelty exploration[59]) or dynamics models that predict the entire (sequence of) next observation(s) (as in early versionsof artificial curiosity [48]).
Revised: This is different from only predicting a feature of the current state (as in state novelty exploration [65]) or dynamics models that predict the entire (sequence of) next observation(s) (as in early versions of artificial curiosity [52]).

Original: At time step t , the current observation o t is mapped to a collection ofpseudo-rewards z t +1 ∈ R
Revised: At time step t , the current observation o t is mapped to a collection of pseudo-rewards z t +1 ∈ R

Original: d which together with the policy and discount factor define a question thatcan be asked about the future: “what is the expected discounted cumulative sum of these pseudorewards under a given policy?”.
Revised: d which together with the policy and discount factor define a question that can be asked about the future: “what is the expected discounted cumulative sum of these pseudorewards under a given policy?”.

Original: The answer to this question (i.e. the prediction target) is based onoutcomes in the environment, as in Equation 3.
Revised: The answer to this question (i.e. the prediction target) is based on outcomes in the environment, as in Equation 3.

Original: In this paper, we investigate whether a set of randompseudo rewards are sufficient to drive exploration in this way.
Revised: In this paper, we investigate whether a set of random pseudo rewards are sufficient to drive exploration in this way.

Original: Previously it was found that random features extracted by a neural network are often sufficient tocapture useful properties of the input [41].
Revised: Previously it was found that random features extracted by a neural network are often sufficient to capture useful properties of the input [45].

Original: Similarly, neural network architectures can be used toexpress prior knowledge about useful features [65, 14].
Revised: Similarly, neural network architectures can be used to express prior knowledge about useful features [71, 14].

Original: To this end, we generate pseudo-rewardsfrom a fixed and randomly initialised neural network Z ϕ : O → R d with parameters ϕ that mapsobservations to pseudo-rewards.
Revised: To this end, we generate pseudo-rewards from a fixed and randomly initialised neural network Z ϕ : O → R d with parameters ϕ that maps observations to pseudo-rewards.

Original: This choice also bypasses the difficult problem of discoveringmeaningful general value functions [66].
Revised: This choice also bypasses the difficult problem of discovering meaningful general value functions [73].

Original: Training the GVF predictors Our exploration mechanism is to reward the agent for takingactions that generate previously unknown outcomes.
Revised: Training the GVF predictors Our exploration mechanism is to reward the agent for taking actions that generate previously unknown outcomes.

Original: To facilitate this, we train a separate (recurrent)neural network – which we call the predictor – to predict these values.
Revised: To facilitate this, we train a separate (recurrent) neural network–which we call the predictor –to predict these values.

Original: Concretely, a predictorˆ v π,z : H → R d maps histories (of observations, actions, and pseudo rewards) to pseudo-values.
Revised: Concretely, a predictor ˆ v π,z : H → R d maps histories (of observations, actions, and pseudo rewards) to pseudo-values.

Original: The predictor is trained on-policy, implying that the GVFs are evaluated under the current policy.
Revised: The predictor is trained on-policy, implying that the GVFs are evaluated under the current policy.

Original: One motivating factor for this choice is that it couples the prediction task to the current policy, thuscreating an incentive to vary behaviour for additional exploration [18].
Revised: One motivating factor for this choice is that it couples the prediction task to the current policy, thus creating an incentive to vary behaviour for additional exploration [18].

Original: We use the (truncated)
Revised: We use the (truncated)

Original: λ -return as the target for the predictor, which can be recursively expressed as
Revised: λ -return as the target for the predictor, which can be recursively expressed as

Original: [Equation]
Revised: [Equation]

Original: Here, λ z ∈ [0 , 1] is the usual parameter that allows balancing the bias-variance trade off by interpolating between TD(0) and Monte Carlo estimates of the pseudo-return [58].
Revised: Here, λ z ∈ [0 , 1] is the usual parameter that allows balancing the bias-variance trade off by interpolating between TD(0) and Monte Carlo estimates of the pseudo-return [64].

Original: The predictor is trainedto minimise the mean squared TD the λ - target notation
Revised: The predictor is trained to minimise the mean squared TD-error with the λ -return target.

Original:  we will 
Revised: For convenience in notation, we will denote G zt ( λ z ) as G zt , despite its dependence on λ z .

Original: 3.2 Disagreement and prediction error as intrinsic reward
Revised: 3.2 Disagreement and prediction error as intrinsic reward

Original: To generate an intrinsic reward, a straightforward choice could be to consider the error between thetemporal difference target (from Equation 4) and the predictor’s output at the current observation:
Revised: To generate an intrinsic reward, a straightforward choice could be to consider the error between the temporal difference target (from Equation 4) and the predictor’s output at the current observation:

Original: [Equation]
Revised: [Equation]

Original: However, the presence of aleatoric uncertainty, is a problem that arises as a consequence of extendingthe horizon of the predictive task.
Revised: However, the presence of aleatoric uncertainty, is a problem that arises as a consequence of extending the horizon of the predictive task.

Original: From an exploration perspective, the agent should focus on thereducible epistemic uncertainty, and not the irreducible aleatoric uncertainty.
Revised: From an exploration perspective, the agent should focus on the reducible epistemic uncertainty, and not the irreducible aleatoric uncertainty.

Original: Directly minimising
Revised: Directly minimising

Original: Equation 5 with the predictor would ignore the inherent variance in the TD-target (due to stochasticityin the policy and environment), and thus using the prediction error of the GVF target as an intrinsicreward does not distinguish between aleatoric and epistemic uncertainty [15, 22].
Revised: Equation 5 with the predictor would ignore the inherent variance in the TD-target (due to stochasticity in the policy and environment), and thus using the prediction error of the GVF target as an intrinsic reward does not distinguish between aleatoric and epistemic uncertainty [15, 23].

Original: Another way to recognize this is by decomposing the expected prediction error as described in
Revised: Another way to recognize this is by decomposing the expected prediction error as described in

Original: Jain et al.
Revised: Jain et al.

Original: The expected loss of a predictor ˆ f ( · ) at an input o , E [ L ( t, ˆ f ( o ))] , can be split intoepistemic ( E ) and aleatoric ( A ) components: (cid:82) L ( t, ˆ f ( o ))
Revised: The expected loss of a predictor ˆ f ( · ) at an input o , E [ L ( t, ˆ f ( o ))] , can be split into epistemic ( E ) and aleatoric ( A ) components: (cid:82) L ( t, ˆ f ( o ))

Original: dP ( t | o ) =
Revised: dP ( t | o ) =

Original: E ( ˆ f, o ) + A ( o ) , where P ( t | o )is the conditional distribution over targets for input o and L ( t, · ) is a chosen loss function.
Revised: E ( ˆ f, o ) + A ( o ) , where P ( t | o ) is the conditional distribution over targets for input o and L ( t, · ) is a chosen loss function.

Original: Thealeatoric uncertainty at input o , A ( o ) , is defined as the expected prediction error of a Bayes optimalpredictor function.
Revised: The aleatoric uncertainty at input o , A ( o ) , is defined as the expected prediction error of a Bayes optimal predictor function.

Original: E ( ˆ f, o ) denotes the epistemic uncertainty of a predictor at o .
Revised: E ( ˆ f, o ) denotes the epistemic uncertainty of a predictor at o .

Original: To overcome this issue with aleatoric uncertainty in RC-GVF, we propose to train an ensemble ofpredictors and utilise the variance across their predictions as a multiplicative factor on the predictionerror.
Revised: To overcome this issue with aleatoric uncertainty in RC-GVF, we propose to train an ensemble of predictors and utilise the variance across their predictions as a multiplicative factor on the prediction error.

Original: Concretely, we train K predictors ˆ v kπ,z , k ∈ { 1 , 2 . . .
Revised: Concretely, we train K predictors ˆ v kπ,z , k ∈ { 1 , 2 . . .

Original: K } .
Revised: K } .

Original: The prediction target for eachmember of the ensemble is the same λ -pseudo-return (Equation 4) by using bootstrapped values fromthat member’s predictions.
Revised: The prediction target for each member of the ensemble is the same λ -pseudo-return (Equation 4) by using bootstrapped values from that member’s predictions.

Original: Using the ensemble of predictors, the intrinsic reward is given by
Revised: Using the ensemble of predictors, the intrinsic reward is given by

Original: [Equation]
Revised: [Equation]

Original: where ⊙ corresponds to element-wise multiplication.
Revised: where ⊙ corresponds to element-wise multiplication.

Original: In this formulation, even when prediction errorremains, the exploration bonus will vanish when the predictors converge to the same expectation.
Revised: In this formulation, even when prediction error remains, the exploration bonus will vanish when the predictors converge to the same expectation.

Original: 3.3 Effective horizon and relation to random network distillation
Revised: 3.3 Effective horizon and relation to random network distillation

Original: The effective horizon over which predictions are considered depends on the choice of the discountfactor γ z .
Revised: The effective horizon over which predictions are considered depends on the choice of the discount factor γ z .

Original: It can be shown that for a horizon H γ z ,ε ∈ R , such that
Revised: It can be shown that for a horizon H γ z ,ε ∈ R , such that

Original: [Equation]
Revised: [Equation]

Original: the discounted sum of (pseudo) rewards beyond this horizon is bounded by ε [27, 28].
Revised: the discounted sum of (pseudo) rewards beyond this horizon is bounded by ε [28, 30].

Original: From Equation 4 we can see that for the special case of γ z = 0 and any choice of λ z , we have
Revised: From Equation 4 we can see that for the special case of γ z = 0 and any choice of λ z , we have

Original: G zt
Revised: G zt

Original: =
Revised: =

Original: Z t +1 =
Revised: Z t +1 =

Original: Z ϕ ( o t ) .
Revised: Z ϕ ( o t ) .

Original: Note that in this case using the TD-error between the predictor and target isequivalent to intrinsic rewards provided by random network distillation (RND; Equation 2).
Revised: Note that in this case using the TD-error between the predictor and target is equivalent to intrinsic rewards provided by random network distillation (RND; Equation 2).

Original: Since theprediction targets are now deterministic, there is no issue of having aleatoric uncertainty, and a singlepredictor suffices (as opposed to using an ensemble).
Revised: Since the prediction targets are now deterministic, there is no issue of having aleatoric uncertainty, and a single predictor suffices (as opposed to using an ensemble).

Original:  Related work
Revised:  Related work

Original: Exploration In tabular RL, explicit counts for states or state-action pairs can be maintained toenable exploration and achieve efficient learning [56, 4].
Revised: Exploration In tabular RL, explicit counts for states or state-action pairs can be maintained to enable exploration and achieve efficient learning [61, 4].

Original: Count-based bonuses have been scaled tolarger state spaces through the use of density models, which provide pseudo-counts [7, 37].
Revised: Count-based bonuses have been scaled to larger state spaces through the use of density models, which provide pseudo-counts [7, 41].

Original: Statenovelty bonuses are similarly inspired heuristics that provide a bonus based on estimated novelty ofthe visited state [10].
Revised: State novelty bonuses are similarly inspired heuristics that provide a bonus based on estimated novelty of the visited state [10].

Original: Perhaps the simplest way of implementing exploration through ‘curiosity’ is to reward the policyfor encountering transitions to states that surprise a learning predictor [48].
Revised: Perhaps the simplest way of implementing exploration through ‘curiosity’ is to reward the policy for encountering transitions to states that surprise a learning predictor [52].

Original: This is suitable fordeterministic environments but it suffers from the ‘noisy TV problem’ in stochastic environments [52,9].
Revised: This is suitable for deterministic environments but it suffers from the ‘noisy TV problem’ in stochastic environments [56, 9].

Original: To overcome this limitation, intrinsic rewards have been defined by learning progress (the firstderivative of the prediction error)
Revised: To overcome this limitation, intrinsic rewards have been defined by learning progress (the first derivative of the prediction error)

Original: [47], information gain [55, 21], or compression progress [51].
Revised: [51], information gain [60, 22], or compression progress [55].

Original: Also compare such bonuses to pseudo-counts [7].
Revised: Also compare such bonuses to pseudo-counts [7].

Original: Alternatively, it is possible to mitigate sensitivityto noise by measuring prediction errors in a latent space [49, 39].
Revised: Alternatively, it is possible to mitigate sensitivity to noise by measuring prediction errors in a latent space [53, 43].

Original: Our method(RC-GVF) is different from state count and novelty approaches in that it modelstemporally extended values beyond the current observation.
Revised: Our method is different from state count and novelty approaches in that it models temporally extended values beyond the current observation.

Original: It includes a state-novelty measure, RND [10], as a special case when the GVF discount factors are zero.
Revised: It includes a state-novelty measure, RND [10], as a special case when the GVF discount factors are zero.

Original: Unlike most artificial curiosityapproaches, RC-GVF does not model the entire environment dynamics, in line with the idea ofconducting temporally extended experiments whose outcomes are abstract summaries (represented inlatent variables) of long observation sequences [50].
Revised: Unlike most artificial curiosity approaches, RC-GVF does not model the entire environment dynamics, in line with the idea of conducting temporally extended experiments whose outcomes are abstract summaries (represented in latent variables) of long observation sequences [54].

Original: We mitigate sensitivity to aleatoric uncertaintyby measuring the disagreement of an ensemble of predictors, inspired by prior work (for single-stepprediction models) [54, 40].
Revised: We mitigate sensitivity to aleatoric uncertainty by measuring the disagreement of an ensemble of predictors, inspired by prior work (for single-step prediction models) [59, 44].

Original: Ensembles for exploration have also been explored in the contextof posterior sampling [57].
Revised: Ensembles for exploration have also been explored in the context of posterior sampling [62].

Original: Bootstrapped-DQN [34] uses an ensemble of Q-value functions for amodel-free interpretation of posterior sampling [35].
Revised: Bootstrapped-DQN [38] uses an ensemble of Q-value functions for a model-free interpretation of posterior sampling [39].

Original: Unlike in our approach, these value functionsare restricted to the original task rewards, and can not capture arbitrary pseudo-rewards.
Revised: Unlike in our approach, these value functions are restricted to the original task rewards, and can not capture arbitrary pseudo-rewards.

Revised: Recent works suggest incorporating model-based planning to directly optimise for long-term novelty [63, 58, 31].

Revised: Our intrinsic reward–which is generated by considering multiple future steps–could motivate approaches for optimising long-term novelty without explicit rollout-based planning.

Original: General Value Functions Sutton et al.
Revised: General Value Functions Sutton et al.

Original: [61] proposed general value functions (GVFs) as anapproach to represent predictive knowledge about the world.
Revised: [67] proposed general value functions (GVFs) as an approach to represent predictive knowledge about the world.

Original: Closely related is the vector-valuedadaptive critic [46] which predicts and controls cumulative values of special input vectors.
Revised: Closely related is the vector-valued adaptive critic [50] which predicts and controls cumulative values of special input vectors.

Original: Severalworks have studied the impact of using auxiliary value functions to improve representation learningin RL [23, 8].
Revised: Several works have studied the impact of using auxiliary value functions to improve representation learning in RL [24, 8, 35].

Original: Further, it was previously found that GVFs of randomly generated pseudo-rewardscan also be useful for shaping and learning representations [31, 70] (see also related work onforecasts [43], TD-networks [60] and predictive state representations [29]).
Revised: Further, it was previously found that GVFs of randomly generated pseudo-rewards can also be useful for shaping and learning representations [34, 78] (see also related work on forecasts [47], TD-networks [66] and predictive state representations [32]).

Original: This can be viewedas ‘answering’ questions about cumulative quantities of interest under a particular policy π anddiscount factor γ z .
Revised: This can be viewed as ‘answering’ questions about cumulative quantities of interest under a particular policy π and discount factor γ z .

Original: In our implementation, the predictor is an entirely separate network and thus
Revised: In our implementation, the predictor is an entirely separate network and thus

Original: GVF prediction can not be used to improve the agent’s internal representation.
Revised: GVF prediction can not be used to improve the agent’s internal representation.

Original: Instead, we use the
Revised: Instead, we use the

Original: GVF prediction to derive a useful signal to guide exploration.
Revised: GVF prediction to derive a useful signal to guide exploration.

Original: The discovery/selection of useful GVF ‘questions‘ is an open problem [44].
Revised: The discovery/selection of useful GVF ‘questions’ is an open problem [48].

Original: Existing approaches learnquestions related to optimizing performance on the main task [66, 26].
Revised: Existing approaches learn questions related to optimizing performance on the main task [73, 27].

Original: However, such an approachmight not be suitable for sparse-reward environments.
Revised: However, such an approach might not be suitable for sparse-reward environments.

Original: In our approach, we side-step the GVFdiscovery problem through the use of randomly generated pseudo-rewards and on-policy predictions.
Revised: In our approach, we side-step the GVF discovery problem through the use of randomly generated pseudo-rewards and on-policy predictions.

Original:  Experiments
Revised:  Experiments

Original: In this section we present an empirical evaluation of RC-GVF.
Revised: In this section we present an empirical evaluation of RC-GVF 2 .

Original: First, we consider a hard explorationdiabolical lock problem from the literature [32] and demonstrate the benefits of predicting beyondthe immediate observation.
Revised: First, we consider a hard exploration diabolical lock problem from the literature [36] and demonstrate the benefits of predicting beyond the immediate observation.

Original: We then proceed to experiments with the standard Minigrid suite ofpartially observable and procedurally generated environments, demonstrating how RC-GVF is ableto outperform several strong baselines [18, 69].
Revised: We then proceed to experiments with the standard Minigrid suite of partially observable and procedurally generated environments, demonstrating how RC-GVF is able to outperform several strong baselines [18, 77].

Original: 5.1 Diabolical locks
Revised: 5.1 Diabolical locks

Original: [Table] [Table] The diabolical lock problem [32] is considered adifficult environment for exploration due to noisyhigh-dimensional observations, stochastic dynamics and anti-shaped rewards that deter the agentfrom finding the sparse optimal reward at the endof the lock.
Revised: The diabolical lock problem [36] is considered a difficult environment for exploration due to noisy high-dimensional observations, stochastic dynamics and anti-shaped rewards that deter the agent from finding the sparse optimal reward at the end of the lock.

Original: In the literature, RND is often shownto fail on these types of environments due to onlyconsidering the immediate future [32].
Revised: In the literature, RND is often shown to fail on these types of environments due to only considering the immediate future [36].

Original: Due to
Revised: Due to

Original: RC-GVFs policy-conditioned predictions of futurequantities , this task is instructive to demonstratethe qualitative advantages of our method and how this results in markedly different performance. 
Revised: RC-GVFs policy-conditioned predictions of future quantities, this task is instructive to demonstrate the qualitative advantages of our method and how this results in markedly different performance. [Table]

Original: Environment details The environment consists of states organised in 3 rows { a, b, c } and Hcolumns from { 1 , 2 , . . .
Revised: Environment details The environment consists of states organised in 3 rows { a, b, c } and H columns from { 1 , 2 , . . .

Original: H } (see Figure 5 in Appendix A.1).
Revised: H } (see Figure 6 in Appendix A.1).

Original: In each episode, the agent is initialisedin one of two possible starting states, either at a 1 or b 1 .
Revised: In each episode, the agent is initialised in one of two possible starting states, either at a 1 or b 1 .

Original: At each state the agent has L available actions,only one of which is ‘good’ and transitions to a ‘good’ state in the next column with equal probability.
Revised: At each state the agent has L available actions, only one of which is ‘good’ and transitions to a ‘good’ state in the next column with equal probability.

Original: Taking the good action gives a negative (anti-shaped) reward of − 1 /H , except at the end of the lock(states a H or b H ), where the sparse optimal reward of 10 is received.
Revised: Taking the good action gives a negative (anti-shaped) reward of − 1 /H , except at the end of the lock (states a H or b H ), where the sparse optimal reward of 10 is received.

Original: The remaining L − 1 actions atany state are ‘bad’, causing a deterministic transition to the ‘dead’ row c at the next column (zeroreward).
Revised: The remaining L − 1 actions at any state are ‘bad’, causing a deterministic transition to the ‘dead’ row c at the next column (zero reward).

Revised: All actions from a dead state lead to the dead state in the next row, and hence the optimal policy is to take the respective good actions in each state.

Revised: The good action at each state is assigned randomly as part of the MDP specification.

Original: The agent never observes the state directly andis only given access to a high-dimensionalnoisy observation (details in Appendix A.1).
Revised: The agent never observes the state directly and only has access to a high-dimensional noisy observation (details in Appendix A.1).

Original: In our experiment, we consider a problem with horizon H = 100 , L = 10 , and observationnoise σ o = 0 .
Revised: In our experiment, we consider a problem with horizon H = 100 , L = 10 , and observation noise σ o = 0 .

Revised: As discussed by Misra et al.

Original: This exploration problem is hard because the probability of reaching the optimalreward through a uniformly random policy is L − H ( 10 − 100 in our instance).
Revised: [36], this exploration problem is hard because the probability of reaching the optimal reward through a uniformly random policy is L − H ( 10 − 100 in our instance).

Original: Further, the stochastictransitions from good states prevent a solution that relies on memorising a  successful sequence of H actions independent of the state.
Revised: Further, the stochastic transitions from good states prevent a solution that relies on memorising a state-independent successful sequence of H actions independent of the state.

Original: Implementation We use PPO in an actor-critic framework as the base agent and use 128 pseudorewards for RND and RC-GVF.
Revised: Implementation We use Proximal Policy Optimization (PPO) [57] in an actor-critic framework as the base agent and use 128 pseudo-rewards for RND and RC-GVF.

Original: For RC-GVF we set γ z = 0 .
Revised: For RC-GVF we set γ z = 0 .and

Original: and use two prediction heads in theensemble.
Revised: use two prediction heads in the ensemble.

Original: Further details about the neural network architectures and training are in Appendix A.1.
Revised: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards. Further details are in Appendix A.1.

Original: Results Table 1 compares RND to RC-GVF in this environment.
Revised: Results Table 1 compares RND to RC-GVF in this environment.

Original: It can be seen how RC-GVF succeeds in completing the lock on most of the runs (7 out of 10), while the best run of RND only reaches60% of the lock (and does not complete it).
Revised: It can be seen how RC-GVF succeeds in completing the lock on all runs, while the best run of RND only reaches 60% of the lock (and does not complete it).

Original: Indeed, once the agent takes an incorrect action and ‘falls’into the bottom row, the future consequences (and thus, the GVFs) become highly predictable.
Revised: Indeed, once the agent takes an incorrect action and ‘falls’ into the bottom row, the future consequences (and thus, the GVFs) become highly predictable.

Original: Bystaying alive, the agent will encounter new observations and unpredictable transitions which appearinteresting.
Revised: By staying alive, the agent will encounter new observations and unpredictable transitions which appear interesting.

Revised: Further experiments with additional baselines are presented in Appendix D.

Original: This result is illustrative of RC-GVFs exploration capabilities as we will see next.
Revised: This result is illustrative of RC-GVFs exploration capabilities as we will see next.

Original: 5.2 MiniGrid
Revised: 5.2 MiniGrid

Original: We evaluate RC-GVF on procedurally generated environments from MiniGrid [13], which is a standard benchmark in the deep reinforcement learning literature for exploration [42, 11, 18, 69, 68, 38].
Revised: We evaluate RC-GVF on procedurally generated environments from MiniGrid [13], which is a standard benchmark in the deep reinforcement learning literature for exploration [46, 11, 18, 77, 76, 42].

Original: Exploration in these environments is challenging due to partial observability, extremely sparserewards, and the procedural generation of mazes and objects.
Revised: Exploration in these environments is challenging due to partial observability, extremely sparse rewards, and the procedural generation of mazes and objects.

Original: Environment Details Broadly speaking, we will consider three classes of environments organisedaccording to several difficulty levels.
Revised: Environment Details Broadly speaking, we will consider three classes of environments organised according to several difficulty levels.

Original: First, we study MultiRoom-N7-S8 and MultiRoom-N12-S10 ,which are navigation tasks in a maze with seven and twelve rooms respectively.
Revised: First, we study MultiRoom-N7-S8 and MultiRoom-N12-S10 , which are navigation tasks in a maze with seven and twelve rooms respectively.

Original: Next, we considertwo levels of difficulty in KeyCorridor-S4R3 and KeyCorridor-S5R3 .
Revised: Next, we consider two levels of difficulty in KeyCorridor-S4R3 and KeyCorridor-S5R3 .

Original: Here, the agent needs to pickup a ball behind a locked door.
Revised: Here, the agent needs to pick up a ball behind a locked door.

Original: Finally, we examine the ObstructedMaze set of environments wherethe agent has a similar task of picking up a ball behind a locked door, but keys are hidden in boxes( ObstructedMaze-2Dlh ), and doors can be obstructed ( ObstructedMaze-2Dlhb ). 
Revised: Finally, we examine the ObstructedMaze set of environments where the agent has a similar task of picking up a ball behind a locked door, but keys are hidden in boxes ( ObstructedMaze-2Dlh ), and doors can be obstructed ( ObstructedMaze-2Dlhb ). The observations [Figure]

Original: The observationscorresponds to an egocentric view of the cells in the front of the agent.
Revised: t u [Figure] corresponds to an egocentric view of the cells in the front of the agent.

Original: Further details about the
Revised: Further details about the

Original: MiniGrid environments are available in Appendix A.2.
Revised: MiniGrid environments are available in Appendix A.2.

Original: Implementation We compare RC-GVF to RND [10], AGAC [18], and NovelD [69].
Revised: Implementation We compare RC-GVF to RND [10], AGAC [18], and NovelD [77].

Original: We use
Revised: We use

Original: Proximal Policy Optimization (PPO) [53] as our base agent for all approaches.
Revised: Proximal Policy Optimization (PPO) [57] as our base agent for all approaches.

Original: The agent is trainedto maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.
Revised: The agent is trained to maximize the expected sum of a weighted combination of intrinsic and extrinsic rewards.

Original: Ateach time step t , the agent receives a reward R t =
Revised: At each time step t , the agent receives a reward R t =

Original: R e ( s t ) + βR i ( o t ) , where β ∈ R + balances thecontribution of the extrisic R e ( s t ) and intrinsic R i ( o t ) rewards.
Revised: R e ( s t ) + βR i ( o t ) , where β ∈ R + balances the contribution of the extrisic R e ( s t ) and intrinsic R i ( o t ) rewards.

Original: For RC-GVF we set γ z = 0 . 6 , usetwo prediction heads in the ensemble of predictors and use 128 pseudo-rewards (same as for RND).
Revised: For RC-GVF we set γ z = 0 . 6 , use two prediction heads in the ensemble of predictors and use 128 pseudo-rewards (same as for RND).

Original: Other important hyper-parameters, such as the intrinsic reward coefficient ( β ), entropy coefficient,and learning rate of the predictor are obtained via an extensive hyper-parameter search for allbaselines (see Appendices C.2 and C.3 for details, including on our implementation of baselines.).
Revised: Other important hyper-parameters, such as the intrinsic reward coefficient ( β ), entropy coefficient, and learning rate of the predictor are obtained via an extensive hyper-parameter search for all baselines (see Appendices C.2 and C.3 for details, including on our implementation of baselines.).

Original: We present results averaged over 10 independent runs for each approach in every figure (solid line).
Revised: We present results averaged over 10 independent runs for each approach in every figure (solid line).

Original: An exception is the ObstructedMaze-2Dlhb environment for which we report results with 5 seeds.

Original: Unless mentioned otherwise, the shading indicates 95% bootstrapped confidence intervals.[Figure] [Figure]
Revised: Unless mentioned otherwise, the shading indicates 95% bootstrapped confidence intervals.

Original: 5.2.1 Egocentric observations
Revised: 5.2.1 Egocentric observations

Original: We will first consider the usual setting where the agent receives access to the egocentric observationsfrom the environment.
Revised: We will first consider the usual setting where the agent receives access to the egocentric observations from the environment.

Original: Importantly, we do not provide agents with access to privileged informationabout the environment in the form of so-called ‘episodic counts’.
Revised: Importantly, we do not provide agents with access to privileged information about the environment in the form of so-called ‘episodic counts’.

Original: This is different from severalrecent approaches that have been applied to MiniGrid, which incorporate these values as part of theirintrinsic reward [42, 18, 69].
Revised: This is different from several recent approaches that have been applied to MiniGrid, which incorporate these values as part of their intrinsic reward [46, 18, 77].

Original: The use of episodic counts obtained through the simulator is problematic for solving explorationproblems in a partially observable setting, since it tells the agent precisely how frequently it has beenin each state.
Revised: The use of episodic counts obtained through the simulator is problematic for solving exploration problems in a partially observable setting, since it tells the agent precisely how frequently it has been in each state.

Original: Indeed, the use of episodic counts alone can be sufficient for exploration in MiniGrid(Figure 6 in Appendix D.1).
Revised: Indeed, the use of episodic counts alone can be sufficient for exploration in MiniGrid (Figure 7 in Appendix E.1).

Original: Meanwhile, there exists no good method for obtaining episodic countsin the absence of a simulator, as it is difficult to accurately estimate pseudo-counts from partialobservations (eg.
Revised: Meanwhile, there exists no good method for obtaining episodic counts in the absence of a simulator, as it is difficult to accurately estimate pseudo-counts from partial observations (eg.

Original: as in Figure 1).
Revised: as in Figure 1).

Original: Details about the exact usage of episodic counts by baselines intheir intrinsic reward formulation are provided in Appendix C.2.
Revised: Details about the exact usage of episodic counts by baselines in their intrinsic reward formulation are provided in Appendix C.2.

Original: In Figure 2 we compare RC-GVF, RND, AGAC, and NovelD in the absence of episodic counts,while using egocentric observations.
Revised: In Figure 2 we compare RC-GVF, RND, AGAC, and NovelD in the absence of episodic counts, while using egocentric observations.

Original: It can be seen that RC-GVF explores faster and succeeds moreoften than RND and the other baselines in the KeyCorridor-S4R3 environment and MultiRoomN7-S8 environment.
Revised: It can be seen that RC-GVF explores faster and succeeds more often than RND and the other baselines in the KeyCorridor-S4R3 environment and MultiRoomN7-S8 environment.

Original: In comparison to RND and NovelD that utilise prediction errors in immediaterandom embeddings, it appears that RC-GVF benefits from extended horizon predictions of randompseudo-rewards.
Revised: In comparison to RND and NovelD that utilise prediction errors in immediate random embeddings, it appears that RC-GVF benefits from extended horizon predictions of random pseudo-rewards.

Original: Meanwhile, AGAC only succeeds in ObstructedMaze-2Dlh, where all baselinesperform similarly well.
Revised: Meanwhile, AGAC only succeeds in ObstructedMaze-2Dlh, where all baselines perform similarly well.

Original: Finally, we observe that all approaches struggle with the most difficult levelof considered environments (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb). 
Revised: Finally, we observe that all approaches struggle with the most difficult level of considered environments (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb). [Figure]

Revised: -GVF (using panoramic observations) achieves similar performance without.

Original: 5.2.2 Panoramic observations
Revised: 5.2.2 Panoramic observations

Original: We now demonstrate how using episodic counts from the simulator is crucial to the success of the baselines.
Revised: We now demonstrate how using episodic counts from the simulator is crucial to the success of the baselines.

Original: In Figure 3 it can be seen how AGAC and NovelD improve with episodic counts, and also solvethe more difficult problems (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).
Revised: In Figure 3 it can be seen how AGAC and NovelD improve with episodic counts, and also solve the more difficult problems (KeyCorridor-S5R3, MultiRoom-N12-S10 and ObstructedMaze-2Dlhb).

Original: Panoramic observations
Revised: Panoramic observations

Original: In order to increase performance of RC-GVF further, without introducingepisodic counts, we investigate the use of augmented panoramic views as previously explored by
Revised: In order to increase performance of RC-GVF further, without introducing episodic counts, we investigate the use of augmented panoramic views as previously explored by

Original: Parisi et al.
Revised: Parisi et al.

Original: In the MiniGrid environments, the observation changes almost in its entirety whenthe agent changes the direction it faces.
Revised: In the MiniGrid environments, the observation changes almost in its entirety when the agent changes the direction it faces.

Original: In contrast, moving to the next cell leads to fewer suddenchanges in the observation.
Revised: In contrast, moving to the next cell leads to fewer sudden changes in the observation.

Original: [Figure]

Original: -GVF (using panoramic observations) achieves similar performance without.

Original: The lack of gradual changes to egocentric observations when rotating is a consequence of having fourdiscrete angels for orientation (0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) , which may be an unrealistic depiction of how rotation affect observations for agents situated in the real world.
Revised: The lack of gradual changes to egocentric observations when rotating is a consequence of having four discrete angels for orientation (0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) , which may be an unrealistic depiction of how rotation affect observations for agents situated in the real world.

Original: As a consequence, any prediction errorabout future observations is dominated by predicting the outcome of turning compared to taking anyof the other actions in the environment.
Revised: As a consequence, any prediction error about future observations is dominated by predicting the outcome of turning compared to taking any of the other actions in the environment.

Original: To address this, prior work proposed to make the agent’s observations invariant to rotation by augmenting the observation with all directional observations [38].
Revised: To address this, prior work proposed to make the agent’s observations invariant to rotation by augmenting the observation with all directional observations [42].

Original: Ineffect, this assumes that the agent rotates 360 ◦ after moving to a cell, or alternatively that it is equippedwith additional sensors on its sides and back.
Revised: In effect, this assumes that the agent rotates 360 ◦ after moving to a cell, or alternatively that it is equipped with additional sensors on its sides and back.

Original: Similar to Parisi et al.
Revised: Similar to Parisi et al.

Original: [38], we will consider panoramicviews only for generating intrinsic rewards, and egocentric observations for the base PPO agent.
Revised: [42], we will consider panoramic views only for generating intrinsic rewards, and egocentric observations for the base PPO agent.

Original: In Figure 3, we demonstrate how RC-GVF with panoramic views and no privileged informationabout the underlying state of the MDP, can solve harder environments and is competitive withthe baselines that use episodic counts in five out of the six settings.
Revised: In Figure 3, we demonstrate how RC-GVF with panoramic views and no privileged information about the underlying state of the MDP, can solve harder environments and is competitive with the baselines that use episodic counts in five out of the six settings.

Original: We see that RC-GVF withpanoramic observations comfortably succeeds in the KeyCorridor-S5R3 and MultiRoom-N12-S10environments where it was previously unsuccessful.
Revised: We see that RC-GVF with panoramic observations comfortably succeeds in the KeyCorridor-S5R3 and MultiRoom-N12-S environments where it was previously unsuccessful.

Original: RC-GVF did not succeed in solving the harder
Revised: RC-GVF did not succeed in solving the harder

Original: ObstructedMaze-2Dlhb within the given frames despite the inclusion of panoramic views.
Revised: ObstructedMaze-2Dlhb within the given frames despite the inclusion of panoramic views.

Original: Figure 3shows that the use of panoramic views does not make the exploration problem trivial; RND alsouses panoramic views (and does not use episodic counts).
Revised: Figureshows that the use of panoramic views does not make the exploration problem trivial; RND also uses panoramic views (and does not use episodic counts).

Original: It improves with panoramic views on
Revised: It improves with panoramic views on

Original: KeyCorridor-S4R3 and MultiRoom-N7S8 but struggles with the harder instances.
Revised: KeyCorridor-S4R3 and MultiRoom-N7S8 but struggles with the harder instances.

Original: 5.3 Analysis
Revised: 5.3 Analysis

Original: Changing the temporal prediction horizon We study the effect of changing the temporal prediction horizon by evaluating RC-GVF with different discount factors. 
Revised: Changing the temporal prediction horizon We study the effect of changing the temporal prediction horizon by evaluating RC-GVF with different discount factors. Recall from Equation

Revised: 18 Frames (millions) 0.

Original: Recall from Equation 8 how thediscount factor can me mapped to the prediction horizon, which lets us estimate a lower bound on thehorizon of about 11 time steps for γ z = 0 .       6 with ϵ = 0 .
Revised: A v e r a g e r e t u r n (a) KeyCorridor-S5R       6 Frames (millions) 0. A v e r a g e r e t u r n z =0.99 z =0. z =0.7 z =0. z =0.3 z

Revised: =0.

Revised: z =0.05 z =0.

Revised: [Figure]

Revised: how the discount factor can me mapped to the prediction horizon, which lets us estimate a lower bound on the horizon of about 11 time steps for γ z = 0 . 6 with ϵ = 0 .

Original: In Figure 4 it can be seen how a largerprediction horizon is desirable for KeyCorridorS5R3.
Revised: In Figure 4 it can be seen how a larger prediction horizon is desirable for KeyCorridorS5R3.

Revised: Among the higher discount factors, 0 .

Revised: 95 is a good choice for this environment, but not 0 . 99 .

Original: Compared to our default value of γ z = 0 .
Revised: Compared to our default value of γ z = 0 .

Original: 6 , ashort horizon using 0 .
Revised: 6 , a short horizon using 0 .

Original: 3 reduces performance further, while discount factors closest to 0 yield the worstperformance.
Revised: 3 reduces performance further, while discount factors closest to 0 yield worse performance.

Original: In contrast, for MultiRoom-N12-S10, we find that most values of γ z perform well.
Revised: In contrast, for MultiRoom-N12-S10, we find that low and intermediate values of γ z perform well.

Revised: The high GVF discount factors of 0 . 99 and 0 .

Original:  Comparing γ z = 0 to RND shows how the use of recurrent networks and the variance term as measuredusing the ensemble of predictors contributes most to the performance of RC-GVF in this domain.
Revised: 95 perform worse. Comparing γ z =to RND shows how the use of recurrent networks and the variance term derived from the ensemble of predictors contributes most to the performance of RC-GVF in this domain.

Original: Ablation To better understand which of these components are contributing most to the successof RC-GVF, we consider the following additional ablations: (1) RC-GVF ( γ z = 0 ) without thevariance term in Equation 6, and (2) RC-GVF ( γ z = 0 ) without the recurrent predictor.
Revised: Ablation To better understand which of these components are contributing most to the success of RC-GVF, we consider the following additional ablations: (1) RC-GVF ( γ z = 0 ) without the variance term in Equation 6, and (2) RC-GVF ( γ z = 0 ) without the recurrent predictor.

Original: Figure 4ccompares these variations to RC-GVF ( γ z = 0 ), RC-GVF ( γ z = 0 . 6 ), and RND.
Revised: Figure 5a compares these variations to RC-GVF ( γ z = 0 ), RC-GVF ( γ z = 0 . 6 ), and RND.

Original: It can be seenthat the inclusion of the history-conditioned recurrent predictor and the variance term individuallycontribute to the improved performance of RC-GVF.
Revised: It can be seen that the inclusion of the history-conditioned recurrent predictor and the variance term individually contribute to the improved performance of RC-GVF.

Original: 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Frames (millions) 0.

Original: A v e r a g e r e t u r n (a) KeyCorridor-S5R       6 Frames (millions) 0. A v e r a g e r e t u r n z =0.7 z =0.
Revised: Using the history conditioned recurrent predictor sharply improves the performance of RC-GVF with γ z = 0 .

Original: z =0.3 z =0.
Revised: We hypothesise that this effect arises from the previous pseudo-rewards available to the predictor, which might enable better predictions on input observations it was not explicitly trained on (which is often the case in procedurally generated environments). In support of this hypothesis, we observed that using a recurrent predictor with solely observations as inputs does not produce such an improvement (Figure 9 in Appendix E.3).

Original:  z =0.05 z =0. [Figure] S     8 10 12 14 16 18 Frames (millions) 0.
Revised: In Figure 5b, we present ablations for γ z = 0 .

Original: A v e r a g e r e t u r n RC-GVF ( z =0.0) w/o disagreement RC-GVF ( z =0.0) w/o recurrence RC-GVF ( z =0.6) RC-GVF ( z =0.0) RND [Figure]
Revised: We consider RC-GVF without the variance term in Equation 6, and without the recurrent predictor.

Original: The improvement due to using thehistory conditioned recurrent predictor has a great impact onthe performance of RC-GVF with γ z = 0 . 
Revised: Removing either the disagreement term or the recurrent predictor worsens the performance of RC-GVF with γ z = 0 . 6 .

Original: We hypothesise that this effect arises from the previouspseudo-rewards available to the predictor, which might allow it to better handle input observations itwas not explicitly trained on (which is often the case in procedurally generated environments). Insupport of this hypothesis, we observed  that using a recurrent predictor with solely observations asinputs does not produce such an improvement(Figure 7 in Appendix D.2).
Revised: Interestingly, unlike as was observed for γ z = 0 , here we find that RC-GVF with only disagreement (no recurrent predictor) performs better than RC-GVF with only recurrent predictor (no disagreement). This further emphasises the importance of handling the aleatoric uncertainty for non-zero GVF discounts.

Original:   Limitations and Societal Impact
Revised: Increasing the ensemble size We conduct experiments with larger ensemble sizes in two MiniGrid environments. We obtain comparable results for ensemble sizes of 2, 4, 6 and 8 predictors in Figurein Appendix E.4, indicating that two member ensembles usually suffice for these problems.  Limitations and Societal Impact

Original: Limitations The incorporation of the current policy into the prediction task of GVF predictioncan have benefits with regards to behavioural exploration.
Revised: Limitations The incorporation of the current policy into the prediction task of GVF prediction can have benefits with regards to behavioural exploration.

Original: At the same time, it may also encourageover-exploration.
Revised: However, it may also encourage over-exploration due to the same state appearing ‘interesting’ under different policies.

Original: Possible alternatives include off-policy variants orthe use of explicitly policyconditioned (general) value functions [20, 17].
Revised: Possible mitigation strategies include off-policy variants or explicitly policy-conditioned (general) value functions [17, 20].

Original: In some situations it may be difficult to ascertain what is a good choice of the GVF discount factorγ z , which effectively determines the prediction horizon length.
Revised: In some situations it may be difficult to ascertain what is a good choice of the GVF discount factor γ z , which effectively determines the prediction horizon length, and can be influenced by optimisation effects under function approximation [72].

Original: The best choice may be problemdependent and could further benefit from dynamic adjustment during learning [5, 33].
Revised: The best choice may be problem-dependent and could [Figure] [Figure] A v e r a g e r e t u r n

Revised: RC

Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: ) w/o disagreement RC

Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: A

Revised: [Figure]

Revised: e r e t u r n

Revised: [Figure]

Revised: [Figure]

Revised: disagreement RC

Revised: [Figure]

Revised: GVF ( z

Revised: [Figure]

Revised: [Figure]

Revised: RC

Revised: [Figure]

Revised: z

Revised: [Figure]

Revised: [Figure]

Revised: further benefit from dynamic adjustment during learning [5, 37].

Original: However, ourresults in Figures 2 and 3 indicate that a default of γ z = 0 .
Revised: Nonetheless, our results in Figures 2 and 3 indicate that a default of γ z = 0 .

Original: 6 works well in many different settings.
Revised: 6 works well in many of the considered settings.

Revised: Several other intrinsic reward formulations using RC-GVF are possible (eg.

Revised: based on information/prediction gain, learning progress, or only the disagreement term), which we did not investigate.

Revised: Our specific formulation in Equation 6 was motivated by the prediction error in RND, which does not separate the aleatoric uncertainty when γ z > 0 .

Revised: Starting with the prediction error term also allowed us to closely study the connection between RND and RC-GVF.

Original: A general limitation of curiosity-based exploration for task-specific RL is that more information isgathered than is necessary to solve the given task [36, 30].
Revised: A general limitation of curiosity-based exploration for task-specific RL is that more information is gathered than is necessary to solve the given task [40, 33].

Original: Indeed, the space of potential GVFs is enormous.
Revised: Indeed, the space of potential GVFs is enormous.

Original: In the case of RC-GVF one could aim at focusing it by integrating task-specific information [62]or by considering predictions using a smaller set of policies that adequately cover state space [1].
Revised: In the case of RC-GVF one could aim at focusing it by integrating task-specific information [68] or by considering predictions using a smaller set of policies that adequately cover state space [1].

Original: Societal impact This paper does not focus on real-world applications of exploration and reinforcement learning methods and thus does not have a direct societal impact.
Revised: Societal impact This paper does not focus on real-world applications of exploration in RL and thus does not have a direct societal impact.

Original: In applications, advancedexploration methods may lead to unexpected policy behaviour, which could be mitigated by incorporating safety constraintsfor exploration [19].
Revised: In applications, advanced exploration methods may lead to unexpected policy behaviour, which could be mitigated by incorporating safety constraints [19].

Original:  Conclusion
Revised:  Conclusion

Original: We introduced RC-GVF, an exploration method that intrinsically rewards a reinforcement learningagent for its uncertainty about randomly generated, temporally extended outcomes of action sequencesin an environment.
Revised: We introduced random curiosity with general value functions (RC-GVF) , an exploration method that intrinsically rewards a reinforcement learning agent based on errors and uncertainty in predicting random transformations of observation sequences generated through the agent’s actions.

Original: Unlike state-novelty approaches, ours takes into accountthe multi-step futurebehaviours of policy and environment.
Revised: Unlike statenovelty approaches, ours takes into account multi-step future behaviours of policy and environment.

Original: Different from commonly used artificial curiosity approaches,we do not rely on model rollouts and do not need to predict all details of future observations.
Revised: Unlike commonly used artificial curiosity approaches, RC-GVF does not rely on model rollouts and does not need to predict all details of future observations.

Original: On the diabolical lock problem [32] and the MiniGrid environments [13] we demonstrated thatpredicting abstract quantities from a temporally extended horizon improves exploration in POMDPs.
Revised: On the diabolical lock problem [36] and the MiniGrid environments [13] we demonstrate that predicting abstract quantities of extended time intervals can improve exploration in POMDPs.

Original: Compared to recent methods such as AGAC [18] and NovelD [69] that rely on privileged episodicstate-visitation counts, we demonstrate how RC-GVF achieves competitive results even when usingonly panoramic observations.
Revised: Compared to recent methods such as AGAC [18] and NovelD [77] that rely on privileged episodic state-visitation counts in MiniGrid, our RC-GVF achieves competitive results even when using only panoramic observations.

Original: Importantly, in the natural setting where episodic counts are notavailable, RC-GVF significantly outperforms all baseline methods.
Revised: Importantly, in the natural setting where episodic counts are not available, RC-GVF significantly outperforms all baseline methods.

Original: Our approach can be generalized by moving beyond random pseudo-rewards, considering generalvalue functions under a set of different policies [1], and introducing time or state-dependentdiscounting, which offer exciting avenues for future research. Checklist For all authors...
Revised: Our approach can be generalized by moving beyond random pseudo-rewards, considering general value functions under a set of different policies [1], and introducing time or state-dependent discounting.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations

Original: of

Original: your work

Original: [Yes]

Original: See

Original: section

Original: c ) Did you discuss any potential negative societal impacts
Revised: This offers exciting avenues for future research.

Original: of your

Original: work

Original: [Yes] See section 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms
Revised: Acknowledgments and Disclosure of Funding

Original: to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Included inthe supplementary material.

Original: An open-source implementation will be made available

Original: online. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: We would like to thank Kenny Young, Francesco Faccio, Anand Gopalakrishnan, and Dylan Ashley for valuable comments.

Original: [Yes] Implementation details and hyperparameters are provided in

Original: Appendices B and C

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] We include 95% bootstrapped confidence intervals over 5or 10 seeds for all of our results.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type

Original: of GPUs, internal cluster, or cloud provider)? [Yes] Details provided in Appendix C If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: This research was supported by the ERC Advanced Grant (742870), the Swiss

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [Yes] Provided in Appendix A.2 with the 
Revised: National Science Foundation grant (200021_192356), and by the Swiss National Supercomputing

Original: description of the environment.

Original: (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: Centre (CSCS projects s1090 and s1127).

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [Table]