Curiosity in Hindsight DeepMind
Mesnard
DeepMind Corentin Tallec DeepMind Rémi Munos
DeepMind Florent Altché DeepMind DeepMind
Abstract
Consider the problem of exploration in sparse-reward or reward-free environments, such as Montezuma’s Revenge.
The curiosity-driven paradigm dictates an intuitive technique: At each step, the agent is rewarded for how much the realized outcome differs from their predicted outcome.
However, using predictive error as intrinsic motivation is prone to fail in stochastic environments , as the agent may become hopelessly drawn to high-entropy areas of the state-action space, such as a noisy TV.
Therefore it is important to distinguish between aspects of world dynamics that are inherently predictable (for which errors reﬂect epistemic uncertainty) and aspects that are inherently unpredictable (for which errors reﬂect aleatoric uncertainty):
The former should constitute a source of intrinsic reward, whereas the latter should not.
In this work, we study a natural solution derived from structural causal models of the world:
Our key idea is to learn representations of the future that capture precisely the unpredictable aspects of each outcome—not any more, not any less—which we use as additional input for predictions, such that intrinsic rewards do vanish in the limit.
First, we propose incorporating such hindsight representations into the agent’s model to disentangle “noise” from “novelty”, yielding Curiosity in Hindsight : a simple and scalable generalization of curiosity that is robust to all types of stochasticity.
Second, we implement this framework as a drop-in modiﬁcation of any predictionbased exploration bonus, and instantiate it for the recently introduced BYOL-Explore algorithm as a prime example, resulting in the noise-robust “BYOL-Hindsight”.
Third, we illustrate its behavior under various stochasticities in a grid world, and ﬁnd improvements over BYOL-Explore in hard-exploration Atari games with sticky actions.
Importantly, we show state-of-the-art results in exploring Montezuma’s
Revenge with sticky actions, while preserving performance in the non-sticky setting.
Introduction
Learning to understand the world without supervision is a hallmark of intelligent behavior [1], and exploration is a key pillar of research in reinforcement learning agents [2].
How might an agent learn meaningful behaviors when external rewards are sparse or absent?
A predominant approach is given by the curiosity-driven paradigm [3], in which an agent’s ability to predict the future is used as a proxy for their “understanding” of the world.
Maintaining a learned model of the environment, at each transition the agent receives an intrinsic reward proportional to how much the realized outcome differs from their predicted outcome—which naturally directs them towards new areas that have not been seen.
There are two primary hurdles.
The ﬁrst is dimensionality : While outcomes can be predicted directly at the level of observations [4–8], pixel-based losses have generally not been found to work well in conjunction with complex, high-dimensional spaces [9].
Popular solutions have taken to operating on lower-dimensional latent representations , such as frame-predictive features [10], inverse dynamics features [11], random features [12], or features that maximize mutual information across time [13].
Most recently, bootstrapped features are employed in BYOL-Explore [14]—which achieves superhuman performance on hard-exploration games in Atari with a much simpler design than comparable agents.
The second is stochasticity : Curiosity-driven exploratory agents are often susceptible to bad behavior in environments with stochastic transitions, since they are often hopelessly distracted by high-entropy elements in the state-action space [9].
A classic example is the problem of a “noisy TV” generating a stream of intrinsic rewards, around which predictive error-based agents become stuck indeﬁnitely [15].
More broadly, this problem manifests with respect to any aspect of environment dynamics that is inherently unpredictable, including noise speciﬁc to certain states, or noise actively induced by the agent.
Novelty vs. Noise In the presence of stochasticity, predictive error per se is no longer a good measure for an agent’s lack of “understanding” of the world.
Intuitively, we wish to measure “understanding” by how much epistemic knowledge an agent has acquired (viz.
necessary truths about how the world works in general), which is distinct from how much aleatoric variation each outcome can display (viz.
contingent facts about how the world happens to be).
Precisely, we want to distinguish between aspects of world dynamics that are inherently predictable—for which (reducible) errors stem from “novelty”—and aspects that are inherently unpredictable—for which (irreducible) errors stem from “noise”.
Crucially, while the former should constitute a source of intrinsic reward, the latter should not.
Contributions In this work, we operationalize this distinction by deriving a natural solution based on structural causal models of the world: Our key idea is to learn representations of the future that capture precisely the unpredictable aspects of each outcome—not any more, not any less—which we then use as additional input for predictions, such that intrinsic rewards indeed vanish in the limit.
First, we propose incorporating such hindsight representations into the agent’s model to disentangle “noise” from “novelty”, yielding Curiosity in Hindsight : a simple and scalable generalization of curiosity that is robust to all types of stochasticity (Section 3).
Second, we implement this framework as a drop-in modiﬁcation of any prediction-based exploration bonus regardless of representation space, and instantiate it for BYOL-Explore, resulting in the noise-robust “BYOL-Hindsight” (Section 4).
Third, we illustrate its behavior under various stochasticities in a grid world, and improve over BYOL-Explore in hardexploration Atari games with sticky actions—a standard protocol for introducing stochasticity in training/evaluation in Atari.
Importantly, we show state-of-the-art results in exploring Montezuma’s Revenge with sticky actions, while preserving original performance in the non-sticky setting (Section 5).
Motivation 2.1 Problem Formalism
Consider the standard MDP setup.
We use uppercase for random variables and lowercase for speciﬁc values: Let X denote the state variable, taking on values x ∈ X , and A the action variable, taking on values a ∈ A .
While we keep our notation simple, we allow X to play the role of “contexts”, “features”, “beliefs”, or “embeddings” depending on environment observability and design of the agent.
Denote with τ ∈ ∆( X ) X×A the world dynamics such that X t +1 ∼ τ ( ·| x t , a t ) , and π ∈ ∆( A ) X the agent’s policy such that A t ∼ π ( ·| x t ) .
Finally, let ρ π denote the distribution of states induced by π .
Deﬁnition 1 (Curiosity-driven Exploration)
In this work, we focus on predictive error-based curiosity, which most popular approaches to curiosity fall under.
Denote the intrinsic reward as follows:
[Equation]
where τ η denotes the agent’s model of the environment parameterized by η , which is trained using the trajectories collected by rolling out a policy that seeks to maximize this same prediction error:
[Equation]
Stochastic Traps In stochastic environments, this reward converges to the entropy H [ X t +1 | x t , a t ] , so the agent may become stuck on repeatedly experiencing (intrinsically rewarding) transitions where entropy is high.
What we desire is a reward that converges to zero in the limit.
The notion of “optimistic” exploration offers a hint of what might be possible—Consider constructing a reward that satisﬁes:
[Equation]
upper bounding the distance between the world and the agent’s model.
On the one hand, Deﬁnitionveriﬁes this, but the bound fails to tighten even in the limit.
On the other hand, it is hard to measure this distance directly, as the entropy term is by construction unknown.
As it turns out, we shall later see that our proposed technique effectively gives a reward that veriﬁes the inequality—and is tight in the limit.
[Table]
2.2 Related Work
Our work inherits from the curiosity-driven paradigm [3–5,9–15,17,18], among which some methods have been designed with robustness to certain stochasticities in mind (Table 1).
However, note that our technique of using hindsight information is uniquely characterized by the following properties:
Stochasticity Types : First, it is capable of handling all types of stochasticities in generality.
Speciﬁcally, this includes stochasticity that is entirely random (e.g. a viewport polluted by noise sampled according to a distribution independent of states and actions), stochasticity that is statedependent (e.g. a visible object that performs a random walk within the environment), as well as action-dependent (e.g. a layer of random pixels that only appears if sampled on demand by speciﬁc actions).
For instance, previous works have found that inverse dynamics features can learn to ﬁlter out random noise [11], but may break down in the presence of action-dependent noise [9,19].
Dynamics Awareness :
Second, it does not require discarding the curiosity-driven paradigm.
By way of contrast, consider purely frequency-oriented exploration strategies, such as learning to predict a random projection of observations [12], or simply to predict the constant zero [16].
Since these are deterministic functions of inputs, they are in principle resilient to stochasticity.
However, empirically they can still behave poorly in the presence of action-dependent stochasticities [15]: If the noise is sufﬁciently diffuse , the agent may never learn the function well, so in the absence of any other learning signal—such as the world’s dynamics—they may still become stuck [20].
Generality and Scalability : As a drop-in modiﬁcation, it is generally applicable to any underlying choice of representation space.
In contrast, existing techniques capable of handling stochasticity are often tied to speciﬁc feature spaces, such as to employ inverse dynamics features [11], random features [12], or pixel-space features [15]—which may limit their ﬂexibility of application.
Moreover, unlike ensemble-based or disagreement-based techniques that require training a large number of models [19,21,22], we shall see that incorporating hindsight is simpler and more scalable by only requiring the addition of an auxiliary component to the usual prediction loss.
Alternative Paradigms
Other paradigms have also been studied.
Novelty-based methods encourage exploration on the basis of visitation counts [23], hashes [24], density estimates [25–28], and adversarial guidance [29,30]; further extensions have accounted for the long-term value of exploratory actions [16,31–33], as well as investigating the beneﬁts of episodic memory [34–36].
Knowledge-based methods encourage exploration on the basis of the agent’s uncertainty about the world [19,37], with most work focusing on estimating the information gain from different actions [21,22,38–44], or directly estimating learning progress [45–47].
Finally, diversity-based methods seek to maximize the state entropy [48–51], or to encourage learning diverse skills [52–63] and reaching different goals [64–68].
Curiosity in Hindsight
Consider the game of betting on a hidden dice roll: Suppose we take the action A t = “bet on 6”, then observe the outcome X t +1 =
“lost the bet”.
Two facts are clear: (1) a priori , we could not have predicted this result at all; (2) a posteriori , we may deduce the (latent) fact Z t +1 =
“the die must have rolled 1–5”.
These are not contradictory.
In particular, the former does not imply that we lack an
X t
A t
Z t +
W t
X t +
A t + A t +
X t +
W t + W t +
Z t + 2 Z t
X t +
Z t +(a) X - / A -Dependent Noise
X t
A t
Z t +
W t
X t +
A t + A t +
X t +
W t + W t +
Z t + 2 Z t
X t +
Z t +(b) Reparameterized Model understanding of how the game works, nor does it suggest that we should engage in further such bets to improve our understanding.
Indeed, knowing how the game works, in hindsight (i.e. given what we deduced about Z t +1 ), the outcome is obvious to us (i.e. we can now deterministically identify X t +1 ).
Conversely, suppose we actually didn’t know how the game works: Then we couldn’t have correctly inferred Z t +1 , nor would its knowledge have enabled us to identify X t +1 with certainty.
If so, engaging in additional bets may indeed allow us to learn and improve our understanding of how it works.
Intuitively, we can thus measure our understanding of each transition based on how much the outcome makes sense in hindsight .
In other words, instead of asking “How well can we predict predictpredictpredict predict predict predictpredict predict predictpredict predict predict predictpredictpredictpredict X t +1 a priori ?”
, we actually want to ask “How well can we reconstruct reconstructreconstruct reconstruct reconstruct reconstruct reconstructreconstruct reconstruct reconstructreconstruct reconstruct reconstruct reconstruct reconstructreconstructreconstruct X t +1 a posteriori —given hindsight Z t +1 ?”.
In the sequel, we ﬁrst formalize this intuition using the language of posterior inference when a known model of the world is available (Section 3.1).
Subsequently, we generalize this approach to generating learned hindsight representations when a model of the world needs to be learned at the same time (Section 3.2).
Finally, we derive Curiosity in Hindsight on the basis of these ingredients, showing that it approximates optimistic exploration (Inequality 3) while being robust to stochasticities (Section 3.3).
3.1 Structural Causal Model
Let Z denote a latent variable, taking on values z ∈ Z .
Speciﬁcally, for each observed transition tuple ( x t , a t , x t +1 ) , we let z t +1 encapsulate all sources of unobserved stochasticity in the dynamics, such that—by construction—we have that x t +1 = f
( x t , a t , z t +1 ) for some deterministic function f .
In this construction, a prior p over the latent Z t +1 induces the environment dynamics τ
( X t +1 | x t , a t ) .
Figure 1(a) illustrates the structural causal model for this, where solid squares denote deterministic nodes, shaded circles denote observable stochastic nodes, and unshaded circles denote unobservable stochastic nodes (here we use W to capture any randomness in the agent’s policy).
Note that while stochasticities can be entirely random (i.e. no edges into Z t +1 ), state-dependent (i.e. X t → Z t +1 ), or action-dependent (i.e. A t → Z t +1 ), by the reparameterization lemma it is always possible to represent an environment such that all stochastic variables are effectively exogenous [69,70]—as in Figure 1(b).
From Prediction to Reconstruction Consider the setting in which we know the model f .
Suppose ﬁrst that we somehow had access to each latent z t +1 .
Then the outcome of a transition at state x t and action a t would be deterministically computable with no uncertainty (i.e. reconstruction error = zero):
[Equation]
In reality, of course, the latent variable z t +1 is not observable.
Thus it may seem like the best we can accomplish is to compute the a priori expectation of the outcome (i.e. prediction error = entropy):
[Equation]
However, while z t +1 is not observable, based on the transition ( x t , a t , x t +1 ) we can infer a posteriori what its values could have been.
Importantly, by the consistency property of counterfactuals we know f ( x t , a t , Z t +1 )=
x t +1 for any Z t +1 ∼ p ( ·| x t , a t , x t +1 ) [71].
That is to say, conditioned on hindsight information, the reconstruction error of the true model is zero.
This suggests that when f is unknown and learned by the agent, the reconstruction error is an attractive candidate for an intrinsic reward .
Of course, now the missing piece is how to sample Z t +1 from the posterior—which we discuss next.
X t
A t
Z t +
X t +
A t + A t +
X t +
Z t + 2 Z t
X t +
Z t +
G t +
Z *t +
A t
X t + 1 ˆ X t (a) Single-Step Reconstruction
X t
A t
Z t +
X t +
A t + A t +
X t +
Z t + 2 Z t
X t +
Z t +
G t +
Z *t +
A t
X t + 1 ˆ
A t + A t +
Z *t +
X t + 2 ˆ
Z *t +
X t + 3 ˆ X t
G t +
G t +(b) Multi-Step Reconstruction
3.2 Hindsight Representations
In the practical setting where the model f
( X t , A t , Z t +1 ) is unknown, we learn to approximate it using a reconstructor f η , parameterized by η .
Since exact posterior inference p η ( Z t +1 | X t , A t , X t +1 ) is intractable, we simultaneously learn to approximate it using a generator p θ , parameterized by θ .
Two objectives are key.
First, as noted above, representations Z t +1 should be reconstructive of outcomes
X t +1 ; here we use a squared loss, but in principle any kind of reconstruction loss can be selected:
Objective 1 (Reconstruction) Let the reconstruction loss for a given transition ( x t , a t , z t +1 ,
x t +1 ) — including the generated hindsight representation z t +1 drawn from p θ ( ·| x t , a t , x t +1 )
—be deﬁned as:
[Equation]
and the (state-action) reconstruction bonus for the agent’s policy:
[Equation]
Driven to zero, this requires hindsight representations to encapsulate at least all aspects of the world’s dynamics that are unpredictable (so we don’t reward the agent for irreducible error).
However, we also don’t want Z t +1 to simply leak information about the outcome that is actually predictable to begin with (so we do reward the agent for reducible error).
Thus our second objective requires it to be independent of X t , A t ; here we use a contrastive loss, but in principle any kind of invariance loss can be selected:
Objective 2 (Invariance) Let the invariance loss for a tuple ( x t , a t , z t +1 ) be deﬁned with respect to a batch of K − 1 “negative” samples Z 1 t +1 , ..., Z K − 1 t +1 , using an auxiliary critic g ν parameterized by ν :
[Equation]
and the (state-action) invariance bonus for the agent’s policy:
[Equation]
Driven to minimax optimality over the state-action space—between the critic (i.e. maximizer) and generator (i.e. minimizer)—this requires hindsight representations to encapsulate at most the aspects of the world’s dynamics that are unpredictable.
How does it accomplish this?
We can be more precise:
Proposition 1 (Optimal Invariance)
Denote the pointwise mutual information between state-action x t , a t and hindsight z t by PMI θ ( x t , a t ; z t +1 ):=log p θ
[Equation] 1 | x t ,
[Equation]
Then the invariance bonus satisﬁes:
[Equation]
Moreover, denote the optimal critic parameter:
[Equation]
Then the bound is asymptotically tight:
[Equation]
Proof .
Appendix A.
[Equation]
In other words, in the limit of large batch sizes K → ∞ , at minimax optimality we have that Z t +1 is invariant to the values of x t , a t , and the invariance objective is equal to zero.
One question remains: Can reconstruction be simultaneously driven to zero in the limit of inﬁnite experience?
The answer is yes:
Proposition 2 (Optimal Reconstruction)
Denote with R rec.
θ,η ( x t , a t ) the reconstruction bonus as in
Objective 1, R K, inv.
θ,ν ( x t , a t ) the invariance bonus as in Objective 2, and their weighted sum for any λ :
[Equation]
Then its minimax optimal value is zero:
[Equation]
Proof .
Appendix A.
[Equation]
This suggests that such a weighted combination—of reconstruction loss (of a learned dynamics model) plus invariance loss (of a learned hindsight model)—may serve as a good intrinsic reward.
We now have all the ingredients for Curiosity in Hindsight, which it is instructive to contrast with Deﬁnition 1:
3.3 Optimistic Exploration
Deﬁnition 2 (Curiosity in Hindsight)
Denote the hindsight intrinsic reward function R θ,η,ν ∗ with the following (for now, this is idealized in that the critic is assumed optimal and batch sizes are inﬁnite):
[Equation]
Similar to before, the agent maintains an internal dynamics model trained to minimize this quantity over the trajectories it collects, while rolling out a policy that seeks to maximize this same quantity:
[Equation]
Recall that in the presence of stochasticity, standard curiosity-driven exploration can be seen as a poor approximation to “optimistic” exploration (Inequality 3)—because the bound is never tight even in the limit.
The following observation shows that exploration using Curiosity in Hindsight can resolve this:
Theorem 3 (Optimistic Exploration) Let coefﬁcient λ satisfy 12 log( λπ ) ≤ H θ
[ X t +1 | x t , a t , Z t +1 ] + D KL (cid:0) p θ ( Z t +1 | x t , a t ) (cid:107) p θ ( Z t +1 ) (cid:1) , with π the mathematical constant (not the agent’s policy).
Then:
[Equation] [Equation]
where τ θ,η ( X t +1 | x t , a t ) :
=
E Z t +1 ∼ p θ p η
( X t +1 | x t , a t , Z t +1 ) denotes the learned environment model.
Furthermore, for optimal model parameters θ ∗ , η ∗ we have that R θ ∗ ,η ∗ ,ν ∗ ( x t , a t ) = {+
+} for all x t , a t .
Proof .
Appendix A.
[Equation]
In other words, by choosing a small enough λ term, the hindsight intrinsic reward (Equation 15) is an upper bound on the KL-term we care about (Inequality 3).
Since this intrinsic reward can be driven to zero in the limit (Proposition 2), so is the KL-term, thus the reward-maximizing exploration policy (Equation 16) is approximating precisely the sort of “optimistic” exploration that we desired.
Practical Framework
In practice, K < ∞ , ν is not fully optimized, and λ is a hyperparameter.
The intrinsic reward is now:
[Equation]
(18) and the agent performs:
[Equation]
Overall, our framework involves a simple drop-in modiﬁcation on top of any standard curiosity-driven exploration: Instead of learning a predictive model that speciﬁes X t +1 ∼ τ η ( ·| X t , A t ) , we now learn a (hindsight-augmented) reconstructive model that speciﬁes X t +1 =
f η ( X t , A t , Z t +1 ) .
The main ingredients include the reconstructor f η , the generator p θ ( Z t +1 | X t , A t , X t +1 ) , and the critic g ν ( X t , A t , Z t +1 ) ; the main hyperparameters are the contrastive batch size K and the coefﬁcient λ in the intrinsic reward.
Finally, note that while for simplicity we have focused our exposition on modeling single-step outcomes, generalizing to the case of multi-step horizons is straightforward (see Figure 2).
4.1 Example: BYOL-Hindsight
o t b t a t −w t o t −b t −a t −w t −o t −b t −a t −a t b t ,o t +w t ,o t +w t , 2 w t −b t ,b t ,o t +w t ,a t +a t
+ˆ ˆ ˆ w t +w t
+w t +ℒ t ,ℒ t ,ℒ t ,z t ,z t ,z t ,Ɛ t ,Ɛ t ,Ɛ t ,torso output closedloop openloop target recon-struction hind-sight predic-tion /
BYOL-Explore [14] is a recent technique for curiosity-driven exploration that learns a bootstrapped representation space, an environment dynamics model, as well as an exploration policy simultaneously by optimizing a prediction loss in latent space.
First, an online network ω encodes observations o t into representations w t = ω ( o t ) .
A closed-loop recurrent network then computes representations b t of histories up until each time t .
This is used to initialize an open-loop recurrent network that computes representations b t,i for horizon steps indexed as i .
Finally, these representations are fed to a predictor ψ to output predictions ˆ w t,i , with the key novelty being that the target network ˜ ω is an EMA of the online network ω .
Figure 3 shows the BYOL-Explore setup: The predictive error L t,i at each open-loop step is computed, and the intrinsic reward associated to each transition o s , a s , o s +1 is the sum of its prediction errors (cid:80) t
+ i = s +1 L t,i .
This can be straightforwardly extended to use Curiosity in Hindsight (see dotted red region), yielding “BYOL-Hindsight”:
At each open-loop step, a hindsight vector is ﬁrst sampled as
Z t,i ∼ p θ ( ·| b t,i − 1 , a t + i − 1 , w t + i ) .
Reconstructions ˆ W t,i =
f η ( b t,i − 1 , a t + i − 1 , Z t,i ) are then computed, with the critic g ν simultaneously encouraging Z t,i to be independent of B t,i − 1 , A t
+ i − 1 .
Importantly, intrinsic rewards now come from L t,i = reconstruction + invariance losses, and not prediction losses.
Experiments
[Figure] [Figure]
R R
R R
C C
V V
H H
COINSPAWN
AG E N T
So far, we proposed an intuitive framework for equipping curiositydriven exploration with hindsight, and described an implementation on top of BYOL-Explore.
Three questions deserve empirical investigation: (a) Effectiveness :
In stochastic environments, predictive error-based methods—such as BYOL-Explore—may fail.
Does BYOL-Hindsight circumvent the problem?
(b) Robustness : Is BYOL-Hindsight robust to all of the types of stochasticities, including independent noise, statedependent noise, and action-dependent noise?
(c) Nonspeciﬁcity : In environments with no stochasticity, hindsight should confer no beneﬁt.
Does BYOL-Hindsight match the performance of BYOL-Explore?
Environments We employ two environments for experiments.
First, we use a Pycolab [72] maze environment to experiment with different
[Figure] [Figure]
types of stochasticities in a controlled manner.
Figure 4 shows the map: The agent spawns in the top right corner, and needs to explore its way past four (possibly stochastically oscillating) block elements (V1/2, H1/2), into the lower right corner where a pair of coins are randomly spawned.
The agent is purely intrinsically motivated, and progress is measured by trackers located beyond each of the block elements (R1–4).
The world is partially observable, and the agent only has access to a 5 × 5 frame (i.e. square radius 2) of its immediate surroundings as observations.
Second, we use the popular RL benchmark of Atari games [73], with preprocessed grayscale 84 × 84-pixel images as observations.
We consider some of the most commonly used hard-exploration games, including Montezuma’s Revenge.
Here we experiment with both pure-intrinsic and mixed (intrinsic plus extrinsic) exploration regimes.
Stochasticities In the Pycolab mazes, we use four different settings: “Baseline” (no noise), “Brownian Oscillators” (a form of state-dependent noise, where oscillators perform random walks along their axes of movements), “Random Pixel Noise” (a form of independent noise, which adds an extra layer of randomly sampled pixels with independent probability 0.25), and “On-Demand Pixel Noise” (a form of action-dependent noise, which does so if the no-op action is taken).
In the Atari environments, we use “sticky actions” [74] with stickiness 0.1 as the source of noise (a form of action-dependent noise).
Implementation In all experiments, we use the exact same implementation for BYOL-Explore as given in [14] for Atari, including all hyperparameters such as target network EMA 0.99, open-loop horizon 8, intrinsic reward normalization and prioritization, weight sharing between exploration and
RL closed-loop representations, and using VMPO [75] as the underlying algorithm.
BYOL-Hindsight starts from the same setup but includes hindsight as given in Figure 3.
There are two differences from the published version, which we use on both BYOL-Explore and BYOL-Hindsight for fair comparison:
The predictor MLP uses three hidden layers of 512 instead of one of 256, and the mixing coefﬁcient (in the mixed reward regime) is 0.2 instead of 0.1.
We explicitly indicate the “paper” version in our results.
Speciﬁcally for BYOL-Hindsight, the generator, reconstructor, and critic are all MLPs with three hidden layers of 512, the dimension of the generator noise (cid:15) and hindsight vector is 256, and λ =1 .
Finally, where shown for reference, RND and ICM are also implemented exactly as described in [14].
5.1 Pycolab Results
Figure 5 reports results (100k learner steps, averaged over 3 seeds).
First, the “Baseline” setting tests nonspeciﬁcity: Since there is no noise except for the coins spawned at the end of the maze, we expect predictive error-based exploration to perform similarly with or without hindsight.
For reference, we also show the performance of RND, which is in principle resilient to all types of stochasticity, because it explores by simply learning to predict the output of a deterministic function.
All three algorithms manage to reach all four trackers (with RND eventually losing interest due to vanished rewards, since the environment is small).
Second, in the “Brownian Oscillators” setting, BYOL-Explore fails to explore much beyond the ﬁrst two trackers, since it simply hangs around and reaps the stream of intrinsic
[Figure]
rewards from the unpredictable motion of the oscillators.
In contrast, BYOL-Hindsight (and RND) both still manage to explore the entire maze.
Third, in the “Random Pixel Noise” setting the results are similar, except both BYOL-Explore and RND perform even worse due to the fact that the noise is an entire layer of random pixels (i.e. extremely diffuse), which outcompetes all other dynamics of the world in magnitude.
Interestingly, while BYOL-Hindsight requires ever so slightly longer to adapt, it manages to perform similarly to before.
Even in the presence of high-magnitude, diffuse noise, the use of hindsight to capture the noise quickly allows it to stop bothering to predict it.
Fourth, the “OnDemand Pixel Noise” setting is perhaps the most telling.
BYOL-Explore is immediately trapped by the noise-inducing action, which it selects endlessly to generate a stream of intrinsic rewards.
Differently to before, even RND suffers greatly, which makes sense because the agent is no longer guaranteed a 0.75 probability of observing the world’s unpolluted dynamics.
In contrast, BYOL-Hindsight still performs as nicely as in the noise-free setting, underscoring its robustness to all forms of stochasticity.
5.2 Atari Results
Figure 6 reports results for Montezuma’s Revenge (3M learner steps, averaged over 3 seeds).
We consider both intrinsic-only (using no extrinsic signal) and mixed (using extrinsic + intrinsic rewards) regimes.
Exploration performance is measured by the number of different rooms of the dungeon the agent manages to discover over its lifetime—which requires understanding complex dynamics including navigating around various timed traps and moving enemies, and collecting keys to open doors in sequence.
In the sticky actions setting, BYOL-Explore completely ﬂatlines in the intrinsic-only regime, and only does marginally better in the mixed regime.
In contrast, BYOL-Hindsight manages to explore most of the rooms in both regimes, verifying the fact that the learned hindsight representations are able to disentangle the (unpredictable) stickiness from the rest of the (predictable) dynamics of the world.
Next, to test nonspeciﬁcity we run the same algorithms on the non-sticky setting: In both regimes, we observe that BYOL-Hindsight manages to preserve most of the original exploratory performance of
BYOL-Explore.
Finally, Figure 7 reports broad-based results for Atari hard-exploration games (1.2M learner steps, 1 seed), again for both intrinsic-only “(I)” and mixed “(E+I)” regimes.
Following existing literature, we use the extrinsic reward obtained during evaluation as a proxy for an agent’s ability to display “interesting behavior” in the environment.
We observe that in the vast majority of cases
BYOL-Hindsight improves on the performance of BYOL-Explore (especially when the latter ﬂatlines).
Overall, these results verify that Curiosity in Hindsight consistently bestows resilience to stickiness.
Conclusion
In this work, we studied the problem that stochasticity posed for predictive error-based exploration.
Theoretically, we reﬁned our notion of curiosity to separate (learnable) epistemic knowledge from (unlearnable) aleatoric variation during exploration.
Algorithmically, we proposed a method for learning (future-summarizing) representations of hindsight disentangled from (history-summarizing) representations of context.
Practically, we arrived at a simple and scalable framework for generating (reducible) intrinsic rewards even in the presence of (irreducible) stochastic traps—without having to estimate the problematic entropy term at all.
Our perspective has tight connections with the study of counterfactuals in policy evaluation [69,70], credit assignment [76,77], and fairness [78,79].
Future work may investigate the use of explicitly generative world models to map stochastic latents to outcomes, which may have potential use beyond generating intrinsic rewards—in the RL algorithm itself. A Proofs of Propositions
To simplify our notation, we remove subscripts such that X, A, Y denotes the transition X t , A t , X t +1 , and Z denotes the latent Z t +1 .
Then the environment’s dynamics is given by τ ( Y | x, a ) , the agent’s policy is given by π ( A | x ) , and the induced state visitation given by ρ π ( X ) .
The generator is denoted p θ ( Z | x, a, y ) , the reconstructor f η ( x, a, z ) , and the critic g ν ( x, a, z ) .
We start with several lemmas that will be useful, the ﬁrst being a pointwise version of Barber and Agakov’s variational lower bound:
Lemma 4 (Pointwise Barber-Agakov)
Denote the pointwise mutual information:
[Equation]
Then for any variational distribution q :
[Equation]
Proof .
Starting from the left hand side:
[Equation]
which completes the proof
[Equation]
Next, we deﬁne a generic contrastive expression with K − 1 “negative” samples of Z
[Equation]
and show that taking its expectation with respect to those samples yields a valid (i
[Equation]
e. normalized) probability density:
Lemma 5 (Normalized Variational)
Given independent samples z 1: K − 1 from p θ , deﬁne:
[Equation]
then the following deﬁnes a normalized density:
[Equation]
Proof .
The expectation integrates to one:
[Equation]
which completes the proof.
[Equation]
These two results allow us to show that the information Z contains on a tuple x, a —with respect to the generator parameterized as θ —is lower-bounded by the x, a -conditioned contrastive loss between “positive” samples Z ∼ p
θ ( ·| x, a ) from the posterior and “negative” samples Z ∼ p θ from the prior:
Lemma 6 (State-Action Lower Bound)
The x, a -wise mutual information satisﬁes:
[Equation]
Proof .
Use Lemmas 4 and 5, then Jensen’s inequality:
[Equation]
which completes the proof.
[Equation]
Next, we show that our invariance loss (Objective 2) for a tuple x, a, z is equal to the pointwise mutual information in the limit of inﬁnitely large negative batches, assuming an optimal critic parameter:
Lemma 7 (Pointwise Asymptotic Equality) Deﬁne the pointwise invariance loss:
[Equation]
) and the optimal critic parameter:
[Equation]
Then lim K →∞ L Kθ,ν ∗ ( x, a, z ) =
PMI θ ( x, a ; z ) .
Proof .
The E [ L Kθ,ν ( X, A, Z )] term is just the InfoNCE loss between variables Z and X, A , so we know that ν ∗ satisﬁes g ν ∗ ( x, a, z ) = log p θ ( z | x,a ) p θ ( z )
+ c ( x, a ) .
Substituting this back into L Kθ,ν ( x, a, z ) :
[Equation]
[Equation]
[Equation]
which completes the proof
[Equation]
This gives us what we need to derive Proposition 1, which we restate using our subscript-less notation:
Proposition 8 (Optimal Invariance)
The (state-action) invariance bonus satisﬁes:
[Equation]
and for the optimal ν ∗ the bound is asymptotically tight as K → ∞ :
[Equation]
Proof .
Use Lemma 6 for the ﬁrst part:
[Equation]
and use Lemma 7 for the second part:
[Equation]
which completes the proof.
[Equation]
Next, we show that Proposition 2 is true, which we similarly restate using our subscript-less notation:
Proposition 9 (Optimal Reconstruction)
Denote with R rec.
θ,η ( x, a ) the reconstruction bonus as in
Objective 1, R K, inv.
θ,ν ( x, a ) the invariance bonus as in Objective 2, and their weighted sum for any λ :
[Equation]
Then its minimax optimal value is zero:
[Equation]
Proof .
Take any MDP, as in Figure 1(a).
By reparameterization, we know there exists an equivalent graphical representation under which Z is exogenous, as in Figure 1(b).
Assuming realizability, let η ∗ be such that f η ∗ = f , and let θ ∗ be such that p θ ∗ ( Z | x, a, y )
= p η ∗ ( Z | x, a, y ) for any x, a, y .
First, by construction we have that Z ⊥ X, A , so the mutual information between Z and X, A must be zero:
[Equation]
[Equation]
for optimal critic parameter ν ∗ , where the ﬁrst equality uses Proposition 1.
Second, by consistency of counterfactuals f η ∗ ( x, a, Z )=
y for any Z ∼ p θ ∗ ( ·| x, a, y ) , so the reconstruction term is also zero.
It is easy to verify the optimal critic is a maximizer, and the optimal generator/reconstructor minimizers, which completes the proof.
(cid:3)
Finally, we recall the following basic relationship:
Lemma 10 (Conditional Mutual Information) Conditioned on any x, a , we have that:
[Equation]
Proof .
Starting from the left hand side:
[Equation]
which completes the proof.
[Equation]
Now, in our structural causal model, by construction Z captures all sources of noise—that is, there is no residual noise in each outcome Y .
However, for the purposes of optimization, while learning η we let the residual error be captured by a Gaussian “log-likelihood” (note that λ plays the role of “ 2 σ 2 ”):
[Equation]
and note that θ also induces a log-likelihood of the “ground-truth” conditional:
[Equation]
Now, recall the reconstruction loss and (state-action) reconstruction bonus:
[Equation]
as well as the invariance loss and (state-action) invariance bonus:
[Equation]
Moreover, recall the hindsight intrinsic reward function:
[Equation]
We can now show that Theorem 3 is true, which we similarly restate using our subscript-less notation:
Theorem 11 (Optimistic Exploration) Let λ satisfy the inequality 12 log( λπ ) ≤
H θ
[ Y | x, a, Z ]+
D KL (cid:0) p θ ( Z | x, a ) (cid:107) p θ ( Z ) (cid:1) , with π here being the mathematical constant (not the agent’s policy).
Then:
[Equation]
where τ θ,η ( Y | x, a ) :
=
E
Z ∼ p θ p η
( Y | x, a, Z ) denotes the learned environment model.
{+Furthermore, for optimal model parameters θ ∗ , η ∗ we have that the intrinsic reward R θ ∗ ,η ∗ ,ν ∗ ( x, a ) = 
+}
for all x, a .
Proof .
Use Proposition
[Equation]
then the constraint on λ , then Lemma 10:
[Equation]
which completes the proof.
[Equation]
The intuition is as follows:
Assuming realizability, at convergence “hindsight information” and “total stochasticity” cancel (i.e. neither more nor less), and the “remaining stochasticity” term goes to zero.
B Further Experiment Detail
[Figure]
[Figure]
-Explore.
The derivation above makes it clear that “hindsight information” and “total stochasticity” can be mis-aligned in two ways: First, hindsight may capture less information than in the “true” latent (which means the reconstruction error is not driven to zero).
Second, hindsight may capture more information than in the “true” latent (which means it leaks some information about the current state and action).
[Figure]
[Figure]
[Figure]
Regarding the former
[Figure]
Figure 9 shows that the reconstruction loss indeed converges to very small values
[Figure]
but not exactly zero, which is consistent with the fact that Z does not perfectly capture the entirety of what is unpredictable.
(This may be due to a variety of the usual factors, such as non-realizability and errors in optimization and estimation of expectations).
Regarding the latter, to assess how well the invariance constraint between Z and X
[Figure]
A is enforced, Figure 8 shows the invariance loss over time
[Figure]
We observe that—to the best of the critic’s ability to tell—the constraint is relatively well-enforced.
As another view into this
[Figure]
we can also gauge the amount of informational overlap between Z and X, A as follows:
In addition to learning f η ( X, A, Z ) , consider training additional predictors to predict Y , but only using states and actions as input (as in the usual forward prediction), or only using hindsight as input.
From Figure 9 we observe that this hindsight-only error (i.e. using only Z as input) is the highest; prediction error (i.e. using only X, A ) is lower, but not as low as reconstruction error (i.e. using X, A, Z as input).
For reference, the variance of target vectors is around 0 .
These observations are consistent with the fact that leakage indeed occurs, but much of it is regularized away by the invariance constraint.
Precisely, from the loss comparison we see that Z alone contains strictly less information than in X, A for predicting Y , and that their union X, A, Z contains the most information for predicting Y .
(Note that much information about each room is statically determined by the room number, thus even a tiny amount if leakage may be sufﬁcient to determine large portions of the future).
Finally, we stress that care is required when interpreting these curves due to the bootstrapped nature of the latent space in BYOL-Explore/Hindsight, as well as the fact that the dataset on which the predictor/reconstructor, generator, and critic are trained is endogenous to the rollout policy trained on the basis of the intrinsic reward.
For reference, Figure 10 additionally shows the intrinsic rewards arising from BYOL-Explore and BYOL-Hindsight, and Figure 11 shows the variance of those rewards.
For the other hard-exploration games, Figure 12 shows the intrinsic rewards from BYOL-Explore and
BYOL-Hindsight during training, corresponding to the performance results in Figure 7.
Lastly, Tablecompares BYOL-Explore and BYOL-Hindsight based on maximum scores obtained during training.
[Table]