Exploration via Empowerment Gain: Combining Novelty, Surprise and Learning Progress
Abstract
Exploration in the absence of a concrete task is a key characteristic of autonomous agents and vital for the emergence of intelligent behaviour.
Various intrinsic motivation frameworks have been suggested, such as novelty seeking, surprise maximisation or empowerment.
Here we focus on the latter, empowerment, an agent-centric and information-theoretic measure of an agent’s perceived inﬂuence on the world.
By considering improving one’s empowerment estimator – we call it empowerment gain (EG) – we derive a novel exploration criterion that focuses directly on the desired goal: exploration in order to help the agent recognise its capability to interact with the world.
We propose a new theoretical framework based on improving a parametrised estimation of empowerment and show how it integrates novelty, surprise and learning progress into a single formulation.
Empirically, we validate our theoretical ﬁndings on some simple but instructive grid world environments.
We show that while such an agent is still novelty seeking, i. e. interested in exploring the whole state space, it focuses on exploration where its perceived inﬂuence is greater, avoiding areas of greater stochasticity or traps that limit its control.
1. Introduction
In reinforcement learning (RL) we tend to give an agent a speciﬁc task to solve and use exploration heuristics to speed up training.
While these heuristics may be useful, biological systems have a more efﬁcient approach.
Even when faced with no concrete task, natural autonomous agents (like children) explore the world playfully to acquire new skills that may be used later.
This exploratory behaviour is an autonomous and active endeavour guided by intrinsic motivation which forms the core of a system for task-independent learning (Oudeyer et al., 2007; Oudeyer & Kaplan, 2009).
Intrinsic motivations have been formalised in RL literature in various ways (Aubret et al., 2019).
In particular, the concepts of novelty and surprise are believed to be vital to exploration (Berlyne, 1960; Barto et al., 2013).
We follow a common distinction (Berlyne, 1960; Barto et al., 2013;
Xu et al., 2021) in deﬁning the key difference of the terms, although the formalisation of novelty is particularly difﬁcult and controversial.
Following common intuition, novelty may be deﬁned as being a statistical outlier; something is completely novel if we have not seen it before.
One way this has typically been formalised in machine learning (Bellemare et al., 2016; Ostrovski et al., 2017;
Tang et al., 2017) is
Novelty
[Equation]
where p ( s ) models the visitation frequency of state s based on previous experience.
However, not everything that is novel is necessarily surprising.
And complex states may reveal something surprising even after having encountered them often before.
Surprise (also called contextual novelty or curiosity ) requires an internal world model that formulates an expectation about the future.
The deviation between these predictions and the observed reality quantiﬁes the amount of surprise.
One way to fomalize this is to use the the forward model’s prediction p ξ ( s t + 1 | s t , a t ) for computing the inverse likelihood of the observation s t + 1 (Schmidhuber, 1991a; Lopes et al., 2012;
Stadie et al., 2015; Achiam & Sastry, 2017; Pathak et al., 2017):
Surprise
[Equation]
But just seeking novelty or surprise faces a problem.
Consider static TV noise: it is highly entropic and unpredictable, hence remains highly surprising and novel.
To address this, it has been suggested to consider an agent’s learning progress (Schmidhuber, 1991a;
Storck et al., 1995; Lopes et al., 2012; Achiam & Sastry, 2017; Pathak et al., 2019).
Since collecting more data in this environment will not lead to better prediction of the noisy observation, an explorer should avoid these areas.
We should rather track and optimize the learning progress of the agent, which has been deﬁned as
[Equation]
where ξ (cid:48) and ξ are the updated and old parameters, after cq.
before observing some new data.
A related formulation of learning progress can be derived from a Bayesian deﬁnition of surprise (information gain) that similarly has been used for exploration (Houthooft et al., 2016).
While learning progress is an important insight, we would like to offer an alternative reason as to why the noisy TV screen is uninteresting for an exploring agent: it is really the failure of increasing its capability to interact with the world.
While improving one’s world model is an absolutely vital part of that process, it alone is not sufﬁcient – the goal is not just to become a perfect simulator.
Even a predictable pattern (e. g. a movie) remains uninteresting without meaningful interaction (e. g. TV remote) or without informing the agent about how to act in the world.
As another example, consider an agent exploring a chair.
It may not be necessary to predict every nuance of its physical appearance, but it is crucial to ﬁnd out about its important practical uses such as exploring which surface is suitable for sitting.
One formalism to measure an agent’s capability to interact with the world is called empowerment (Klyubin et al., 2005a;b).
It is an information-theoretic concept that measures the maximal ﬂow of information over an agent’s perception–action loop.
It is formally deﬁned as the channel capacity between reachable terminal states S (cid:48) and the possible action sequences A 1: T over a horizon T :
[Equation]
where I denotes the mutual information.
Intuitively, an agent is empowered if it can predictably reach many states, and it has low empowerment if its actions have little to no perceived inﬂuence on the world.
In this work, we propose a novel criterion for exploratory behaviour in autonomous agents which we call Empowerment
Gain (EG) .
Its goal can be stated as follows: in the absence of a concrete task, an agent should take those actions in the environment which provide the most information for improving its empowerment estimator, i. e. it should take those actions that maximise the increase in perceived inﬂuence over its environment after observing new data.
New experiences should help me recognise my capability to interact with the world .
We show how EG combines and puts into relation common notions of novelty seeking, surprise maximisation and learning progress in a single formulation.
This formulation takes into account an agent’s limitations in actuation and sensation as well as inherent stochasticity of the environment.
We provide a number of illustrative experiments in simple and discrete environments which support our theoretical ﬁndings and provide some more intuition about the EG criterion.
In particular, we show how EG guides an agent towards those areas of the state and action space where it has more potential for improving its inﬂuence in the world.
Since EG considers an extended time horizon
T instead of a single time step, we show that it tends to avoid inescapable traps in the environment that limit its future control.
Both of these behaviours cannot be achieved by novelty seeking, surprise maximisation or learning progress alone.
2. Background 2.1. Empowerment
Empowerment (Klyubin et al., 2005a;b; Salge et al., 2014) is a measurement of an agent’s perceived control over its environment.
It is deﬁned in terms of an agent’s embodiment; the coupling of sensors and actuators via the environment (perception–action loop).
Deﬁnition 1 Empowerment for state s is deﬁned as the channel capacity between terminal state distribution S (cid:48) and the possible action sequences A 1: T over a time horizon T : E
[Equation]
where I denotes mutual information and H denotes entropy.
Interpreting the environment as a communication channel, the agent ﬁnds a source distribution over action sequences
A 1: T such that its sensors at time T +1 can recover the most information about them.
Writing the mutual information as differences of entropies gives an intuitive understanding.
Interpreting eq.
(6), empowerment ﬁnds balance between diversity and predictability.
It wants to maximise the diversity of reachable states (entropy of ﬁnal states S (cid:48) ) while still being able to predict the outcome when conditioned on the taken action sequence.
In highly stochastic environments, our empowerment will always be relatively small, because even though we (accidentally) reach a lot of states, we can not predict the outcome.
By symmetry of mutual information, in eq.
(7) we want to maximise the diversity of action sequences, but every action sequence should ideally lead to a unique ﬁnal state s (cid:48)
such that the action sequence can be recovered from ﬁrst and ﬁnal state.
For discrete and deterministic environments, empowerment simpliﬁes signiﬁcantly and becomes proportional to the number of reachable states within a horizon T and the source distribution learns how to reach those states uniformly.
In this sense, we can view an action sequence a 1: T as a skill that transforms the world state from s to s (cid:48) and our source distribution as a distribution over those skills.
3. Empowerment Gain (EG)
In this section, we develop an exploration criterion based on expected improvement of an agent’s empowerment estimator.
We show how it encapsulates and relates other intrinsic motivations such as novelty seeking, surprise maximisation, learning progress and information gain.
We start out by looking at the components that make up empowerment estimation:
[Equation]
This estimation consists of 3 distributions, the source distribution ω φ ( a 1: T | s ) , the transition or forward model p ξ ( s T +1 | s, a 1: T ) and the ﬁnal state marginal distribution p φ,ξ ( s T +1 | s ) where we have split our parameters θ = { φ, ξ } .
The idea of empowerment gain is that an agent should act such that it maximises its expected improvement of its empowerment estimator.
Concretely, in any given state s , it should perform action a ∗ such that it maximises its expected improvement of its empowerment estimator ˆ E Tθ ( s ) after updating the parameters θ using the newly collected data d .
Formally, the objective is deﬁned as
[Equation]
where θ (cid:48) are the updated parameters after observing new data d .
By rewriting this objective in various ways, we develop a deeper understanding and discover that novelty seeking and learning progress of a forward or inverse model are already contained as part of the EG objective.
3.1. Empowerment Gain in a Maximum Likelihood Setting
Let us rewrite EG in a setting that is broadly applicable to any parametrised estimate of one’s empowerment:
[Equation]
where S φ (cid:48) ,ξ (cid:48) T + 1 is deﬁned by pdf
[Equation]
where the approximation is relatively tight for small updates of φ and ξ (for derivation see Appendix C).
We see that the empowerment estimate can be improved in two distinct ways.
First by (local) novelty seeking, i. e. by increasing reachable state entropy (eq.
(10)), which can be achieved by either detecting new states or by realising how to reach them more uniformly.
Second, by improving the forward model (learning progress) in eq.
Different to other methods focussing on learning progress on a forward model, the divergence between current and past forward model is put into context by the empowerment-realising source distribution ω φ (cid:48)
( a 1: T | s ) .
That is, the improvement of the forward model is weighted in so far as it helps the agent realise its inﬂuence in the world.
So even if learning progress of a forward model is large, if it does not help the agent in attaining states more reliably, empowerment gain may still be small.
Inversely, a small update in the forward model can still lead to a large empowerment gain if the improved transition is central to many skills (samples of ω φ (cid:48)
( a 1: T | s ) ).
Jointly optimising these two terms results in a tradeoff between novelty seeking and learning progress.
4. Experiments
In our experiments we investigated the exploratory behaviour of an EG-maximising agent in various simple but illustrative grid world scenarios.
We looked at the effect of various types of action noise, the shape of the state space, traps (sinks in the state space the agent cannot get out of) and empowerment’s computation horizon.
We compared this to an agent maximising its information gain on a forward model (IG explorer) and to a surprise-based explorer maximising the prediction error (PE explorer).
Our main goal was to illustrate and compare what these exploration criteria would do in a perfect information setting, but it should be noted that efﬁcient and scalable algorithms may be developed on top of existing work on empowerment approximation (Mohamed & Rezende, 2015; Karl et al., 2019).
This, however, is not the focus of our work.
Thus, for our purposes we remain in a simple grid world setting that allows for exact computation of empowerment gain, information gain and closed-form solutions for model updates.
This setting also leads to the exploration algorithm (see Algorithm 1 in Appendix A) where we made some simplifying assumptions.
In particular, when we explore, we ﬁrst try out each action an agent could take in the real world, update our parametrised model(s), and compute either expected empowerment gain or information gain.
Then we only execute and count the best action according to the respective criterion.
While this is impractical, we think this study is very illustrative of the behaviour induced by the various exploration criteria and gives and intuitive understanding of the theoretical results of the previous section.
Our grid worlds, built on top of MiniGrid (ChevalierBoisvert et al., 2018), have a discrete state space S and action space A .
The agent may perform ﬁve different actions that move to a neighbouring cell (left, right, up, down) or keep the agent where it is (stay).
An action that leads into a wall has no effect on the agent’s position.
As observations we use the global x and y coordinates unless stated otherwise.
For estimating EG and IG, we maintain a Bayesian forward model.
For each cell, the forward model is realised by a Categorical distribution with a Dirichlet prior.
The parametrisation of the forward model is, of course, essential to the resulting behaviour of these exploration criteria and should be taken into account when interpreting the results.
With our choice, the agent cannot generalise information among states as would be the case with e. g. a neural network.
We use the iterative Blahut-Arimoto algorithm to directly compute an empowerment estimate, including the distribution parameters of the source ω φ ( a 1: T | s ) and inverse model p ψ ( a 1: T | s, s T + 1 ) .
Information gain can be computed in closed form since the Dirichlet distribution is a conjugate prior to the Categorical distribution.
For more details, we refer to Appendix A.
4.1. Deterministic Environments
[Figure]
We start out by discussing the fully observable and deterministic setting.
For IG, in every state an agent performs each action uniformly which also leads to a uniform visitation frequency of every state (see Figure 1a).
In contrast, rather than performing each action uniformly, the EG explorer, from any state, visits each reachable state uniformly.
This leads to a visitation frequency that qualitatively resembles the empowerment landscape for the environment in that more empowered states (centre of the room) are visited more frequently during exploration than less empowered states (bordering walls).
Importantly, the EG explorer still makes sure to visit every state and does not remain strictly on high empowered states, it just shifts the exploration focus towards them.
This remains true for more complicated wall structures and is also affected by the empowerment horizon as shown in Figures 1c and 1d.
Since a state being more empowerment means having more ways to interact with the world, exploring these states more thoroughly should in general be more helpful for downstream tasks.
4.2. Sinks in the State Space
[Figure]
What happens if we have detrimental states in the environment that severely limit control?
Avoiding such states should generally be helpful for exploration.
We investigated the extreme scenario of perfect sinks where an agent, once entered, cannot get out again.
In our grid world, we model these states as lava cells (Figure 2a).
We can see in Figure 2b that, while the IG explorer skews away from the lava cells to some degree, this is just due to the random walk being reset randomly whenever an agent enters the lava.
For the EG explorer, avoidance of lava is actively pursued for an empowerment horizon greater than 1.
One may ask: why is EG not fully avoiding the lava since the agent’s empowerment in this state is 0 as its actions have no inﬂuence at all?
This is because the lava cell is still a state in and of itself, and it is still more empowering to know that you can reach that state more reliably.
However, improving empowerment after having entered a lava cell is impossible and hence the increase of the estimated empowerment for the rest of the trajectory is 0.
That’s why a horizon of 1 is not enough to realise the detrimental effect and longer horizon skew more and more away from the lava as shown in Figure 2d.
Interestingly, a purely novelty seeking agent would still aim for a uniform visitation frequency, visiting lava cells even more frequently than the IG explorer.
4.3. Actions of Varying Reliability
Next, we investigated the inﬂuence of varying action noises, i. e. in any state different actions are augmented with different level of noise.
Concretely, when taking action a , there is a probability p a that a random action a
(cid:48) ∈
A is performed instead.
We share the same action noise model for every state in the environment.
For IG exploration, we observe that the more noisy action is taken more frequently (Figure 3).
This is because occasionally we observe an unexpected event which leads to a bigger change to the model and thus to high information gain.
Similarly, surprise-based exploration (PE explorer) prefers the noisy action as well because the prediction error for noisy actions remains larger than for less noisy ones.
Interestingly, expected EG skews in the opposite direction, generally preferring the more reliable actions as they have greater potential of increasing one’s empowerment estimate.
This effect augments the results we found in the deterministic setting and also depends on the empowerment’s horizon, which push the agent away from the walls (see Figures 3b to 3d).
Again, we argue that this exploration behaviour is more desirable as exploring the actions that have a more reliable effect should in general be more helpful for interacting with the world.
4.4. State-Dependent Action Noise
Let us now look at state-dependent action noise, in which all actions in one state share the same noise model, but differ amongst different states.
For the IG and PE explorer we observe the same behaviour as in the deterministic setting, as these criteria cannot see ahead enough; after all, all actions in any state have an identical effect (except next to a wall).
Conversely, we found that an EG explorer skews towards the part of the state space with less noise (Figure 4).
While it still explores the whole state space, it focuses more on areas with greater potential for inﬂuencing the world.
However, as with the previously discussed sinks (lava), EG needs to be computed over at least a horizon of 2 in order to realise that going towards the less noisy region helps us with increasing our empowerment estimate to a greater degree.
For this experiment we used a slightly different objective that considered the impact of action sequences on the empowerment estimate instead of just an individual actions as [Figure] we detail in Appendix B.
[Figure]
5. Related work
Numerous ways of motivating exploration have been suggested over the years.
Among the ﬁrst was Schmidhuber (1991b) which based reward signals on prediction errors of predictive models.
This prediction error has been used in various ways using function representation learning (Stadie et al., 2015; Pathak et al., 2017).
Focussing purely on pre- diction error has the downside of being attracted unpredictable traps (static TV noise problem), where the error remains large even if it has been observed countless times.
Hence, learning progress of a predictive model has been suggested and formalised in various works (Schmidhuber, 1991a; Lopes et al., 2012; Stadie et al., 2015; Achiam &
Sastry, 2017).
Houthooft et al.
(2016) formulates learning progress in a Bayesian setting as information gain.
More recently, ensembles of predictors have been used to form some kind of internal disagreement metric as opposed to directly comparing to the real world difference (Pathak et al., 2019; Shyam et al., 2019; Sekar et al., 2020).
Different from these surprise based approaches, various ways of novelty seeking have similarly been formalised over the years.
Bellemare et al.
(2016) (extended by Ostrovski et al.
proposed using a pseudo-count using density estimation for novelty estimation in high-dimensional state spaces.
Tang et al.
(2017) suggested a computationally simple but effective generalization of count-based approaches using hashing.
Lee et al.
(2019) recast exploration as a problem of state marginal matching, where they aim to learn a policy for which the state marginal distribution matches a given target state distribution (often a uniform distribution).
Savinov et al.
deﬁnes novelty through reachability within a certain time horizon from observations in a memory.
However, there are also other exploration approaches based on skill acquisition that are related to our method.
Sharma et al.
(2019) encourages novel skill acquisition via mutual information maximisation between skills and states.
DIAYN (Eysenbach et al., 2018) encourages exploration by incentivising different skills that lead to distinguishable outcomes.
Empowerment (Klyubin et al., 2005a;b) as an intrinsic motivation has been gaining a more attention over the recent years and various ways of approximation have been suggested (Mohamed & Rezende, 2015; Gregor et al., 2016;
Karl et al., 2019; Zhao et al., 2021).
6. Conclusion
In this work we propose a novel framework for embodiment driven exploration.
Building on the universally applicable and information-theoretic measure of an agent’s perceived inﬂuence in the world – empowerment – we suggest an exploration criterion based on the expected improvement of one’s estimation thereof.
Theoretically, we show that it captures and puts into relation many previously suggested exploration criteria such as novelty seeking, surprise maximisation and learning progress.
Different to the use of these individual criteria, EG accounts for the agent’s current capabilities and boundedness, and focuses directly on what we are interested in – recognising one’s capability to interact with the world.
In various discrete grid world environments different noise models, we showcase our theoretical ﬁndings where an agent’s exploration is focused on areas with greater potential for increasing its inﬂuence.
For future work, the focus will be on smart approximations that can capture the spirit of EG while being scalable to scenarios of interest for current state of the art research.
A. Experiments
Algorithm 1 describes the exploration procedure in pseudocode form.
We usually perform just one long rollout without resetting the environment, however in the scenario with lava, we reset the agent randomly once the agent is stuck (the agent can not observe this reset).
Overall, the agent takes 50 , 000 steps in the environment in a whole experiment.
For IG exploration the algorithm works analogously, but naturally we compute information gain instead of empowerment gain in line 11.
To limit the effect of randomness in stochastic environments, we collect 10 instead of just 1 sample in line 8 to collect data for the model update.
For all experiments, we executed 10 separate runs to ensure reproducible results.
As the results are quite stable across runs and our results are mostly qualitative in nature, we picked representative plots from individual runs mostly at random.
Where quantitative measures are mentioned,
Algorithm
1: Initialise forward model m = p ξ ( s T +1 | s, a 1: T ) with
Dirichlet ( α ) prior, α = 0 .
2: for each episode do 3: s = env.reset() 4: for t=1..
T do 5: empow = blahut_arimoto(m) 6: for a in Actions do 7: # Try out all actions 8: s’ = env.step(a) 9: m’ =
m.update(s, a, s’) 10: empow’ = blahut_arimoto(m’) 11: gain[a] = empow’ - empow 12: end for 13:
# Perform the best action for exploration 14: a = argmax a gain 15: s’ = env.step(best_action) 16: m = m.update(s, a, s’) 17: end for 18: end for they are averaged across runs. For the environments in Section 4.3, the noise levels are: small noise p a = 0 . 05 , medium noise p a = 0 .
1 and large noise p a = 0 . There is noise when performing action "up" (with probability p a ) and "left" (with probability 2 p a ). The forward model of each cell is modelled by a Categorical distribution with a Dirichlet( α ) prior with α = 0 .
For empowerment estimation, BlahutArimoto is run until the change of empowerment from one iteration to the next is less than 1e − 6 or after at mostiterations, whichever comes ﬁrst.
B. An alternative formulation
As an alternative objective to eq.
(9), we want to discuss a different formulation that focuses on action sequences instead of individual actions:
[Equation]
[Table]
Different to the original formulation, here we are comparing and choosing the best action sequence over an horizon T that matches the empowerment’s horizon.
Intuitively, as we compute the empowerment over a speciﬁc horizon, it might be helpful to consider how execution of a whole skill over the same horizon instead of a single action impacts our empowerment estimate.
In particular when our forward model does not have the capacity to generalise over the state-action space, comparing EG after and before experiencing a single state action transition modiﬁes empowerment only in so far as it changes the transition probabilities from the starting state.
While choosing among skills may be desirable, it problematically comes with greater computational cost.
Naively computing the argmax becomes exponentially more expensive with the horizon T as we now need to compare all possible action sequences instead of individual actions.
C. Derivations
The approximation of eq.
(10) is relatively tight for small updates of ω ∗ φ ( a 1: T | s ) and p ξ ( s T +1 | s, a 1: T ) :
[Equation]
=
[Equation]
(cid:90) [Equation] ) log [Equation]
[Equation]
s ) − max
[Equation]
[Equation]
) (cid:90) a 1: T (cid:90) s T + ω [Equation] [Equation] ( s T +
[Equation]
s
[Equation]
{+a 1: T ) log p ξ ( s T +1 | s, a 1: T ) p φ,ξ ( s T +
+}
| s ) ω
[Equation]
φ (cid:48) ( a 1: T | s ) and ω ∗ φ ( a 1: T | s ) be the resp.
max. dist.
= (cid:90) a 1: T (cid:90) s t + ω
[Equation]
(cid:48) (
[Equation] [Equation] s ) [Equation]
(cid:48) ( s
[Equation]
| s, a 1: T ) log p [Equation] (cid:48) [Equation]
(cid:48)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
− (cid:32) (cid:90) a 1: T (cid:90) s T + ω ∗ φ ( a 1: T | s ) p ξ ( s T +1 | s, a 1:
[Equation] [Equation]
ω
[Equation]
with α = ω
[Equation]
s
[Equation]
) ω
[Equation]
= H
[Equation] s [Equation]
H
[Equation]
S
[Equation]
[Table]
[Equation]