An information-theoretic perspective on intrinsic motivation in reinforcement learning
Abstract
The standard reinforcement learning (RL) framework faces the problem of transfer learning and exploration with sparse rewards.
To address these problems, a large number of heterogeneous intrinsic motivation have been proposed, like reaching unpredictable states or unvisited states.
Yet, it lacks a coherent view on these intrinsic motivations, making hard to understand their relations as well as their underlying assumptions.
Here, we propose a new taxonomy of intrinsic motivations based on information theory: we computationally revisit the notions of surprise, novelty and skill learning and identify their main implementations through a short review of intrinsic motivations in RL.
Our information theoretic analysis paves the way towards an unifying view over complex behaviors, thereby supporting the development of new objective functions.
Introduction
Intrinsic motivation (IM) aims to describe to natural drive of children to explore their environment and acquire new skills.
Simply stated, intrinsic motivation is about doing something for its inherent satisfaction rather than to get a positive feedback from the environment Ryan and Deci (2000).
Taking inspiration from the psychological concept, numerous intrinsic motivations have been introduced in RL: an agent learns by trials and errors to maximize its expected discounted cumulative intrinsic rewards.
They show tremendous improvements on sparse rewards tasks and transfer learning with respect to standard RL methods Eysenbach et al.
; Bellemare et al.
For example, one can incite an agent to reach unvisited states Bellemare et al.
In practice, there is no consensus on a qualitative deﬁnition of intrinsic motivations, such that one can name “intrinsic" any task-agnostic rewards.
Two issues result from this absence of a common theory: 1- there is a plethora of heterogeneous IMs;
2- the core mechanisms that underpin their success is unclear.
In practice, it makes hard to identify relevant avenues of works.
Here, we bound important results in IMs to information theoretic objectives.
We identify three important information theoretic objectives and relate them to psychological concepts as well as the current deep RL literature.
First, we deﬁne the surprise as the expected information gain over true forward models and highlight that, under harmful approximations, it encompasses intrinsic rewards based on the expected information gain, prediction error or learning progress.
Second, we revisit novelty as actively learning representations and exhibit that it is currently maximized through two lines of works, one based on variational inference and one with k-nearest neighbors.
Third, we introduce the skill learning objective and ﬁnd that either discriminator based or states-goals based methods maximize it.
We expect our computational taxonomy to play a crucial role in understanding the principles that underlie complex behaviors, and thus to help building intrinsically motivated agents. [Table]
Surprise : I ( S (cid:48) ; Φ T | h, S, A )
Formalism
IG over forward model
Prediction error
Learning progress
Rewards
D KL ( p (Φ | h, s, a, s (cid:48) )
|| p (Φ | h ))
|| s (cid:48) − ˆ s (cid:48)
||∆ || s (cid:48)
− ˆ s (cid:48)
||
Novelty : I ( S ; Z )
Formalism
K-nearest neighbors
Variational inference
Rewards − log || s (cid:48) − nn k ( S b , s (cid:48) )
||− log q d ( s (cid:48) | z ) + D KL ( q e ( z | s (cid:48) )
|| p ( z ))
Skill learning : I ( G ; u ( T ))
Formalism
[Table]
goals [Table]
log p [Table] s [Table]
) −|| s [Table] s [Table]
||(1 + α skew ) log p ( s g )
Background
Reinforcement learning.
In a Markov Decision Process (MDP), an agent interacts with an unknown environment in the following way: the agent gets a state s 0 ∈ S that follows an initial state distribution p 0 ( s ) .
Its policy π selects actions a ∈ A to execute depending on its current state s ∈ S , a ∼ π ( ·| s ) .
Then, a new state is returned according to the transition dynamics φ T , s (cid:48) ∼ p ( ·| s, a, φ T ) .
The agent repeats the procedure until it reaches a particular state or exceeds a ﬁxed number of steps
T .
In RL, the agent learns a policy π to maximize the expected cumulative discounted reward given at each time step by the reward function R ( s t , a t , s t +1 ) .
In goal-conditioned RL, the reward function and the policy π g both depends on a given goal.
Information theory.
Here, we provide the basics about information theory.
The Shannon entropy quantiﬁes the mean necessary information to determine the value of a random variable.
Let X be a random variable with a law of density p ( X ) satisfying the normalization and positivity requirements, we deﬁne its entropy by H ( X )
= − (cid:82) X p ( x ) log p ( x ) dx .
The mutual information allows to quantify the information contained in a random variable X about an other random variable Y and is deﬁned by I ( X ; Y )
=
H ( X ) − H
( X | Y ) .
Three information theoretic objectives
In this section, we explain our three information theoretic objectives and shortly review the main approaches they encompass.
Table 1 sums up our ﬁndings.
3.1 Surprise
In this section, we assume the agent learns either a forward model of the environment parameterized by φ ∈ Φ .
The forward model computes the next-state distribution conditioned on a tuple state-action p ( S (cid:48) | S, A, φ ) .
Typically, φ can be the parameters of a neural network.
Trying to approximate the true model, the agent maintains an approximate distribution p (Φ | h ) of models,
where h t =
h refers to the ordered history of interactions (( s 0 , a 0 , s 1 ) , . . .
, ( s t − 1 , a t − 1 , s t )) .
In this paper, we adopt the deﬁnition of Itti and Baldi (2009), which deﬁnes surprise as the discrepancy between a prior distribution of beliefs and the posterior probability distribution following an observation.
In our formalism, we assume that there is a distribution of true models p (Φ T ) that underpins the transition function of the environment T .
In contrast with Φ , this is a property of the environment.
One can see this distribution as a Dirac distribution if only one model exists or as a categorical distribution of several forward models.
We deﬁne the expected information gain over the true models:
[Equation]
1b
[Equation]
We found three approximations of our formalism of surprise
[Equation]
Surprise as prediction error.
First, we can assume that the true forward model is a deterministic forward model, i.e log p ( s (cid:48) | s, a, h, φ T ) =
Const .
Then, assuming a Gaussian probability distribution, we end up with (cf.
Appendix A.1):
[Equation]
where ˆ s (cid:48) is the prediction of state following ( s, a ) .
Several works implement such objective using the ground state space or a learnt representation Ermolov and Sebe (2020); Pathak et al.
; Kim et al.
Despite the simplicity, their underlying assumption makes them sensitive to stochastic transitions Burda et al.
Surprise as learning progress.
One can also relax the determinism constraint and set log
[Equation]
where h (cid:48) concatenates h with an arbitrary number of additional interactions (cf.
Appendix A.2).
Such loss is called the learning progress and has also been used several times in the literature because it is more robust to stochastic transitions than prediction errors Schmidhuber (1991); Oudeyer et al.
; Kim et al.
However, we are not aware of an application to complex RL environments.
Surprise as prediction disagreement.
By assuming Φ ≈ Φ T , it is also possible to approximate the surprise by a form of prediction disagreement:
IG
[Equation]
where we simply marginalize out the φ in the right-hand term.
This amounts to compute the difference of prediction among the forward models.
Similar equations have been investigated in Shyam et al. ; Pathak et al.
; Yao et al.
; Sekar et al.
Surprise as expected information gain.
By extending Φ ≈ Φ T to the expectation part, we can further increase the approximation to match the expected information gain Sun et al.
(2011), used as an intrinsic motivation Houthooft et al.
; Achiam and Sastry (2017):
[Equation]
The last two approaches are computationally expensive as they require either
1- several forward models or 2- having a tractable probability distribution over forward models.
We consider a density of a model rather than a forward model, Equation 5 amounts to maximize the pseudo-count Martin et al.
(2017); Ostrovski et al.
(2017) (cf.
Bellemare et al.
3.2 Novelty
Barto et al.
(2013) defend that an observation is novel when a representation of it is not “close enough” to any representation found in memory .
Following this deﬁnition, we propose to formalize novelty seeking behaviors as those that actively maximize I ( S ; Z )
=
H ( S ) − H
( S | Z ) where Z is a low-dimensional space ( | Z | ≤ | S | ).
This objective is commonly known as the infomax principle
Linsker (1988); in our case, it amounts to actively learning a representation of the environment.
Most of the works focus on actively maximizing the entropy of state distribution while a representation learning function minimizes H ( S | Z ) .
Furthermore, if one assumes that Z = S , the infomax principle collapses to an entropy maximization H ( S ) .
Thus, we will focus on the maximization of H ( S ) .
Novelty with variational inference The most evident way to maximize the entropy of states consists in maximizing H ( ρ ( s ))
where ρ ( s ) =
p ( s | ρ ) approximates the state-visitation probability distribution.
But computing ρ ( s ) is challenging in high-dimensional state spaces.
Several methods propose to estimate ρ ( s ) using variational inference Zhang et al.
; Lee et al.
; Pong et al.
(2020) based on variational autoencoder architectures.
Assuming z is a compressed latent variable, p ( z ) a prior distribution and q d a neural network that ends with a Gaussian with a unit-diagonal co-variance matrix:
[Equation]
Novelty with k-nearest neighbors It is also possible to approximate the entropy of a distribution using samples and their k-nearest neighbors (k-nn) Singh et al.
; Kraskov et al.
Assuming nn k ( S b , s i ) is a function that outputs the k-th closest state to s i in the set S b , the unbiased approximation can be written as:
[Equation]
Intuitively, it means that the entropy is proportional to the average distance between states and their neighbors.
Few methods provably relate to such estimations, but several approaches take advantage of the distance between state and neighbors to generate intrinsic rewards Tao et al.
; Yarats et al. ; Seo et al.
In practice, k-nn methods are simpler than variational methods as they do not require to ﬁt a model.
They achieve SOTA results on the hard exploration task Montezuma’s revenge Bougie and Ichise (2020); Badia et al.
3.3 Skill learning
In our everyday life, nobody has to think about having to move his arm muscles to grasp an object.
A command to take the object is just issued.
This can be done because an acquired skill can be effortlessly reused.
Skill abstraction denotes the ability of an agent to learn a representation of diverse skills.
We formalize skill learning as maximizing the mutual information between the goal g ∈ G and some of the rest of the contextual states u ( τ ) ∈ u ( T ) , denoted as I ( G ; u ( T )) where τ ∈ T is a trajectory and u a function that extracts a subpart of the trajectory (last state for example).
Most of works maximize I ( G ; S )
=
H ( G ) − H ( G | S ) so that we will refer to this objective.
Skill learning with a discriminator The ﬁrst approach Florensa et al.
Eysenbach et al. ; Zhang et al.
(2020); Baumli et al.
; Sharma et al.
; Choi et al.
; Aubret et al. Hansen et al.
(2020) assumes the goal space is arbitrarily provided except for the semantic meaning of a goal.
In this setting, the agent samples goals uniformly from G (maximal H ( G ) ), and it progressively assigns all possible goals to a part of the state space.
To do this assignment, the agent learns a discriminator that approximates q ( g | s )
≈ p ( g | s ) and maximizes with q and the its skills:
[Equation]
such that each skill strives to reach the area assigned to it by the discriminator.
Skill learning as reaching states-goals Another set of works considers that the goal space is the state space ( G = S g ) and separately maximize − H ( S g | S ) and
H ( S g ) .
Let’s start with the ﬁrst term.
If log p ( s g | s (cid:48) ) is modelled as an unparameterized Gaussian with a unit-diagonal co-variance matrix, we have log p ( s g | s (cid:48) )
∝ −|| s g − s (cid:48)
||
22 + Const
so that one can reward an agent according to Levy et al.
; Zhao et al.
; Nachum et al.
; Kim et al.
; Nair et al.
; Li et al. (2021b) (where s can also be replaced by a learnt representation):
[Equation]
[Equation]
Let us now consider the maximization of H ( S g ) .
It can be maximized by over-sampling low-density state-goals Warde-Farley et al.
; Pitis et al.
Zhao and Tresp (2019); Aubret et al.
Li et al.
(2021a).
This can be formalized Pong et al.
(2020) as having a high-level agent that samples goals to maximize:
[Equation]
where α skew < 0 determines the weight of low-density state-goals.
Conclusion
Is there an information theoretic principle that supports the emergence of complex behaviors Our analysis supports that crucial behaviors like exploration and skill learning are explained by three information theoretic objectives maximization.
We argued for the relation between our unifying objective to notions of psychology like novelty, surprise or skill learning.
As such, we expect that our results will favor the design of new information theoretic open-ended learners.
Future works may reﬁne our Skill learning objective, tackle the problem of unifying our three objectives or may try to encompass more works like bottleneck research (McGovern and Barto, 2001).
A Appendix A.1 Surprise as prediction error
To avoid the need of the true forward model in our surprise deﬁnition, the agent can omit the left-hand term of Equation 1b by assuming the true forward model is modelled as a deterministic forward model.
In this case, we can write:
[Equation]
where we applied the Jensen inequality in Equation 11c and φ T ∼ p ( · ) is ﬁxed.
One can model p ( s (cid:48) | h, s, a, φ ) with a unit-variance Gaussian distribution in order to obtain a tractable loss.
This way, we have:
[Equation]
where
[Equation]
represents the mean prediction and φ parameterizes a deterministic forward model.
Following the objective [Equation] we can extract a generic intrinsic reward as:
[Equation]
A.2 Surprise as learning progress
Let’s analyze the following approximation:
[Equation]
As h (cid:48) becomes large enough and the agent updates its forward model, its forward model converges to the true transition model.
Formally, if one stochastic forward model can describe the transitions, we can write:
[Equation]
Finally, applying the procedure of Appendix A.1 in both terms of Equation 15, we end up with
[Equation]
[Equation]
i.e , the change of prediction error caused by some interactions.
A.3 Surprise as expected information gain
Let’s approximate our surprise by the expected information gain:
[Equation]
Then, the expected information gain Sun et al.
(2011); Little and Sommer (2013) over a forward or density model parameterized by φ can be formulated as:
[Equation]