Compressed information is all you need: unifying intrinsic motivations and representation learning
Abstract
Humans can recognize categories, shapes, colors, grasp/manipulate objects, run or take a plane.
To reach this level of cognition, developmental psychology identiﬁes two key elements:
1- children have a spontaneous drive to explore and learn open-ended skills, called intrinsic motivation;
2- perceiving and acting are deeply intertwined: a chair is a chair because I can sit on it.
This supports the hypothesis that the development of perception and skills may be continually underpinned by one guiding principle.
Here, we investigate the consequence of maximizing the multi-information of a simple cognitive architecture, modelled as a causal model.
We show that it provides a coherent unifying view on numerous results in unsupervised learning of representations and intrinsic motivations.
This makes our framework a serious candidate to be a guiding unifying principle.
Introduction
Children spontaneously explore their environment and learn skills in an open-ended way [Piaget and
Cook, 1952, Ryan and Deci, 2000], driven by the so called intrinsic motivation (IM).
During this process, perception and action selection are deeply intertwined.
Humans gather observations as a result of their actions, select actions based on their representation of their state [Byrge et al., 2014] and perceive by looking at their action effect [O’regan and Noë, 2001, Gibson, 1977].
It results a progressive construction of perception and skills.
Yet, in machine learning (ML), IMs are often unrelated to unsupervised representation learning objectives like invariances learning Chen et al.
[2020], covariances learning Dangovski et al. [2021] or disentanglement Kingma and Welling [2014].
In practice, even IMs are highly heterogeneous; for example, one can maximize like bottleneck research [McGovern and Barto, 2001, Menache et al., 2002], expected cover time [Jinnai et al., 2019], saliency search [Bruce and Tsotsos, 2005] or novelty seeking behaviors Bellemare et al.
The information compression principle [Schmidhuber, 2008], which states that biological organisms aim to compress the data, is a natural candidate to unify IMs and representation learning objectives.
This is because of its ability to explain diverse behaviors ( e.g novelty) and feelings ( e.g subjective beauty)
[Schmidhuber, 2008].
However it lacks a quantitative instance of this principle that could guide unsupervised learning methods.
[Table]
Here, we hypothesize that an agent is driven to maximize the information it compresses in its cognitive architecture.
We model the cognitive architecture of an agent as a modular Bayesian network (BN) decomposed into modular goal/representation loops.
We quantify the compressed information as the multi-information of the BN and propose that an agent maximize it through learning representations and skills.
Our main contribution is the following: we show that the multiinformation of modular goal/representation loops uniﬁes
1- an information-theoretic taxonomy of IMs [Aubret et al., 2021] accounting for exploration behaviors and the learning of a hierarchy of skills;
2- several approaches in unsupervised representation learning based on learning invariant, covariant and disentangled representations.
3- the framework of Active Efﬁcient Coding (AEC)
[Teulière et al., 2015], which simulates early infants behaviors.
This makes our framework a serious candidate to be a guiding principle.
For an introduction to information theory, Bayesian networks and hierarchical reinforcement learning (HRL), please, refer to Appendix A.
Information compression in goal/representation loops
We aim to propose a computational framework that uniﬁes objectives/results in representation learning and IMs.
To unify these methods, our framework entails two key steps: 1) Proposing a simpliﬁed cognitive architecture, modelled as a BN; 2) Deriving the multi-information of the model.
In the following, we successively describe the two steps.
Causal model of the cognitive architecture.
We consider a simple architecture endowed with high-level ﬁndings in the brain: we integrate neural-based representations [Kruger et al., 2012,
Quiroga et al., 2005] and goals [Koechlin et al., 2003].
These groups of neurons perceive and act over increasingly larger time windows
[Badre and D’Esposito, 2007] and increasingly larger number of modalities [Wallace and Stein, 1997].
Figure 1 shows the Bayesian network B of a goal-making step at the ﬁrst and second level of a
HRL framework, assuming for simplicity that the second-level goals last for two timesteps.
Each module (e.g M 1 ) corresponds to a tuple (Representation, Goal) at one level of hierarchy with one set of modalities.
First, an observation o t is processed through a representation function to get a representation R M 1 t = f
( o t ) and reprocessed with potentially other representations R M 1 ∆ t = ( R M 1 t − 1 , R M 1 t ) into a higher-level representation R M 2 t = f M 2 ( R M 1 ∆ t ) .
Then it uses a high-level goal-conditioned policy, π M 2 to select a goal G M 2 t ∼ π M 2 ( ·| R M 2 t , . . . ) .
The goal-conditioned policy that achieves the goal G M 2 t in R M 2 t , also named a skill , selects the ground action G M 1 t ∼ π M 1 ( ·| R M 2 t , G M 2 t , R M 1 t ) .
The agent assumes that the environment deals with the ground action and the previous observation to output a new observation.
The sensori-motor loop continues this way.
In Figure 1(right), each observation corresponds to a modality, which is initially assumed independent from other modalities by the agent.
Actions can impact several observations (here all) and representations can aggregate different modalities (like visual, auditory, proprioceptive, ...). [Table] [Equation] [Table]
[Equation] Covariance Skill learning [Equation]
∅ Controllability Information compression. Our principle sums up as maximizing the information compressed by our causally represented modular cognitive model B .
We quantify the information compressed with the multi-information MI ( B ) .
In order to better understand the result of this maximization, we rewrite the multi-information as the sum of mutual information terms between a node and its parents [Slonim et al., 2001].
However, each combination of time × observation is a random variable in the Bayesian Network, making difﬁcult to maximize MI ( B ) .
Thus, we assume that the conditional dependencies between variables are stationary over time, i.e p ( X t | Pa ( X t ))
=
p ( X | Pa ( X )) where X represents any random variables and Pa represents the time-dependent parents (cf.
Appendix B).
It leaves us with:
MI
[Equation]
B
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table] [Equation]
[Table]
[Equation]
where T 0 is the current step for a given module ( V T 0 )
=
V for all random variables V ) and T the step corresponding to the computation of the next representation; O the set of modal-speciﬁc observations; M next are the modules that directly follow M (immediate larger timescale or set of modalities);
M prev the ones that precede M including observations and R M prev ∆ t the representations of M prev between R M and R MT .
To the best of our knowledge, Goal achievement does not directly relate to an existing loss.
To compensate this, we propose to relate information transmission of the Goal achievement to information transmission between two high-level representation.
We can quantify the relation between the two and we exhibit our lower-bound in Equation 2 (cf.
Appendix C for the derivation),
[Equation]
Mt
[Equation]
with M apnext being all the modules that precede those that follow M .
In the next section, we interpret the different parts of Equation 1 and Equation 2 relatively to IMs and representation learning methods.
Works uniﬁed through the compressed information
Let us look at the maximization of our lower bound with the representation functions and the policies.
We sum up our set of objectives in Table 1.
Compression Compression typically refers to an infomax loss [Linsker, 1990, Hjelm et al., 2019].
Originally, infomax maximizes the mutual information between inputs X and ouputs Y of a neural network I ( X ; Y ) .
In our case, we set X =
R M prev ∆ t and Y =
R MT
such that an agent builds a representation that maximally keeps information about downstream trajectories and modalities.
In some case, we may have H max ( R MT ) <
H max ( R M prev ∆ t ) because of some architectural constrains.
It results in a lossy compression setup, i.e the agent necessarily lose information.
In the case of a multimodal agent, lossy compression induces an interesting property.
Our architecture implies that incoming modalities are a priori independent, which may be wrong in practice: haptic feedbacks possibly give information about visual inputs.
Because of this wrong prior knowledge, the agent may focus on redundant information across modalities [Wilmot and Triesch, 2021] (cf.
Appendix D).
In developmental psychology, this privileged perception refers to the intersensory redundancy hypothesis [Wilcox et al., 2007, Bahrick et al., 2004].
For example, an agent endowed with touch and vision may encode the shape of the object as a priority.
We leave to Appendix E an analysis of how compression can induce disentanglement.
Covariance.
The Covariance impacts what information is kept in a representation.
We are only aware of works considering its non-hierarchical unimodal variant without considering ground observations ( M apnext ≈ ∅ ) through contrastive learning approximations [Poole et al., 2018].
Let us focus on the maximization of I ( R MT ; G M , R M )
[Nachum et al., 2019, Shu et al., 2020].
This makes R M covariant to G M , meaning that the agent must keep both low-frequency information (as before) and the higher-frequency variation caused by the action (or augmentation).
In RL, the resulting representation can be optimal to solve a RL task (extrinsic rewards) [Rakelly et al., 2021].
In visual/video contrastive learning, explicit sensitivity to some augmentations G M can improve object categorization [Dangovski et al., 2021, Zhang et al., 2019, Xiao et al., 2020], even though they mostly apply biologically not plausible image transformation like color jittering.
We note an exception [Jenni and Jin, 2021] which shows improved video representations.
Our derivation essentially augments this loss to scale it to hierarchically ordered representations.
Novelty The Novelty has recently been formalized in Aubret et al.
[2021] as maximizing I ( O ; R )
=
H ( O ) − H ( O | R ) .
This tells us where an agent must go to improve its representation of the environment and is known to incite a wide exploration of an environment (maximizing H ( O ) )
Aubret et al.
As such, it may avoid the dark-room problem faced by the Free-Energy Principle (FEP)
[Friston, 2010] without adding ad hoc priors [Friston et al., 2012].
Our derivation emphasizes the need of looking for new trajectories and multi-modal combinations rather than observations.
In case of multimodal agents, we have seen with the Compression term that multimodal compression favors cross-modal redundant information.
An agent may also actively look for such redundant information.
It has been explored under the principle of Active Efﬁcient coding (AEC)
[Teulière et al., 2015], which is an IM that states humans act and perceive to better compress their sensory inputs.
Several formalisms of AEC consider multimodal agents, an agent that separately perceives through its two eyes [Zhao et al., 2012, Eckmann et al., 2020, Vikram et al., 2014] or through visual and proprioceptive inputs [Wilmot and Triesch, 2021].
Our formalism of multimodal novelty directly matches [Eckmann et al., 2020], where they show that it can model the accommodation-convergence reﬂex in humans.
This reﬂex describes the ability of humans to focus or defocus the vision on nearby or distant objects.
Our loss suggests that similar results could be obtained for higher level behaviors.
Skill learning.
Skill learning makes possible learrning of reusable skills, i.e goal-conditioned policies that have a predictable trajectory.
In the DRL literature, an agent learns abstract skills by ensuring that low-level skills follow its high-level assigned goals [Aubret et al., 2021].
This is formalized as maximizing I ( G ; R δt ) where u extracts a subpart of the trajectory ( e.g a state).
For an agent that controls its torques (low-level actions), an example of time-extended skill can be to directly navigate in cardinal directions in a maze [Li et al., 2021].
Our derivation differs from the original formalism [Aubret et al., 2021] by including the previous representations R M apnext .
This highlights two properties.
First, skills are relative to their starting position, so that they apply a shift in the representation (like going to the right) rather than targeting a particular representation (reaching the wall).
In practice, this may require to know which skill can be executed where, i.e skills affordances [Gibson, 1977, Khetarpal et al., 2020].
Second, it scales the loss to a hierarchy of time-extended skills as they also impact lower-level representations.
At the lowest observations/actuators level, skill learning collapses to Controllability such that the agent tries to smartly select the actions to make according to how well the consequences are predictable [Touchette and Lloyd, 2004].
Conclusion
We hypothesized that an intrinsically motivated entity spontaneously tries to compress information.
We propose to instantiate it as the maximization the multi-information of the constrained cognitive architecture of an agent.
We showed that this amounts to maximize close-to previously known objectives within a modular hierarchy of goals/representations loops, thereby accounting for the autonomous learning of temporally/modality-extended representations and skills.
Our framework presents several limitations.
First, we consider a very simple cognitive architecture.
Second, our framework does not aim to explain how to maximize the local information theoretic terms.
It implies to approximate information theoretic values which is known difﬁcult [Belghazi et al., 2018, Poole et al., 2019].
Third, we hope future works will connect our contribution to studies about the impact of multi-information maximization at the level of neurons
[Ay, 2002].
A Background
In this section, we introduce the basic components of our theory, namely Bayesian networks, information theory and hierarchical reinforcement learning.
A.1 Bayesian networks
Bayesian networks (or graphical models) are directed acyclic graphs for which 1-vertices correspond to random variables which represent part of the state of the system; 2-edges reﬂect statistical dependencies between these variables, i.e a causal relationship.
Figure 2 (left) illustrates a simple example of Bayesian network.
Given the dependencies, a joint probability p ( X, Y, C, Z ) is conform to the graphical model if p ( X, Y, C, Z ) =
p ( X ) p ( Y | X ) p ( Z ) p
( C | Z, Y ) .
More generally, if ( V 0 , . .
, V N ) are the N + 1 vertices of a graph, and Pa ( V ) sums up the parents of V with respect to the edges, we can write [Pearl, 2014]:
[Equation]
These models are convenient to model action-perception loops [Touchette and Lloyd, 2004, Klyubin et al., 2004, Levine, 2018] and allow to compute information theoretic measures.
In this setting, parameters, actions, states, decisions etc...are all random variables.
Figure 2 (right) shows the
Bayesian network induced by a typical perception-action loop which is unrolled through time.
This kind of unrollment is typical through Dynamical Bayesian networks [Dagum et al., 1992].
In the following, while we could also use a Structured Graphical Model [Pearl, 2010], we assume that the standard Bayesian network is simpler and makes our results more reachable.
A.2 Measuring information
The Shannon entropy quantiﬁes the mean necessary information to determine the value of a random variable.
Let X be a random variable, its entropy is deﬁned by:
[Equation]
The entropy is maximal when p ( X ) encodes a uniform distribution, and minimal when p ( X ) follows a Dirac distribution.
Similarly to the entropy, we can deﬁne the entropy conditioned on a random variable Y :
[Equation]
Using these deﬁnition, we can introduce the mutual information, which quantiﬁes the information that a random variable Y contains about another random variable X :
[Equation]
The mutual information between two independent variables equals zero (since H ( X | Y )
=
H ( X ) ).
Similarly to the conditional entropy, we can introduce the mutual information between variables X and Y knowing another random variable W :
[Equation]
Following Equation 8, one can interpret the mutual information as being the difference between the joint distribution of two variables and the distributions of variables assuming they are independent.
It is straightforward to generalize mutual information to several random variables, it leads to the multi-information [Slonim et al., 2001], also named total correlation [Watanabe, 1960]:
[Equation]
It quantiﬁes the information that variables X 0 , . . .
, X N contain about each other.
As above, it is described as the discrepancy between the joint distribution p ( X 0 , . . .
, X N ) and the same distribution if variables were independent.
A.3 Hierarchical and developmental reinforcement learning
A Markov Decision process (MDP) is deﬁned through S the set of possible states; A the set of possible actions; T the transition function T : S ×
A × S → p ( s (cid:48) | s, a ) ; R the reward function
R : S ×
A × S → R ; ρ 0 : S
→ R the initial distribution of states.
An agent starts in a state s 0 given by ρ 0 .
At each time step t , the agent is in a state s t and performs an action a t ; then it waits for the feedback from the environment composed of a state s t +1 sampled from the transition function T , and a reward r t given by the reward function R .
The agent repeats this interaction loop until the end of an episode.
In reinforcement learning [Sutton and Barto, 1998], an agent aims to associate actions a to states s through a policy π in order to maximize the expected discounted reward deﬁned by (cid:80) ∞ t =
γ t R ( s t , a t , s t +1 .
In developmental machine learning [Colas et al., 2020], a goal is deﬁned by the pair ( g, R G ) where G ⊂ R d , R G is a goal-conditioned reward function and g ∈ G is the d -dimensional goal embedding.
We can now deﬁne the skill associated to each goal as the goal-conditioned policy π g ( a | s ) =
π ( a | g, s ) ; in other words, a skill refers to the sensori-motor mapping that achieve a goal [Thill et al., 2013].
This skill may by learnt or unlearnt according to the expected intrinsic rewards it gathers.
It implies that, if the goal space is well-constructed (as often a ground state space for example, R G = S ), the agent can generalize its policy across the goal space, i.e the corresponding skills of two close goals are similar.
The approach can be extended to a hierarchy of RL agents, which is then denoted as hierarchical reinforcement learning.
Typically, in feudal reinforcement learning [Dayan and Hinton, 1993], an upper-level manager learns a policy that assigns goals g ∈ G to workers .
To guide the learning process of the workers, the manager rewards them according to their actions.
Once a worker achieves its goal, the manager sets another one and the execution loop continues.
In practice, we can stack several levels of managers, so that there may be k > 2 levels in the hierarchy.
B Time independence
Let us derive of the time-independent mutual information between variables.
Assuming we have p ( X t )
=
p ( X | t ) = P ( X ) and p ( X t | Pa ( X t ))
=
p ( X | Pa ( X )) where X can represent any random variables and Pa represent the time-dependent parents, we have, with t = 0 , . .
, N : (cid:88) t
[Equation]
where we applied p ( X | t )
= p ( X ) in (1) and noticed that p ( t )
=
1 N in (2).
[Table]
C Proof of Equation
Before startign the proof, let us rigorously deﬁne a module and its corresponding notations.
A module represents a bio-inspired building block [Mussa-Ivaldi and Solla, 2004, Mussa-Ivaldi and Bizzi, 2000, Butz, 2016] associated to 1- a temporal/inter-modal receptive ﬁeld over incoming observations [Hasson et al., 2015];
2- an independent part of a lateral decomposition.
The temporal receptive ﬁeld refers to the number of temporally different indirectly processed observations, while the inter-modal receptive ﬁeld refers to the set of indirectly processed modalities.
This generic module is composed of a representation and goal variables (and their respective functions f and π ) with the temporallydependent causal relationships displayed in Figure 1 that makes them interact with each other.
Let us deﬁne a module formally, assuming causal relationships are consistent over time:
Deﬁnition 1.
A module M is a set of random variables ( G Mt , R Mt ) with a causal relation R Mt → G Mt and a conditioning module.
Deﬁnition 2.
A module M depends on an other module M p according to a temporal scale ∆ T if we have for a given T 1- one causal relation R M p ∆ T → R MT where R M p ∆ T = { R M p T − ∆ T , ..., R M p T } ; 2several ( R MT − ∆ T , G MT − ∆ T ) → G M p t for each t ∈ [ T − ∆ T, ..., T [ .
Inversely, M p conditions M .
Table 2 sums up the notations that derive from these two deﬁnitions.
Now, let us recall Equation ??
and start our derivation.
[Equation]
∈ [ R Mnext ; R MnextT ] [ I ( G Mt ; G M next , R M next , R M )] + I ( R MT ; R M prev ∆ t ) (cid:124) (cid:123)(cid:122) (cid:125) Compression/Novelty (cid:105) + (cid:88) o ∈O
I ( O oT
[Equation]
O
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125)
Controllability
To derive the lower-bound of the Skill learning , we mostly take advantage of the data processing inequality (DPI):
We want to lower-bound: (cid:88)
[Equation]
Thanks to conditional independence, add representations conditioning R M next :
[Equation]
Apply DPI to condition all goals on the same representations:
[Equation]
Bring together sequential goals into one group of variables:
[Equation]
We separate the term in two parts with the chain rule:
[Equation]
Thanks to conditional independence, we can extend the left part:
[Equation]
We can aggregate all goals and observations at the current timestep like in Figure
[Chang and Fung, 1989]:
[Equation]
We apply the DPI on E M to exhibit R M apnext T
[Equation]
The ﬁrst and last lower-bounds becomes narrow when lower-level modules maximize their own objectives.
For the second lower-bound we use the fact that I ( X, Y ; V | W ) ≤ I ( X ; V | W ) +
I ( Y ; V | W ) .
Finally, we leave a rigorous proof to future work and hypothesize that the lower-bound in Equationis narrow.
It may be because the right-hand term is unlikely to also maximize our Novelty/Covariance term.
[Equation]
D Multi-modal Compression/Novelty
According to our objective, a multimodal agent subject to lossy compression favors cross-modal redundant information with its novelty/compression term.
Let us formalize this observation.
In Figure 1 (right), our agent processes each modality independently and maximizing our Novelty amounts to compression maximization.
Since our modal-speciﬁc modules M 1 and M 2 have a timescale equal to one, our Novelty becomes I ( R M 1 T , R M 2 T ; R M 3 T ) where M 3 depends on M 1 and M 2 .
We can decompose it into
[Equation]
) where we apply the independence assumption in Equation 16.
The best way to maximize Equationunder lossy compression is to ﬁrst maximize
I ( R M 1 T , R M 2 T ) − I ( R M 1 T , R M 2 T | R M 3 T ) , i.e the information shared by R M 1 T and R M 2 T about R M 3 T as it will simultaneously maximize both terms.
We will now review this objective through concrete works applied to multimodal agents.
E Disentanglement
Let us consider the architecture in Figure 4, displayed with notations introduced in Appendix C. It displays a typical separate/rebranch pattern without extension of the temporal receptive ﬁeld.
Unlike when compressing multimodal data, we can not assume M 2 , ..., M l to be independent because of their common predecessor.
Let us compute the compression term of M l +1 :
[Equation]
Interestingly, Independency objective has been explored in the context of (Beta-) Variational AutoEncoder (VAE)
[Esmaeili et al., 2019, Chen et al., 2018, Gao et al., 2019, Kim and Mnih, 2018] and independent component analysis [Bell and Sejnowski, 1995, Lee et al., 2000].
These work suggest that it rules the disentanglement of a representation, i.e its decomposition into semantically different parts of the data (color, shape, motion, size, orientation ...).