U NSUPERVISED L EARNING OF G OAL S PACES FOR I NTRINSICALLY M OTIVATED E XPLORATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Intrinsically motivated goal exploration algorithms enable machines to explore and discover a diversity of policies in large and complex environments.
These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous action and state spaces.
However, they have so far assumed that self-generated goals are sampled in a speciﬁcally engineered space.
In this work, we propose to use deep representation learning algorithms to learn a goal space, leveraging observations of world changes produced by another agent.
We present experiments with a simulated robot arm interacting with an object and we study how the performances of exploration algorithms on such learned representations relate with their performances on engineered representations.
We also uncover a link between the exploration performances and the quality of the learned representation regarding the underlying state space
I NTRODUCTION
Spontaneous exploration plays a key role in the development of knowledge and skills in human children.
For example, young children spend a large amount of time exploring what they can do with their body and external objects, independently of external objectives such as ﬁnding food or following instructions from adults.
Such intrinsically motivated exploration (Berlyne, 1966; Gopnik et al., 1999) leads them to make ratcheting discoveries, such as learning to locomote or climb in various styles and on various surfaces, or learning to stack and use objects as tools.
Equipping machines with similar mechanisms of intrinsically motivated exploration should also be an essential dimension for lifelong open-ended learning and artiﬁcial intelligence.
In the last two decades, several families of computational models have both contributed to a better understanding of such exploration processes in infants, and how to apply them efﬁciently for autonomous lifelong machine learning (Oudeyer et al., 2016).
One general approach taken by several research groups (Baldassarre et al., 2013; Oudeyer et al., 2007; Barto, 2013; Friston et al., 2017) has been to model the child as intrinsically motivated to make sense of the world, exploring like a scientist that imagines, selects and runs experiments to gain knowledge and control over the world.
These models have focused in particular on three kinds of mechanisms argued to be essential and complementary to enable machines and animals to efﬁciently explore and discover skill repertoires in the real world (Oudeyer et al., 2013; Cangelosi et al., 2015): embodiment 1 , intrinsic motivationand social guidance 3 .
This article focuses on challenges related to learning goal representations for intrinsically motivated exploration, but also leverages models of embodiment, through the use of parameterized Dynamic Movement Primitives controllers (Ijspeert et al., 2013) and social guidance, through the use of observations of another agent.
Given an embodiment, intrinsically motivated exploration 4 consists in automatically and spontaneously conducting experiments with the body to discover both the world dynamics and how it can be controlled through actions.
Computational models have framed intrinsic motivation as a family of mechanisms that self-organize agents exploration curriculum, in particular through generating and selecting experiments that maximize measures such as novelty (Andreae & Andreae, 1978; Sutton, 1990), predictive information gain (Little & Sommer, 2013), learning progress (Schmidhuber, 1991; Kaplan & Oudeyer, 2003), competence progress (Baranes & Oudeyer, 2013) or empowerment (Salge et al., 2014).
When used in the Reinforcement Learning (RL) framework (e.g. (Sutton, 1990;
Schmidhuber, 1991; Kaplan & Oudeyer, 2003; Barto, 2013)), these measures have been called intrinsic rewards, and they are often applied to reward the ”interestingness” of actions or states that are explored.
They have been consistently shown to enable artiﬁcial agents or robots to make discoveries and solve problems that would have been difﬁcult to learn if one would have used a classical optimization or RL approach considering only the target reward (which is often rare or deceptive)
(Chentanez et al., 2005; Baranes & Oudeyer, 2013; Stanley & Lehman, 2015).
Recently, they have been similarly used to guide exploration in difﬁcult Deep Reinforcement Learning problems with rare rewards, e.g. (Bellemare et al., 2016; Houthooft et al., 2016;
Tang et al., 2017; Pathak et al., 2017).
However, many of these computational approaches have considered intrinsically motivated exploration at the level of micro-actions and states (e.g. considering low-level actions and pixel level perception).
Yet, children’s exploration leverages abstractions of the environments, such as objects and qualitative properties of the way they may move or sound, and explore by setting self-generated goals (Von Hofsten, 2004), ranging from objects to be reached, toy towers to be built, or paper planes to be ﬂown.
A computational framework proposed to address this higher-level form of exploration has been Intrinsically Motivated Goal Exploration Processes (IMGEPs) (Baranes & Oudeyer, 2009;
Forestier et al., 2017), which is closely related to the idea of goal babbling (Rolf et al., 2010).
Within this approach, agents are equipped with a mechanism enabling them to sample a goal in a space of parameterized goals 5 , before they try to reach it by executing an experiment.
Each time they sample a goal, they dedicate a certain budget of experiments time to improve the solution to reach this goal, using lower-level optimization or RL methods for example.
Most importantly, in the same time, they take advantage of information gathered during this exploration to improve solutions to other goals 6 .
While this property of cross-goal learning often enables efﬁcient exploration even if goals are sampled randomly (Baranes & Oudeyer, 2013), more powerful goal sampling strategy exists.
A particular one consists in using meta-learning algorithms to monitor the evolution of competences over the space of goals and to select the next goal to try, according to the expected competence progress resulting from practicing it (Baranes & Oudeyer, 2013).
This enables to automate curriculum sequences of goals of progressively increasing complexity, which has been shown to allow high-dimensional real world robots to acquire efﬁciently repertoires of locomotion skills or soft object manipulation Baranes & Oudeyer (2013), or advanced forms of nested tool use (Forestier et al., 2017).
Similar ideas have been recently applied in the context of multi-goal deep reinforcement learning, where architectures closely related to intrinsically motivated goal exploration are used by procedurally generating goals and sampling them randomly (Cabi et al., 2017; Najnin & Banerjee, 2017) or adaptively (Florensa et al., 2017).
Yet, a current limit of existing algorithms within the family of Intrinsically Motivated Goal Exploration Processes is that they have assumed that the designer 7 provides a representation allowing the autonomous agent to generate goals, together with formal tools used to measure the achievement of these goals (e.g. cost functions).
For example, the designer could provide a representation that enables the agent to imagine goals as potential continuous target trajectories of objects (Forestier et al., 2017), or reach an end-state starting from various initial states deﬁned in euclidean space (Florensa et al., 2017), or realize one of several discrete relative conﬁgurations of objects (Cabi et al., 2017), which are high-level abstractions from the pixels.
While this has allowed to show the power of intrinsically motivated goal exploration architectures, designing IMGEPs that sample goals from a learned goal representation remains an open question.
There are several difﬁculties.
One concerns the question of how an agent can learn in an unsupervised manner a representation for hypothetical goals that are relevant to their world before knowing whether and how it is possible to achieve them with the agent’s own action system.
Another challenge is how to sample ”interesting” goals using a learned goal representation, in order to remain in regions of the learned goal parameters that are not too exotic from the underlying physical possibilities of the world.
Finally, a third challenge consists in understanding which properties of unsupervised representation learning method enable an efﬁcient use within an IMGEP architecture so as to lead to efﬁcient discovery of controllable effects in the environment.
In this paper, we present one possible approach where aspects of these difﬁculties are approached by leveraging the possibility to observe changes made to the word by another agent 8 .
These observations are used for training an unsupervised learning algorithm to build a lower-dimensional representation that can be used afterwards for sampling goals in intrinsically motivated goal exploration algorithms.
Here, we do not assume that the learner actually knows that these observed world changes are caused by another agent, and we do not assume it can perceive or infer the action program of the other agent 9 .
We experimented with different unsupervised representation learning algorithms, such as AE Bourlard & Kamp (1988), VAE Rezende et al.
(2014)Kingma & Ba (2015), VAE with Normalizing Flow Rezende & Mohamed (2015), Isomap Tenenbaum et al.
(2000) or PCA Bishop (2006);
and we compared their performances under different metrics.
In particular, we measured the diversity of environment conﬁgurations visited during the exploration using those representations as goal space.
We also evaluated the average skill performance to produce various environment conﬁgurations translated as goals by the representation algorithm.
We also study which properties of the learned embedding are correlated with the acquired average skill levels.
Experiments presented in this article focus on two variants of a simulated experimental setup where a simulated multi-joint arm robot has to discover how to move a simulated object in various places and with various orientations, similar to the setup used in Forestier & Oudeyer (2016).
G OALS R EPRESENTATION LEARNING FOR E XPLORATION A LGORITHMS 2. E XPLORATION A LGORITHMS IN D EVELOPMENTAL R OBOTICS
Intrinsically Motivated Goal Exploration Processes (IMGEP), are powerful algorithmic architectures which were initially introduced in Baranes & Oudeyer (2009) and formalized in Forestier et al.
(2017), as heuristics to drive the exploration of large action spaces to learn forward and inverse control models in difﬁcult robotic problems.
To clearly understand the essence of IMGEPs, we must envision the robotic agent as an experimenter seeking information about an unknown physical phenomenon, by the mean of sequential experiments.
In this perspective, the main elements of an exploration are:
• A context c , element of a Context Space C .
This context represents the initial experimental factors that are not under the robotic agent control.
In most cases, the context is considered fully observable.
• A parameterization θ , element of a Parameterization Space Θ .
This parameterization represents the experimental factors that can be controlled by the robotic agent.
• An outcome o , element of an outcome space O .
The outcome contains information qualifying the execution of the experiment regarding interests of the agent.
• A phenomenon dynamics D : C ,
Θ
(cid:55)→ O , which in most interesting cases is unknown.
If we take the example of the Arm-Ball problem 10 in which a multi-joint robotic arm can interact with a ball, the context could be the initial state of the robot and the ball, the parameterization could be a set of motor torque commands for N time steps, and the outcome could be the position of the ball at the last time step.
Developmental roboticists are interested in developing autonomous agents, that learn two models, the forward model ˜ D :
C × Θ
(cid:55)→ O which approximates the phenomenon dynamics, and the inverse model ˜ I : C × O (cid:55)→ Θ which allows to produce desired outcomes under given context by executing the proper parameterization.
Using the aforementioned elements one could imagine a simple strategy that would allow the agent to gather tuples { c, θ, o } to train those models, by uniformly sampling a random parameterization θ ∼ U ( θ ) and executing the experiment,
See section3 for
details
a strategy which we will refer to as Random Parameterization Exploration .
The problem for most interesting applications in robotics, is that only a small subspace of Θ is likely to produce interesting outcomes.
Indeed, considering again the Arm-Ball problem with time-bounded action sequences as parameterizations, very few of those will lead the arm to touch the object and dragging it.
In this case, a random sampling in Θ would be a terrible strategy to yield interesting samples allowing to learn useful forward and inverse models.
To overcome this difﬁculty, one must come up with a better approach to sample parameterizations that lead to informative samples.
Intrinsically Motivated Goal Exploration Strategies propose a way to address this issue by giving the agent a set of tools to handle this situation:
• A Goal Space T whose elements τ represent goals that can be targeted by the autonomous agent.
• A Goal Policy γ ( τ ) , which is a probability distribution over the Goal Space.
It can be stationary, but in most cases, it will be updated each time following the intrinsic motivation strategy employed.
Note that in some cases, this Goal Policy will be a conditional on the context γ ( τ | c ) .
• A set of Goal-parameterized Cost Functions C
τ : O (cid:55)→ R deﬁned over all O , which maps every outcome with a real number representing the goodness-of-ﬁt of the outcome o regarding the goal τ .
A Meta-Policy Π : T , C (cid:55)→ Θ which is a mechanism to approximately solve the minimization problem
Π( τ, c ) =
arg min θ C τ ( ˜ D ( θ, c )) , where ˜ D is a running forward model, trained on-line during exploration.
In some applications, a de-facto ensemble of such tools can be used.
For example, if O is a Euclidean space, we can allow the agent to set goals in the Outcome Space T
= O , in which case for every goal τ we have a Goal-parameterized cost function C τ ( o )
= (cid:107)
τ − o (cid:107) .
In the case of the ArmBall problem, the ﬁnal position of the ball is used as Outcome Space, and hence the Euclidean distance between the goal position and the ﬁnal ball position at the end of the episode can be used as Goal-parameterized cost function.
Moreover, we can set the goal policy to follow an isotropic
Gaussian prior γ = N (0 , I ) , and the Meta-Policy mechanism to return the parameterization θ min = arg min θ ∈H C τ ( ˜ D ( θ, c )) where H is the history of executed parameterizations.
Using those tools, the autonomous agent can sample a parameterization for the next experiment, by ﬁrst sampling a goal τ using the Goal Policy γ , and second, executing the meta-policy Π to search the parameterization that will minimize the Goal-parameterized cost function C τ .
This two steps procedure has multiple interests.
First it accounts for the actual state of knowledge of the forward model, and second it allows to incorporate an intrinsic motivation mechanism to drive the exploration in the long term.
Intrinsic Motivation mechanisms can be seen as sequential update rules applied to the goal policy to optimize a given metric 11 such as the expected learning progress, novelty or information gain.
This procedure is the basis of the IMGEP architecture, which is represented in Algorithm 2.
This procedure has been experimentally shown to bring an effective exploration in robotic setups involving a complicated physical phenomenon such as object manipulations using tools (Forestier & Oudeyer, 2016; Forestier et al., 2017) or soft deformable objects Nguyen & Oudeyer (2014).
As we saw, deﬁning goals and the subsequent Goal-parameterized cost function can be made simple by sampling goals in the Outcome Space.
Nevertheless, an issue arises when it comes to using these algorithms in real-life setups, and within a fully autonomous learning approach.
First, there are many real world cases where providing an Outcome Space to the agent is is difﬁcult, since the designer may not himself understand well the space that the robot is learning about.
The approach taken until now (Forestier et al., 2017), was to create an external program which extracted information out of images, such as tracking all objects positions.
This information was presented to the agent as a point in [0 , 1] n , which was hence considered as an Outcome Space.
In such complex environments, the designer may not know what is actually feasible or not for the robot, and the Outcome space may contain many infeasible goals.
This is the reason why advanced mechanisms for sampling goals and discovering which ones are actually feasible have been designed Baranes & Oudeyer (2013);
Forestier et al.
Second, a system where the engineer designs the representation of a goal space is limited in its autonomy.
A question arising from this is: can we design a mechanism that allow the agent to construct a well-behaved Outcome Space by the mean of examples?
Representation Learning methods, in particular Deep Learning algorithms, constitute a natural approach to this problem as it has shown outstanding performances in learning representations for images.
In the next part we present the main Deep Representation Learning algorithms tested; the Auto-Encoders, along with their more recent Variational counter-part.
2. D EEP R EPRESENTATION L EARNING
We experimented over different Deep Representation Learning algorithms which are rapidly mentioned here.
For a more in-depth introduction to those models, the reader can refer to the Appendix
B which contains details on the derivations of the different Cost Functions and Architectures.
Auto-Encoders (AEs) are a particular type of Feed-Forward Neural Networks that were introduced in the early hours of neural networks Bourlard & Kamp (1988).
They are trained to output a reconstruction ˜x of the input vector x of dimension D , through a representation layer of size d < D .
They can be trained in an unsupervised manner using a large dataset of unlabeled samples
D = { x ( i ) } i ∈{ 0 ...N } .
Their main interest lies in their ability to model the statistical similarities existing in the data.
Indeed, during training, the network learns the regularities allowing to encode most of the information existing in the input in a more compact representation.
Put differently, AEs can be seen as learning a non-linear compression for data coming from an unknown distribution.
Those models can be trained using different algorithms, the most simple one being Stochastic Gradient Descent, to minimize a loss function J ( D ) that penalizes ˜x to be different from x for all samples in D .
Variational Auto-Encoders (VAEs) are a recent alternative to classic AEs (Rezende et al., 2014;
Kingma & Ba, 2015), that can be seen as an extension to a stochastic encoding.
The argument underlying this model is slightly more involved than the simple approach taken for AEs, and relies on a statistical standpoint presented in AppendixB.
In practice, this model simpliﬁes to an architecture very similar to an AE, differing only in the fact that the encoder f θ outputs the parameters µ and σ of a multivariate Gaussian distribution N ( µ, diag ( σ 2 )) with diagonal covariance matrix, from which the representation z will be sampled.
Moreover, an extra term will be added to the Cost Function, to condition the distribution of z in the representation space.
Under this restriction of factorial
Gaussianity, the neural network can be made fully differentiable thanks to a reparameterization trick , allowing a training using Stochastic Gradient Descent.
In practice VAEs tends to yield smooth representations of the data, and are faster to converge than
AEs from our experiments.
Despite these interesting properties, we emphasize that the derivation of the actual cost function relies mostly on the assumption that the factors can be described by a factorial Gaussian distribution.
This hypothesis can be largely erroneous, for example if one of the factor is periodic, multi-modal, or discrete.
In practice our experiments showed that even if the training could converge for non Gaussian factors, it tends to be slower and to yield badly conditioned representations.
Normalizing Flow proposes a way to overcome this problem, by allowing more expressive distributions (Rezende & Mohamed, 2015).
It uses the classic rule of change of variables for random variables, which states that considering a random variable z 0 ∼ q ( z 0 ) , and an invertible transformation t : R d (cid:55)→ R d , if z = t ( z 0 ) then q ( z ) =
q ( z 0 )
| det ∂t/∂ z 0 | − 1 .
Using this, we can chain multiple transformations t 1 , t 2 , . . .
, t K to produce a new random variable z K
= t K ◦· · ·◦ t 2 ◦ t 1 ( z 0 ) .
One particularly interesting transformation is the Radial Flow , which allows to radially contract and expand a distribution as can be seen in Figure 7 in Appendix.
As we can see, this transformation seems to give the required ﬂexibility to encode periodic factors.
2. G OALS R EPRESENTATION L EARNING FOR IMGEP
We can now introduce the augmented IMGEP architecture we used during this study.
If we consider the representation function resulting from the algorithm training ˜ R : X (cid:55)→
O as learning a mapping from a space of Raw Signals X to an Outcome Space O , we can rewrite the architecture as:
Algorithmic Architecture 1: Intrinsically Motivated Goal Exploration Strategy with Unsupervised Goal Space learning
Input:
Regressor ˜ D running , Regressor ˜ D , Regressor ˜
I , Goal Policy γ , Minimizer Π , Representation function ˜ R 1 begin for A ﬁxed number of Observation iterations n r do
Observe the phenomenon with raw sensors to gather a sample x i
Add this sample to a sample database D
= { x i } i ∈ [0 ,n r ]
Learn the Representation function ˜ R using the chosen representation learning algorithm trained with D for A ﬁxed number of Bootstrapping iterations do
Observe context c
Sample θ ∼ U ( θ )
Perform experiment and retrieve outcome with raw sensor signal o = ˜
R ( x )
Update Regressor ˜ D running with tuple { c, θ, o } for A ﬁxed number of Exploration iterations do
Observe context c
Sample a goal τ ∼ γ
Compute θ = arg min θ C τ ( ˜ D running ( θ, c )) using an optimization algorithm
Perform experiment and retrieve outcome from raw sensor signal o = ˜
R ( x )
Update Regressor ˜ D running with a tuple { c, θ, o }
Update Goal Policy γ , according to Intrinsic Motivation strategy while exploration running, asynchronously do
Update regressors ˜ D and ˜ I with acquired samples { c, θ, o } i 20 return The forward model ˜ D , the inverse model ˜
I and the representation ˜ R .
We will now present the experimental evaluation and the results of this novel architecture.
E XPERIMENTAL SETUP
To assess the performance of the novel exploration architecture, we did various experiments on two simulated environments.
We will now present in depth the experimental campaign we performed.
The anonymous code to reproduce the experiments is available at https://www.dropbox.com/s/zf4cmd2anzldbz5/UnsupervisedGoalSpaceLearning
Environments: We experimented on two different Simulated Environments derived from the classic Arm-Ball benchmark represented in Figure 1, namely the Arm-Ball and the Arm-Arrow environments, in which a multiple joint arm evolves in an environment containing an object it can handle and move around in the scene.
The environments are presented in depth in Appendix C.
Algorithmic Instantiation of the Architecture: As mentioned in Section 2.2, we experimented over different types of Deep Representation Learning Algorithms: Auto-Encoders , Variational AutoEncoders and Radial Flow Variational Auto-Encoders 12 .
In order to compare those to classic algorithms, we also experimented with the classic manifold learning algorithm Isomap (RGE-ISOMAP)
(Tenenbaum et al., 2000) along with the classic Principal Component Analysis (RGE-PCA).
Regarding the general approach presented earlier, we considered the following elements:
• Context Space C = ∅ :
In the implemented environments, the initial positions of the arm and the object were reset at each episode 13 .
Consequently, the context was not observed nor accounted for by the agent.
• Parameterization Space Θ =
[0 , 1] 21 : During the experiments, we used a Dynamic Movement Primitive controller to generate time-bounded motor actions sequences.
Since the DMP controller was parameterized by 3 basis function for each joint of the arm (7), the parameterization of the controller was represented by a point in [0 , 1] 3 × 7 . • Outcome Space O ⊂ R 10 : The Outcome Space is the subspace of R l spanned by the representations of the ensemble of images that observed in the ﬁrst phase of learning (see Annex C).
In our case, we constrained the Representation Learning algorithm to learn a representation of size l =
10 14 . • Goal Space T
=
O : The Goal Space was taken to equate the Outcome Space, as was presented earlier.
• Goal-Parameterized Cost function C τ ( · )
= (cid:107)
τ − · (cid:107) 2 : Sampling goals in the Outcome Space allows us to use the Euclidean distance as Goal-parameterized cost function.
Considering those elements, we used the instantiation of the IMGEP architecture represented in
Appendix in Algorithm 3.
We implemented a simple goal sampling strategy known as Random
Goal Exploration (RGE), which consists, given a stationary distribution over the Outcome Space p ( o ) , to sample a random goal o ∼ p ( o ) each time.
Since the Outcome Space was learned by the agent, it had no prior knowledge of p ( o ) .
We used a Gaussian kernel density estimation (Parzen, 1962; Rosenblatt, 1956) to estimate this distribution from the projection of the images observed by the agent in the observation phase into the learned goal space representation.
This allowed to sample points in the neighborhood of situations observed by the agent during the observation phase.
We used a simple k -neighbors regressor to implement the running forward model ˜ D , and the MetaPolicy mechanism considered was simply returning the nearest achieved outcome in the outcome space, and taking the same parameterization perturbed by an exploration noise.
Quality and Performance Measures: One of the fundamental point we wanted to assert, was how the quality of the representation geometry was affecting the performances, regarding the geometry of the underlying environment state space.
To measure the quality of the different learned representations, we gathered inspiration from the manifold learning literature.
After the agent learned the representation (at line 5 in the Algorithm 3), we performed different measures using a set of tuples { s i , o i } where s i is a point of the underlying environment state-space s i ∈ R n and o i =
˜ R
( f ( s i )) is the projection in the latent space of the image generated by the simulation from state s i .
With those
[Figure]
tuples, we assessed both the local and global quality of the different representations using two different measure; the Quality Q T & C and the Kruskal Stress S Kr , which are presented in much details in Appendix F
[Figure]
We also monitored the performances of the exploration algorithm during the exploration using two measures inspired from the Developmental Robotics literature; the Exploration Ratio Ratio explo and the Mean Squared Error of the meta-policy MSE .
The methods used to compute those measures are also detailed in the Appendix F.
Comparisons with other approaches: In order to compare our results with known algorithms, we also did experiments using two other algorithms:
• Full Information (FI): We compared our results with the performances achieved with an algorithm using directly the state-space underlying the environment as Outcome Space 15 , and has knowledge of p ( o ) =
U ( O ) , in which case, the autonomous agent does not need to learn a representation.
We expect this algorithm to upper bound the performances of our novel architecture.
• Random Parameterization Exploration (RPE): We also compared our results with the Random Parameterization Exploration approach, which consists in using no Outcome Space, nor Goal Policy, and only sample a random parameterization θ ∼ U (Θ) at each episode.
We expect this algorithm to lower bound the performances of our novel architecture.
R ESULTS
Exploration Performances: In the Figure 2 we can see the evolution of the Exploration Ratio through experiment episodes.
First we can see that following our expectations, for both environments the ER of the different architectures using a learned Outcome Space is always superior to the
ER achieved by the RPE architecture.
Moreover, the performances achieved by the RGE-FI architecture on the environments is superior or equal to the performances given by the novel architectures.
Regarding the different representation learning algorithms, on the Arm-Ball environment, all algorithms attain a reasonable ﬁnal ER, with a particularly good performances of the RGE-ISOMAP architecture which achieves almost as good performance as RGE-FI architecture.
On the ArmArrow environment, RGE-(ISOMAP/PCA/AE) architectures gave almost the same performances, slightly underneath the FI architecture.
RGE-(VAE/RFVAE) architectures performed fairly worse than the other representations.
[Figure] [Figure]
In the Figure 3, we represented the evolution of the Meta-Policy MSE through exploration episodes.
It is worth noting that the Meta-Policy MSE is not as simple to interpret, since it does not re Again, we can see that the ﬁnal MSE achieved by the representation based architectures is mostly contained within the ﬁnal MSE of the RGE-FI architecture and the RP architecture.
In particular, we observe that for both environments, RGE-(VAE/RFVAE) architectures yield almost as low performances as RPE.
Concerning RGE-(ISOMAP/PCA) architectures, their ﬁnal performance on the Arm-Ball environment equals the RGE-FI performance.
The same is not observed on the Arm-Arrow environment for which RGE-(ISOMAP/PCA/AE) architectures gives almost the same performance; lower than the one achieved by the RGE-FI architecture.
Representation Quality and Correlations with Final Exploration Performances: In the ﬁgure 4, we can observe the Kruskal Stress and Q T & C of the different learned representations, on both environments.
Concerning the Kruksal stress, which represents the global geometric quality of the representation, we can see that most algorithms gives almost equivalent performances on this point.
The RGE-RFVAE architecture performs particularly worse than the others on the Arm-Arrow environment.
This is surprising, since this environment contains a periodic parameter, which was supposed to be well represented thanks to Radial Flow.
Concerning the Local geometric quality, measured with Q T & C , we observe that it is good for the Arm-Ball environment, and falls equivalently for all representation algorithms on the Arm-Arrow problem.
In order to check the existence of any correlations between the quality of the learned representations and the exploration performances, we artiﬁcially degraded the performances of the different representation learning algorithms by adding noise to the training images 16 .
This allowed us to gather data on the exploration performances on representations of different qualities.
In the Figure 5, we have represented the joint plots of the ﬁnal MSE and ER achieved, against the Kruskal Stress and the Q T & C , for both environments.
First we can observe various correlation structures (measured by
[Table]
[Figure]
K r u s k a l S t r e ss pearsonr =
-0.31; p = 0.
[Figure] [Figure]
[Figure]
K r u s k a l S t r e ss pearsonr = 0.34; p = 3e-
[Figure]
Pearson tests), and the relationships between Exploration performances and representation quality seem to follow the same dynamics on both environments.
In both cases, a low Q T & C tends to correspond to lowers ﬁnal ER and larger ﬁnal MSE.
While not giving always the best performances, a better local geometric quality seems to be related with better ﬁnal ER and MSE.
Regarding the
Kruskal Stress, we can see that a high Kruskal Stress is almost always related with poor exploration performances both in ER and MSE.
Those visual interpretations are corroborated by the evaluation of diverse correlation measures represented in Appendix in Table 1.
Globally, those results follow our ﬁrst intuitions, that having an Outcome Space badly conditioned regarding the underlying phenomenon seems to consistently lead to poorer exploration performances.
On the other side, the exploration does not always beneﬁt from a better Local, nor Global, representation quality.
This tends to acknowledge the point that exploration performance relies on other factors; one being the distribution used as stationary Goal Policy that we present in the next paragraph.
Impact of Sampling Kernel Density Estimation One of the other factor impacting the exploration was assessed during our experiments was the importance of the distribution used as stationary
Goal Policy.
If in most cases, the representation algorithm gives no particular prior knowledge of p ( o ) , in the case of Variational Auto-Encoders, it is assumed in the derivation
that p ( o ) =
N (0 , I ) .
Hence, the isotropic Gaussian distribution is a good candidate to be used as stationary Goal Policy, rather than using a Kernel Density Estimation.
In Figures 6 and 10, we can see the comparison between exploration performances achieved with RGE-VAE and RGE-RFVAE using a Kernel Density
Estimated distribution or an isotropic Gaussian as Goal Policy.
We can see that the performances are severely degraded by using the isotropic Gaussian.
In terms of Exploration Ratio, the performances are as poor as the RPE performances, and concerning Mean Squared Error, the performances are lower than the RPE.
In this we can see that, in the case of stationary Goal Policy, the actual distribution used to sample goals seems to be of great importance to give good exploration results.
C ONCLUSION
In this paper, we proposed a new Intrinsically Motivated Goal Exploration Strategy architecture where the Outcome Space representation is learned using pixel-level observations of world changes caused by another agent, rather than being crafted by an engineer.
We tested this architecture on two simulated benchmarks parameterized by bounded and periodic continuous parameters.
We compared this architecture with classic architectures involving Full Information on the environment
[Figure]
state-space, and random parameterizations exploration.
Compared to those boundary cases, our new architecture gives good exploration performances, but stays back in terms of mean squared error.
In particular, we observe that the several deep representation learning algorithms did not give competitive performances regarding more classical algorithms such as Isomap and PCA.
We also gave evidences on the factors of performances of those exploration architectures.
We saw that a representation with poor geometric quality regarding the underlying phenomenon was almost always related with poor performances in explorations.
We also brought evidences on the importance of having a good goal policy in the stationary case, with possibly catastrophic impact on exploration performances.
This calls for studying in future work the impact of using adaptive goal sampling policies, for e.g. with the SAGG-RIAC architecture where goals are sampled based on an estimation of expected competence progress Baranes & Oudeyer (2013).
This work was limited to a fairly restricted set of environments.
Experimenting over a larger set of environments would beneﬁt our understanding of the exploration algorithms in general.
In particular, as Forestier et al.
(2017) has shown the beneﬁts of using modular representations of goal spaces when the environment is composed of independantly controllable objects, a topic of future work will be to consider representation learning algorithms that enable to learn such modular representations.
Moreover, in this paper, we only studied a learning scenario where representation learning happens before autonomous exploration.
The ability to jointly learn an Outcome Space and exploring would, as well, be an interesting topic for future work
Appendix A I NTRINSICALLY M OTIVATED G OAL E XPLORATION P ROCESS
Intrinsically Motivated Goal Exploration Processes are algorithmic architectures that can be instantiated into different exploration algorithms depending on the problem to explore.
The general architecture is represented on Algorithm 2.
Algorithmic Architecture 2: Intrinsically Motivated Goal Exploration Strategy
Input:
Regressor ˜ D running , Regressor ˜ D , Regressor ˜
I , Goal Policy γ , Minimizer Π 1 begin for A ﬁxed number of Bootstrapping iterations do
Observe context c
Sample θ ∼ U ( θ )
Perform experiment and retrieve outcome o
Update Regressor ˜ D running with tuple { c, θ, o } for A ﬁxed number of Exploration iterations do
Observe context c
Sample a goal τ ∼ γ
Compute θ = arg min θ C τ ( ˜ D running ( θ, c )) using a minimizing algorithm
Perform experiment and retrieve outcome o
Update Regressor ˜ D running with a tuple { c, θ, o } Update Goal Policy γ , according to Intrinsic Motivation strategy while exploration running, asynchronously do Update regressors ˜ D and I with acquired
{ c θ o } i 16 return The forward model ˜ D and the inverse model ˜ I .
B D EEP R EPRESENTATION L EARNING A LGORITHMS
Cost Functions used to train the different Deep Representation Learning algorithms used, can be motivated by a few theoretical arguments we summarized here.
Auto-Encoders (AEs)
The choice of the cost function can be motivated by considering the network as composed of:
An encoder network parameterized by weights θ that maps an input x to its deterministic representation z
= f θ ( x ) .
• A decoder network parameterized by weights φ that maps a representation z to a vector ξ parameterizing a distribution p ξ ( x | z ) with ξ = g φ ( z ) .
Under this stochastic decoding assumption, the Maximum Likelihood principle is used to train the model, i.e. AEs can maximize the likelihood of data under the model.
In the case of Auto-Encoders, this principle is compatible with Gradient Descent, and we can use the negative log-likelihood as a cost function to be minimized.
If input x is binary valued, p ( x | z ) is assumed to follow a multivariate
Bernouilli distribution of ξ parameters 17 , and the log likelihood of the dataset D is expressed as:
[Equation]
with ξ ( i ) =
g φ ( f θ ( x ( i ) )) .
For a binary valued input vector x ( i ) , the unitary Cost Function to minimize is:
[Equation]
provided that f θ is the encoder part of the architecture and g φ is the decoding part of the architecture.
This Cost Function can be minimized using Stochastic Gradient Descent (Bottou, 1998), or more advanced optimizers such as Adagrad (Duchi et al., 2011) or Adam (Kingma & Ba, 2015).
Depending on the depth of the network 18 , those architectures can prove difﬁcult to train using vanilla
Stochastic Gradient Descent.
A particularly successful procedure to overcome this difﬁculty is to greedily train each pairs of encoding-decoding layers and stacking those to sequentially form the complete network.
This procedure, known as stacked AEs, accelerates convergence.
But it has shown bad results with our problem, and thus was discarded for the sake of clarity.
Variational Auto-Encoders (VAEs)
If we assume that the observed data are realizations of a random variable x ∼ p ( x | ψ ) , we can hypothesize that they are conditioned by a random vector of independent factors z ∼ p ( z | ψ ) .
In this setting, learning the model would amount to searching the parameters ψ of both distributions.
We might use the same principle of maximum likelihood as before to ﬁnd the best parameters by computing the likelihood log L ( D ) =
(cid:80)
Ni =1 log p ( x ( i ) | ψ ) by using the fact that p ( x | ψ ) =
(cid:82) p ( x , z | ψ )
d z = (cid:82) p ( x | z , ψ ) p ( z | ψ ) d z .
Unfortunately, in most cases, this integral is intractable and cannot be approximated by Monte-Carlo sampling in reasonable time.
To overcome this problem, we can introduce an arbitrary distribution q ( z | x , χ ) and remark that the following holds:
[Equation]
with the Evidence Lower Bound being:
[Equation]
Looking at Equation (3), we can see that since the Kullback Leibler (KL) Divergence is nonnegative, L ( q, ψ ) ≤ log p ( x | ψ ) − D KL ([ q ( z | x , χ ) (cid:107) p ( z | x , ψ )]
whatever the q distribution, hence the name of Evidence Lower Bound (ELBO).
Consequently, maximizing the ELBO have the effect to maximize the log likelihood, while minimizing the KL-Divergence between the approximate q ( z | x ) distribution, and the true unknown posterior p ( z | x , ψ ) .
The approach taken by VAEs is to learn the parameters of both conditional distributions p ( x | z , ψ ) and q ( z | x , χ ) as non-linear functions.
Under some restricted conditions, Equation (4) can be turned into a valid cost function to train a neural network.
First, we hypothesize that q ( z | x , χ ) and p ( z | ψ ) follow Multivariate Gaussian distributions with diagonal covariances, which allows us to compute the b term in closed form.
Second, using the
Gaussian assumption on q , we can reparametrize the inner sampling operation by z = µ + σ 2 (cid:12) (cid:15) with (cid:15) ∼ N (0 , I ) .
Using this trick, the Path-wise Derivative estimator can be used for the a member of the ELBO.
Under those conditions, and assuming that p ( x | ψ ) follows a Multivariate Bernouilli distribution, we can write the cost function used to train the neural network as:
[Equation]
where f χ represents the encoding and sampling part of the architecture and g ψ represents the decoding part of the architecture.
In essence, this derivation simpliﬁes to the initial cost function used in
AEs augmented by a term penalizing the divergence between q ( z | x , χ ) and the assumed prior that p ( x | ψ ) =
N (0 , I ) .
Normalizing Flow overcomes the problem stated earlier, by permitting more expressive prior distributions (Rezende & Mohamed, 2015).
It is based on the classic rule of change of variables for random variables.
Considering a random variable z 0 ∼ q ( z 0 ) , and an invertible transformation t : R d (cid:55)→ R d , if z = t ( z 0 ) , then:
[Equation]
We can then directly chain different invertible transformations t 1 , t 2 , . .
t K to produce a new random variable z K
= t K ◦ · · · ◦ t 2 ◦ t 1 ( z 0 ) .
In this case, we have:
[Equation]
The great interest of this formulation is that the Law Of The Unconscious Statistician allows us to compute expectations over q ( z k ) without having a precise knowledge of it:
[Equation]
provided that h does not depends on q ( z k ) .
Using this principle on the ELBO allows us to derive the following:
[Equation]
This is nothing more than the regular ELBO with an additional term concerning the log-determinant of the transformations.
In practice, as before, we use p ( z 0 ) =
N ( z 0 ; 0 , I ) , and q ( z 0 | x ) =
N ( z 0 ; µ ( x ) , diag ( σ ( x ) 2 )) .
We only have to ﬁnd out parameterized transformations t , whose parameters can be learned and have a deﬁned log-determinant.
Using radial ﬂow , which is expressed as:
[Equation]
where r = | z − c | , h ( α, r )
= 1 α + r and α, β, c are learnable parameters of the transformation, our cost function can be written as:
[Equation]
provided that f χ represents the encoding, sampling ad transforming part of the architecture, g ψ represents the decoding part of the architecture, and β k , α k ,
c k are the parameters of the different transformations.
The effect of this ﬂow can also be seen in Figure 7.
Other types of transformations have been proposed lately.
The Householder ﬂow Tomczak & Welling (2016) is a volume preserving transformation, meaning that its log determinant equals 1, with the consequence that it can be used with no modiﬁcations of the loss function.
A more convoluted type of transformations based on a masked autoregressive auto-encoder, the Inverse Autoregressive Flow, was proposed in Kingma &
Welling (2013).
We did not explore those two last approaches.
C E XPERIMENTAL E NVIRONMENTS
The following environments were considered:
• Arm-Ball: A 7 joints arm, controlled in angular position, can move around in an environment containing a ball.
The environment state is perceived visually as a 50x50 pixels image.
The arm has a sticky arm tip: if the tip of the arm touches the ball, the ball sticks to the arm until the end of the movement.
The underlying state of the environment is hence parameterized by two bounded continuous factors which represent the coordinates of the ball.
A situation can be sampled by the experimenter by taking a random point in [0 , 1] 2 .
• Arm-Arrow: The same arm can manipulate an arrow in a plane, an arrow being considered as an object with a single symmetry that can be oriented in space.
Consequently, the underlying state of the environment is parameterized by two bounded continuous factors representing the coordinates of the arrow , and one periodic continuous factor representing its orientation.
A particular situation can hence be sampled by taking a random point in [0 , 1] 3 .
The physical situations were represented by small 70x70 images very similar to the dSprites dataset proposed by Higgins et al.
We took the approach to consider that the arm was not contained in the ﬁeld of view of the (virtual) camera used to gather images for Representation Learning, and consequently, only the objects were represented on the images.
We used a robotic arm composed of 7 joints, whose motions were parameterized by Dynamic Movement Primitives (DMP) (Ijspeert et al., 2013) using 3 basis functions (hence action policies have 21 continuous parameters), during 50 time-steps.
An example of such a DMP executed in the environment is represented in Figure 8.
The ﬁrst phase, where the learner observes changes of the environment (= ball moves) caused by another agent, is modeled by a process which samples iteratively a random state in the underlying state space, e.g. in the case of Arm-Ball s ∼ U ([0 , 1] 2 ) , and then generating the corresponding image x = f ( s ) that is observed by the learner.
D A LGORITHMIC I MPLEMENTATION
For the experiments, we instantiated the Algorithmic Architecture ??
by the following Algorithm 3.
In the text, we denote the (RGE- (cid:63) ) this algorithm used with the (cid:63) representation learning algorithm, which can be (RGE-AE) for Auto-Encoders, (RGE-VAE) for Variational Auto-Encoders, (RGERFVAE) for Radial Flow Variational Auto-Encoders, (RGE-ISOMAP) for Isomap, (RGE-PCA) for
Principal Component Analysis and (RGE-FI) for Full Information.
To denote the Random Parameterization Exploration, we use the acronym (RPE).
Input: k -neighbors regressor ˜ D running
Representation function ˜ R with a learning algorithm
Random exploration noise γ m
Random exploration ratio Γ e 1 begin for 10000 Observation Iterations do
Observe a random environment image x
i
Add this image to a database D
= { x i } i ∈ [0 , 10000]
Train ˜
R using D
Estimate the outcome distribution p kde
( o ) from { ˜ R ( x i ) } i ∈ [0 , 10000]
Set the Goal Policy γ = p kde to be the estimated outcome distribution for 100 Bootstrapping iterations do
Sample a random parameterization θ i ∼ p
( θ )
Execute the experiment θ i (= run a controller with parameters θ i )
Retrieve the outcome from raw image o i =
˜ R ( x i )
Update the forward model with ˜ D
( θ i ) (cid:44) o
i for 5000 Exploration iterations do if
Sample a random parameterization θ i ∼ p ( θ ) else
Sample a goal g i ∼ γ
Sample an exploration noise (cid:15) ∼ ( 0 , I )
Execute Meta-Policy to ﬁnd the approximately optimal θ i
=
Π( g i ) + (cid:15)
Execute the experiment θ i
Retrieve the outcome from raw image o i = ˜
R x i )
Update the forward model with ˜ D
( θ i ) (cid:44) o
i 23 return The forward model ˜ D
E D ETAILS OF N EURAL A RCHITECTURES
The architectures used for Deep Representation Learning algorithms were based on the architecture proposed in Higgins et al.
and was tweaked until the best result were obtained.
• Dense layer: of size 1200 with relu activations • Dense layer: of size 1200 with relu activations • Dense layer: of size• Dense layer: of size 1200 with relu activations • Dense layer: of size 1200 with relu activations • Dense layer: of size deﬁned by images with linear activations
The architecture was trained directly without particular stacking.
The AdaGrad optimizer was used, with initial learning rate of 1 e − 3 , with batches of size 100 , until convergence at 2 e 5 epochs.
Variational Auto-Encoder
• Dense layer: of size 1200 with relu activations • Dense layer: of size 256 with relu activations • Dense layer: of size 10*2 (10 for µ and 10 for σ ) • Dense layer: of size 256 with relu activations • Dense layer: of size 1200 with relu activations • Dense layer: of size deﬁned by images with linear activations
The architecture was trained with a deterministic warmup of 1 e 4 epochs, as proposed in Sonderby et al.
(2016), which shows improved convergence rate.
The Adam optimizer was used, with initial learning rate of 1 e − 3 , with batches of size 100 , until convergence at 1 e 5 epochs.
Radial Flow Variational Auto-Encoder
• Dense layer: of size 1200 with relu activations • Dense layer: of size 256 with relu activations • Dense layer: of size 10*2+10*(10+1+1) (10 for µ , 10 for σ , and 10 for c k , 1 for β k andfor α k for each transformation)
• Flow layer: of 10 chained transformations • Dense layer: of size 256 with relu activations • Dense layer: of size 1200 with relu activations • Dense layer: of size deﬁned by images with linear activations
The architecture was trained with a deterministic warmup of 1 e 4 epochs.
The complete ﬂow was made out of 10 planar ﬂows as proposed in Rezende & Mohamed (2015), whose parameters were learned by the encoder.
The Adam optimizer was used, with initial learning rate of 1 e − 3 , with batches of size 100 , until convergence at 5 e 4 epochs.
F Q UALITY AND P ERFORMANCE MEASURES
To assess the quality of the learned representation, we used classic measures from the Manifold
Learning literature.
Those measures mainly rely on the construction of a k -neighbors graph in both the input space and the output space using tuples of { a i , b i } where a i is a point in the input space, and b i is the representation of this point in the output space 20 .
We considered:
• Local Measure
We used the following measures extracted from Venna & Kaski
[Equation]
): – Trustworthiness: If r i ( j ) is the rank of the node j in the k -Neighbors of i in the input space, and
U k
( i ) the set of nodes present in the k -Neighbors of node i in output space but not in input space, the measure is deﬁned as:
[Equation]
– Continuity: If ˆ r i ( j ) is the rank function in output space, and V k ( i ) the set of nodes present in the k -Neighbors of node i in input space but not in output space, the measure is deﬁned as:
[Equation]
– Quality: Is a composite measure that sums the two previous ones:
[Equation]
The neighbors size is an hyper-parameter of the measure.
In practice we used k = 7 as proposed in the extensive review Gracia et al.
• Global Measure
We used the measure of global reduction stress from the pioneer work on manifold learning of Kruskal (1964).
– Kruskal Stress:
Considering all the distances ξ ij in the input space and their counterpart δ ij in the output space, we perform an Isotonic Regression and compute the stress measure as:
[Equation]
where d (cid:63)ij is the transformation of δ ij on the solution of the Isotonic Regression.
Concerning the performances of the exploration algorithm, we used two classic measures addressing the two main interests of those algorithms:
• Exploration: We monitored the Exploration Ratio (ER) through iterations.
This measure consider a discretized Outcome Space
(In our case, 10 cells per dimensions), and measures:
[Equation]
where V d is the ensemble of visited cells and S d is the ensemble of all state cells.
Competence:
We estimated the Mean Squared Error (MSE) of the meta-policy for a set of random points sampled uniformly in the underlying environment state space and projected into the Outcome Space to deﬁne goals:
[Equation]
with o the outcome of the experiment.
G A DDITIONAL R ESULTS
Explored Cells in Arm-Ball Environments In Figure 9, we have represented the locations explored in example runs of each architectures in the Arm-Ball environment.
We can clearly see the poor level of performances of the RPE architecture, compared to the RGE-FI.
Correlation coefﬁcients between representation quality and exploration performances In Table 1, we have reported the absolute of common correlations coefﬁcients between representation quality measures and exploration performances.
[Table]
MSE Evolution for different stationary Goal Policy In Figure 10 we have represented the evolution of Meta-Policy MSE through exploration episodes, for RGE-RFVAE and RGE-VAE, under sampling following different stationary Goal Policy
[Table] [Table]
[Figure] [Figure]