A Policy Gradient Method for Task-Agnostic Exploration
Abstract
In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy?
In this paper, we argue that the entropy of the state distribution induced by limited-horizon trajectories is a sensible target.
Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a nonparametric, k -nearest neighbors estimate of the state distribution entropy.
In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics.
Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning a variety of meaningful reward-based tasks downstream.
1. Introduction
In recent years, Reinforcement Learning (RL) (Sutton &
Barto, 2018) has achieved outstanding results in remarkable tasks, such as Atari games (Mnih et al., 2015), Go (Silver et al., 2016), Dota
(Berner et al., 2019) and dexterous manipulation (Andrychowicz et al., 2020).
To accomplish these feats, the learning process usually requires a considerable amount of human supervision, especially a hand-crafted reward function (Hadﬁeld-Menell et al., 2017), while the outcome rarely generalizes beyond a single task (Cobbe et al., 2019).
This barely mirrors human-like learning, which is far less dependent on exogenous guidance and exceptionally general.
Notably, an infant would go through an intrinsically-driven, nearly exhaustive, exploration of the environment in an early stage, without knowing much about the tasks she/he will face.
Still, this same unsupervised process will be consequential to solve those complex, externally-driven tasks, when they will eventually arise.
In this perspective, what is a suitable task-agnostic exploration objective to set for the agent in an unsupervised phase, so that the acquired knowledge would facilitate learning a variety of reward-based tasks afterwards?
Lately, several works have addressed this question in different directions.
In (Bechtle et al., 2019; Zheng et al., 2019), authors investigate how to embed task-agnostic knowledge into a transferable meta-reward function.
Other works (Jin et al., 2020; Tarbouriech et al., 2020) consider the active estimation of the environment dynamics as an unsupervised objective.
Another promising approach, which is the one we focus in this paper, is to incorporate the unsupervised knowledge into a task-agnostic exploration policy , obtained by maximizing some entropic measure over the state space (Hazan et al., 2019; Tarbouriech & Lazaric, 2019;
Mutti & Restelli, 2020; Lee et al., 2019).
Notably, an exploration policy is both easier to transfer than a transition model, which would be hardly robust to changes in the environment, and more ready to use than a meta-reward function, which would still require optimizing a policy as an intermediate step.
An ideal maximum-entropy policy, thus inducing a uniform distribution over states, is an extremely general starting point to solve any (unknown) subsequent goal-reaching task, as it minimizes the so-called worst-case regret (Gupta et al., 2018, Lemma 1).
In addition, by providing an efﬁcient estimation of any, possibly sparse, reward function, it signiﬁcantly reduces the burden on reward design.
In tabular settings, Tarbouriech & Lazaric (2019); Mutti & Restelli (2020) propose theoretically-grounded methods for learning an exploration policy that maximizes the entropy of the asymptotic state distribution, while Mutti & Restelli (2020) concurrently consider the minimization of the mixing time as a secondary objective.
In (Hazan et al., 2019), authors present a principled algorithm (MaxEnt) to optimize the entropy of the discounted state distribution of a tabular policy, and a theoretically-relaxed implementation to deal with function approximation.
Similarly, Lee et al.
(2019) design a method (SMM) to maximize the entropy of the ﬁnite-horizon state distribution.
Both SMM and MaxEnt learn a maximum-entropy mixture of policies following this iterative procedure: ﬁrst, they estimate the state distribution induced by the current mixture to deﬁne an intrinsic reward, then, they learn a policy that optimizes this reward to be added to the mixture.
Unfortunately, the literature approaches to state entropy maximization either consider impractical inﬁnite-horizon settings (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), or output a mixture of policies that would be inadequate for non-episodic tasks (Hazan et al., 2019; Lee et al., 2019).
In addition, they would still require a full model of the transition dynamics (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), or a state distribution density estimation (Hazan et al., 2019; Lee et al., 2019), which hardly scale to complex domains.
In this paper, we present a novel policy-search algorithm (Deisenroth et al., 2013), to deal with task-agnostic exploration via state entropy maximization over a ﬁnitehorizon , which gracefully scales to continuous, highdimensional domains.
The algorithm, which we call Maximum Entropy POLicy optimization (MEPOL), allows learning a single maximum-entropy parameterized policy from mere interactions with the environment, combining nonparametric state entropy estimation and function approximation.
It is completely model-free as it requires neither to model the environment transition dynamics nor to directly estimate the state distribution of any policy.
Notably, the entropy of continuous distributions can be speculated by looking at how random samples drawn from them laid out over the support surface (Beirlant et al., 1997).
Intuitively, samples from a high entropy distribution would evenly cover the surface, while samples drawn from low entropy distributions would concentrate over narrow regions.
Backed by this intuition, MEPOL relies on a k -nearest neighbors entropy estimator (Singh et al., 2003) to asses the quality of a given policy from a batch of interactions.
Hence, it searches for a policy that maximizes this entropy index within a parametric policy space.
To do so, it combines ideas from two successful, state-of-the-art policy-search methods: TRPO (Schulman et al., 2015), as it performs iterative optimizations of the entropy index within trust regions around the current policies, and POIS (Metelli et al., 2018), as these optimizations are performed ofﬂine via importance sampling.
This recipe allows MEPOL to learn a maximum-entropy task-agnostic exploration policy while showing stable behavior during optimization.
The paper is organized as follows.
First, we report the basic background (Section 2) and some relevant theoretical properties (Section 3) that will be instrumental to subsequent sections.
Then, we present the task-agnostic exploration objective (Section 4), and a learning algorithm,
MEPOL, to optimize it (Section 5), which is empirically evaluated in Section 6.
In Section 7, we discuss related work.
The proofs of the theorems are reported in Appendix A.
The implementation of MEPOL can be found at https://github.com/muttimirco/mepol .
2. Preliminaries
In this section, we report the essential background and notation we will use throughout the paper.
Markov Decision Processes A discrete-time Markov Decision Process (MDP) (Puterman, 2014) is deﬁned by a tuple
M = ( S , A , P, R, d 0 ) , where S and A are the state space and the action space respectively, P ( s (cid:48) | s, a ) is a Markovian transition model that deﬁnes the conditional probability of the next state s (cid:48) given the current state s and action a , R ( s ) is the expected immediate reward when arriving in state s , and d 0 is the initial state distribution.
A trajectory τ ∈ T is a sequence of state-action pairs τ
=
( s 0 , a 0 , s 1 , a 1 , . . . ) .
A policy π ( a | s ) deﬁnes the probability of taking action a given the current state s .
We denote by Π the set of all stationary Markovian policies.
A policy π that interacts with an MDP, induces a t -step state distribution deﬁned as (let
[Equation]
for every t > 0 .
If the MDP is ergodic, it admits a unique steady-state distribution which is lim t →∞ d πt ( s ) =
d π ( s ) .
The mixing time t mix describes how fast the state distribution d πt converges to its limit, given a mixing threshold (cid:15) :
[Equation]
Differential Entropy Let f ( x ) be a probability density function of a random vector X taking values in R p , then its differential entropy (Shannon, 1948) is deﬁned as:
[Equation]
When the distribution f is not available, this quantity can be estimated given a realization of X
= { x i } Ni =
(Beirlant et al., 1997).
In particular, to deal with high-dimensional data, we can turn to non-parametric, k -Nearest Neighbors ( k -NN) entropy estimators of the form (Singh et al., 2003):
[Equation]
where Ψ is the digamma function, ln k − Ψ( k ) is a bias correction term, V ki is the volume of the hyper-sphere of radius R i =
| x i − x k -NN
i
| , which is the Euclidean distance between x i an its k -nearest neighbor x k -NN i , so that:
[Equation]
where Γ is the gamma function, and p the dimensions of X .
The estimator (1) is known to be asymptotically unbiased and consistent (Singh et al., 2003).
When the target distribution f (cid:48) differs from the sampling distribution f , we can provide an estimate of H ( f (cid:48) ) by means of an ImportanceWeighted (IW) k -NN estimator (Ajgl & ˇSimandl, 2011):
[Equation]
where W i = (cid:80) j ∈N ki w j , such that N ki is the set of indices of the k -NN of x i , and w j are the normalized importance weights of samples x j , which are deﬁned as:
[Equation]
As a by-product, we have access to a non-parametric IW k -NN estimate of the Kullback-Leibler (KL) divergence, given by (Ajgl & ˇSimandl, 2011):
[Equation]
Note that, when f (cid:48)
= f , w j = 1 / N , the estimator (2) is equivalent to (1), while (cid:98) D KL ( f || f (cid:48) ) is zero.
3. Analysis of the Importance-Weighted Entropy Estimator
In this section, we present a theoretical analysis over the quality of the estimation provided by (2).
Especially, we provide a novel detailed proof of the bias, and a new characterization of its variance.
In a similar ﬂavor as in (Singh et al., 2003, Theorem 8) for the estimator (1), we can prove the following theorem.
Theorem 3.1.
(Ajgl & ˇSimandl, 2011, Sec. 4.1) Let f be a sampling distribution, f (cid:48) a target distribution.
The estimator (cid:98) H k ( f (cid:48) | f ) is asymptotically unbiased for any choice of k .
Therefore, given a sufﬁciently large batch of samples from an unknown distribution f , we can get an unbiased estimate of the entropy of any distribution f (cid:48) , irrespective of the form of f and f (cid:48) .
However, if the distance between the two grows too large, a high variance might negatively affect the estimation.
Theorem 3.2.
Let f be a sampling distribution, f (cid:48) a target distribution.
The asymptotic variance of the estimator (cid:98) H k
( f (cid:48) | f ) is given by:
[Equation]
where w = f (cid:48) ( x ) f ( x ) , and C =
N
[Equation]
+1) is a constant.
4. A Task-Agnostic Exploration Objective
In this section, we deﬁne a learning objective for taskagnostic exploration, which is a fully unsupervised phase that potentially precedes a set of diverse goal-based RL phases.
First, we make a common regularity assumption on the class of the considered MDPs, which allows us to exclude the presence of unsafe behaviors or dangerous states when interacting with the environment.
Assumption 4.1.
For any policy π ∈ Π , the corresponding
Markov chain P π is ergodic.
Then, following a common thread in maximum-entropy exploration (Hazan et al., 2019; Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), and particularly (Lee et al., 2019), which focuses on a ﬁnite-horizon setting as we do, we deﬁne the task-agnostic exploration problem: maximize
[Equation]
where ¯ d T = 1 T (cid:80) Tt =1 d πt is the ﬁnite-horizon average state distribution .
An optimal policy w.r.t.
the introduced objective favors a maximal coverage of the state space into the ﬁnite-horizon T , irrespective of the state-visitation order.
Notably, the exploration horizon T has not to be intended as a given trajectory length, but rather as a parameter of the unsupervised exploration phase which allows to tradeoff exploration quality (i.e., state-space coverage) with exploration efﬁciency (i.e., mixing properties).
As the thoughtful reader might realize, optimizing Objective (4) is not an easy task.
Known approaches would require either to estimate the transition model in order to obtain average state distributions (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020), or to directly estimate these distributions through a density model (Hazan et al., 2019;
Lee et al., 2019).
In contrast to the literature, we turn to non-parametric entropy estimation without explicit state distributions modeling, deriving a more practical policy-search approach that we present in the following section.
5. The Algorithm
In this section, we present a model-free policy-search algorithm, called Maximum Entropy POLicy optimization (MEPOL), to deal with the task-agnostic exploration problem (4) in continuous, high-dimensional domains.
MEPOL searches for a policy that maximizes the performance index (cid:98) H k
( ¯ d T ( θ )) within a parametric space of stochastic differentiable policies Π Θ = { π θ : θ ∈ Θ ⊆ R q } .
The performance index is given by the non-parametric entropy estimator (1) where we replace f with the average state distribution ¯ d T ( ·| π θ )
=
¯ d T ( θ ) .
The approach combines the ideas of two successful policy-search algorithms, TRPO (Schulman
Input : exploration horizon T , sample-size N , trust-region threshold δ , learning rate α , nearest neighbors k initialize θ for epoch = 1 , 2 , . . .
, until convergence do draw a batch of (cid:100) NT (cid:101) trajectories of length T with π θ build a dataset of particles
D τ = { ( τ ti , s i ) } Ni =1 θ (cid:48)
= IS-Optimizer
( D τ , θ )
θ ← θ (cid:48) end for Output : task-agnostic exploration policy π θ
IS-Optimizer
Input : dataset of particles D τ , sampling parameters θ initialize h =
and θ h
= θ while D KL ( ¯ d T ( θ 0 )
|| ¯ d T ( θ h )) ≤ δ do compute a gradient step: θ h +
=
θ h
+ α ∇ θ h (cid:98) H k (cid:0) ¯ d T ( θ h ) | ¯ d T ( θ 0 ) (cid:1)
h ← h
+end
while Output : parameters θ h et al., 2015) and POIS (Metelli et al., 2018), as it is reported in the following paragraphs.
Algorithm 1 provides the pseudocode for MEPOL.
Trust-Region Entropy Maximization
The algorithm is designed as a sequence of entropy index maximizations, called epochs, within a trust-region around the current policy π θ (Schulman et al., 2015).
First, we select an exploration horizon T and an estimator parameter k ∈ N .
Then, at each epoch, a batch of trajectories of length T is sampled from the environment with π θ , so as to take a total of N samples.
By considering each state encountered in these trajectories as an unweighted particle, we have D
=
{ s i } Ni =1 where s i ∼ ¯ d T ( θ ) .
Then, given a trust-region threshold δ , we aim to solve the following optimization problem:
[Equation]
subject to D KL
[Equation]
The idea is to optimize Problem (5) via Importance Sampling (IS) (Owen, 2013), in a fully off-policy manner partially inspired by (Metelli et al., 2018), exploiting the IW entropy estimator (2) to calculate the objective and the KL estimator (3) to compute the trust-region constraint.
We detail the off-policy optimization in the following paragraph.
Importance Sampling Optimization
We ﬁrst expand the set of particles D by introducing D τ = { ( τ ti , s i ) } Ni =1 , where τ ti =
( s
i , . . .
, s ti =
s i ) is the portion of the trajectory that leads to state s i .
In this way, for any policy π θ (cid:48) , we can associate to each particle its normalized importance weight:
[Equation]
Then, having set a constant learning rate α and the initial parameters θ
= θ , we consider a gradient ascent optimization of the IW entropy estimator (2),
[Equation]
until the trust-region boundary is reached, i.e., when it holds:
[Equation]
The following theorem provides the expression for the gradient of the IW entropy estimator in Equation (6)
[Equation]
Theorem 5.1.
Let π θ be the current policy and π θ (cid:48) a target policy.
The gradient of the IW estimator (cid:98) H k ( ¯ d T ( θ (cid:48) )
| ¯ d T ( θ )) w.r.t.
θ (cid:48) is given by:
[Equation]
where:
[Equation]
6. Empirical Analysis
In this section
[Equation]
we present a comprehensive empirical analysis, which is organized as follows:
We illustrate that MEPOL allows learning a maximumentropy policy in a variety of continuous domains, outperforming the current state of the art (MaxEnt); 6.2) We illustrate how the exploration horizon T , over which the policy is optimized, maximally impacts the trade-off between state entropy and mixing time; 6.3) We reveal the signiﬁcant beneﬁt of initializing an RL algorithm (TRPO) with a MEPOL policy to solve numerous challenging continuous control tasks.
A thorough description of the experimental set-up, along with some additional results and visualizations, are provided in Appendix B.
6.1. Task-Agnostic Exploration Learning
In this section, we consider the ability of MEPOL to learn a task-agnostic exploration policy according to the proposed objective (4).
Such a policy is evaluated in terms of its induced entropy value (cid:98)
H k ( ¯ d T ( θ )) , which we henceforth refer as entropy index .
We chose k to optimize the performance of the estimator, albeit experiencing little to none sensitivity
[Figure]
.sample
e n t r opy i nd e x (b) GridWorld 2 .5 ·
−−−sample e n t r opy i nd e x (c) MountainCar   ·− 3 .
sample e n t r opy i nd e x (d) Ant   ·−  sample e n t r opy i nd e x (e) Humanoid 2 . ·
−−− sample e n t r opy i nd e x (f) HandReach
[Figure]
to this parameter (Appendix B.3.4).
In any considered domain, we picked a speciﬁc T according to the time horizon we aimed to test in the subsequent goal-based setting (Section 6.3).
This choice is not relevant in the policy optimization process, while we discuss how it affects the properties of the optimal policy in the next section.
Note that, in all the experiments, we adopt a neural network to represent the parametric policy π θ (see Appendix B.2).
We compare our algorithm with MaxEnt (Hazan et al., 2019).
To this end, we considered the practical implementation of their algorithm to deal with continuous, non-discretized domains, 1 in which we employ TRPO instead of SAC (Haarnoja et al., 2018) for the RL component (see Appendix B.3.3).
Note that MaxEnt learns a mixture of policies rather than a single policy.
To measure its entropy index, we stick with the original implementation by generating a batch as follows: for each step of a trajectory, we sample a policy from the mixture and we take an action with it.
We also investigated SMM (Lee et al., 2019) as a potential comparison.
We do not report its results here for two reasons: we cannot achieve signiﬁcant performance w.r.t.
the random baseline, the difference with
MaxEnt is merely in the implementation.
First, we evaluate task-agnostic exploration learning over two continuous illustrative domains: GridWorld (2D states, 2D actions) and MountainCar (2D, 1D), which we depict in Figure 1a.
In these two domains, MEPOL successfully learns a policy that evenly covers the state space in a single batch of trajectories (state-visitation heatmaps are reported in Appendix B.3.5), while showcasing minimal variance across different runs (Figure 1b, 1c).
Notably, it signiﬁcantly outperforms MaxEnt in the MountainCar domain.
Then, we consider a set of continuous control, high-dimensional environments from the Mujoco suite (Todorov et al., 2012):
Ant (29D, 8D), Humanoid (47D, 20D), HandReach (63D, 20D).
While we learn a policy that maps full state representations to actions, we maximize the entropy index over a subset of the state sapce dimensions: 7D for Ant (3D position and 4D torso orientation), 24D for Humanoid (3D position, 4D body orientation, and all the joint angles), 24D for HandReach (full set of joint angles).
Note that this selection of features is not dictated by the inability of MEPOL to cope with even more dimensions (see Figure 2b), but to obtain reliable and visually interpretable behaviors (see Appendix B.3.1).
As we report in Figure 1d, 1e, 1f, MEPOL is able to learn policies with striking entropy values in all the environments.
As a by-product, it unlocks several meaningful high-level skills during the process, such as jumping, rotating, navigation (Ant), crawling, standing up (Humanoid), and basic coordination (Humanoid, HandReach).
Most importantly, the learning process is not negatively affected by the increasing number of dimensions, which is, instead, a well-known weakness of approaches based on explicit density estimation to compute the entropy (Beirlant et al., 1997).
This issue is documented by the poor results of MaxEnt, which struggles to match the performance of MEPOL in the considered domains, as it prematurely converges to a low-entropy mixture.
MountainCar
Ant
Humanoid samples 5 · 102 · 102 ·
MEPOL 4.31 ± 0.
3.67 ± 0.
1.92 ± 0.
MaxEnt 3.36 ± 0.
1.92 ± 0.
0.96 ± 0.
Random 1.98 ± 0.
1.86 ± 0.
0.84 ± 0.
[Figure]
On MaxEnt Results One might realize that the performance reported for the MaxEnt algorithm appears to be much lower than the one presented in (Hazan et al., 2019).
In this regard, some aspects need to be considered.
First, their objective is different, as they focus on the entropy of discounted stationary distributions instead of ¯ d T .
However, in the practical implementation, they consider undiscounted, ﬁnite-length trajectories as we do.
Secondly, their results are computed over all samples collected during the learning process, while we measure the entropy over a single batch.
Lastly, one could argue that an evaluation over the same measure ( k -NN entropy estimate) that our method explicitly optimize is unfair.
Nevertheless, even evaluating over the entropy of the 2D-discretized state space, which is the measure considered in (Hazan et al., 2019), leads to similar results (as reported in Figure 2a).
6.2. Impact of the Exploration Horizon Parameter
In this section, we discuss how choosing an exploration horizon T affects the properties of the learned policy.
First, it is useful to distinguish between a training horizon T , which is an input parameter to MEPOL, and a testing horizon h on which the policy is evaluated.
Especially, it is of particular interest to consider how an exploratory policy trained over T -steps fares in exploring the environment for a mismatching number of steps h .
To this end, we carried out a set of experiments in the aforementioned GridWorld and
Humanoid domains.
We denote by π ∗ T a policy obtained by executing MEPOL with a training horizon T and we consider the entropy of the h -step state distribution induced by π ∗ T .
Figure 2c (left), referring to the GridWorld experiment, shows that a policy trained over a shorter T might hit a peak in the entropy measure earlier (fast mixing), but other policies achieve higher entropy values at their optimum (highly exploring).
It is worth noting that the policy trained over 200 -steps becomes overzealous when the testing horizon extends to higher values, while derailing towards a poor h -step entropy.
In such a short horizon, the learned policy cannot evenly cover the four rooms and it overﬁts over easyto-reach locations.
Unsurprisingly, also the average state entropy over h -steps ( ¯ d h ), which is the actual objective we aim to maximize in task-agnostic exploration, is negatively affected, as we report in Figure 2c (right).
This result points out the importance of properly choosing the training horizon in accordance with the downstream-task horizon the policy will eventually face.
However, in other cases a policy learned over T -steps might gracefully generalize to longer horizons, as conﬁrmed by the Humanoid experiment (Figure 2d).
The environment is free of walls and hallways that can limit the agent’s motion, so there is no speciﬁc incentive to overﬁt an exploration behavior over a shorter T .
6.3. Goal-Based Reinforcement Learning
In this section, we illustrate how a learning agent can beneﬁt from an exploration policy learned by MEPOL when dealing with a variety of goal-based RL tasks.
Especially, we compare the performance achieved by TRPO (Schulman et al.,
epoch a v e r a g
e
r e t
u r n goal    0 .
epoch a v e r a g
e
r e t
u r n goal    0 .
epoch a v e r a g
e
r e t u r n goal(
[Figure] [Figure]
[Figure]
a v e r a g e r e t u r n
[Figure]
a v e r
[Figure]
e r e t u
[Figure]
n
[Figure]
a v e r a g e r e t u r n
[Figure]
initialized with a MEPOL policy (the one we learned in Section 6.1) w.r.t.
a set of signiﬁcant baselines 4 that learn from scratch, i.e., starting from a randomly initialized policy.
These baselines are: TRPO, SAC (Haarnoja et al., 2018), which promotes exploration over actions, SMM (Lee et al., 2019), which has an intrinsic reward related to the state-space entropy, ICM (Pathak et al., 2017), which favors exploration by fostering prediction errors, and
Pseudocount (Bellemare et al., 2016), which assigns high rewards to rarely visited states.
The algorithms are evaluated in terms of average return on a series of sparse-reward RL tasks deﬁned over GridWorld, Ant, and Humanoid environments we considered in the previous sections.
In GridWorld, we test three navigation tasks with different goal locations (see Figure 3a).
The reward is 1 in the states having Euclidean distance to the goal lower than 0.1.
For the Ant environment, we deﬁne three, incrementally challenging, tasks: Escape, Jump, Navigate.
In the ﬁrst, the Ant starts from an upside-down position and it receives a reward of 1 whenever it rotates to a straight position (Figure 3b).
In
Jump, the agent gets a reward of 1 whenever it jumps higher than three units from the ground (Figure 3c).
In Navigate, the reward is 1 in all the states further than 7 units from the initial location (Figure 3d).
Finally, in Humanoid Up, the agent initially lies on the ground and it receives a reward ofwhen it is able to stand-up (Figure 3e).
In all the considered tasks, the reward is zero anywhere except for the goal states, an episode ends when a goal is reached.
As we show in Figure 3, the MEPOL initialization leads to a striking performance across the board, while the tasks resulted extremely hard to learn from scratch.
In some cases (Figure 3b), MEPOL allows for zero-shot policy optimization, as the optimal behavior has been already learned in the unsupervised exploration stage.
In other tasks (e.g.,
Figure 3a)
, the MEPOL-initialized policy has lower return, but it permits for lighting fast adaptation w.r.t.
random initialization.
Note that, to match the tasks’ higher-level of abstraction, in Ant Navigate and Humanoid Up we employed MEPOL initialization learned by maximizing the entropy over mere spatial coordinates (x-y in Ant, x-y-z in Humanoid).
However, also the exact policies learned in
Section 6.1 fares remarkably well in those scenarios (see
Appendix B.4.3), albeit experiencing slower convergence.
7. Related Work
Our work falls into the category of unsupervised reinforcement learning.
Especially, we address the problem of taskagnostic exploration, i.e., how to learn an unsupervised exploration policy that generalizes towards a wide range of tasks.
This line of work relates to intrinsically-motivated learning, state-reaching approaches, and, to some extent, meta-RL.
In this section, we provide a non-exhaustive review of previous work in these areas.
In the context of (supervised) reinforcement learning, many works have drawn inspiration from intrinsic motivation (Chentanez et al., 2005; Oudeyer et al., 2007) to design exploration bonuses that help the agent overcoming the exploration barrier in sparsely deﬁned tasks.
Some of them, initially (Schmidhuber, 1991) and more recently (Pathak et al., 2017; Burda et al., 2019a), promotes exploration by fostering prediction errors of an environment model, which is concurrently improved during the learning process.
Another approach (e.g., Lopes et al., 2012; Houthooft et al., 2016) considers exploration strategies that maximizes the information gain, rather than prediction errors, on the agent’s belief about the environment.
Other works devise intrinsic rewards that are somewhat proportional to the novelty of a state, so that the agent is constantly pushed towards new portions of the state space.
Notable instances are count-based methods (e.g., Bellemare et al., 2016; Tang et al., 2017), and random distillation (Burda et al., 2019b).
Lately, Lee et al.
(2019) propose an exploration bonus that is explicitly related to the state distribution entropy, while they present an interesting unifying perspective of other intrinsic bonuses in the view of entropy maximization.
In the intrinsic motivation literature, other works tackle the problem of learning a set of useful skills in an unsupervised setting (Gregor et al., 2017; Achiam et al., 2018; Eysenbach et al., 2019).
The common thread is to combine variational inference and intrinsic motivation to maximize some information theoretic measure, which is usually closely related to empowerment (Salge et al., 2014; Mohamed & Rezende, 2015), i.e., the ability of the agent to control its own environment.
In (Gregor et al., 2017), the proposed method learns a set of diverse skills by maximizing the mutual information between the skills and their terminations states.
Eysenbach et al.
(2019) consider a mutual information objective as well, but computed over all the states visited by the learned skills, instead of just termination states.
Another relevant line of research focus on state-reaching objectives .
In tabular settings, Lim & Auer (2012) cast the problem in its most comprehensive form, proposing a principled method to incrementally solve any possible state-reaching task, from the easiest to the hardest.
In (Gajane et al., 2019), the same method is extended to nonstationary environments.
Other works, originally (Bonarini et al., 2006a;b) and lately (Ecoffet et al., 2019), provide algorithms to address a relaxed version of the state-reaching problem.
In their case, the goal is to learn reaching policies to return to some sort of promising states (instead of all the states), then to seek for novel states from there on.
In complex domains, Pong et al.
(2019) consider learning maximum-entropy goal generation to provide targets to the state-reaching component.
Notably, they relates this procedure to maximum-entropy exploration.
Meta-learning (Schmidhuber, 1987) has been successfully applied to address generalization in RL (e.g., Finn et al., 2017).
In this set-up, the agent faces numerous tasks in order to meta-learn a general model, which, then, can be quickly adapted to solve unseen tasks in a few shots.
These methods generally relies on reward functions to train the meta-model.
Instead, Gupta et al.
(2018) considers an unsupervised meta-RL set-up, which has some connections with our work.
However, their focus is directed on learning fast adaptation rather than exploration, which is delegated to a method (Eysenbach et al., 2019)
we already mentioned.
8. Conclusions
In this paper, we addressed task-agnostic exploration in environments with non-existent rewards by pursuing state entropy maximization.
We presented a practical policy-search algorithm, MEPOL, to learn an optimal task-agnostic exploration policy in continuous, high-dimensional domains.
We empirically showed that MEPOL achieves outstanding performance in terms of state entropy maximization, and that the learned policy paves the way for solving several rewardbased tasks downstream.
Natural extensions of MEPOL might target vision-based domains, which would require a meaningful distance metric between images, and explicitly accounting for the uncertainty of the entropy estimator in the off-policy component.
Promising future directions includes addressing task-agnostic exploration in unsupervised meta-learning set-ups, and contemplating the use of a non-parametric state entropy regularization in reward-based policy optimization.
We believe that this work can shed some light on the potential of state entropy maximization approaches to perform task-agnostic exploration.
A. Proofs
Theorem 3.1.
(Ajgl & ˇSimandl, 2011, Sec. 4.1) Let f be a sampling distribution, f (cid:48) a target distribution.
The estimator (cid:98) H k ( f (cid:48) | f ) is asymptotically unbiased for any choice of k .
Proof.
The proof follow the sketch reported in (Ajgl & ˇSimandl, 2011, Section 4.1).
First, We consider the estimator (cid:98) G k ( f (cid:48) | f ) = (cid:98) H k
( f (cid:48) | f
[Equation]
− ln k + Ψ
[Equation]
k ) , that is:
[Equation]
By considering its expectation w.r.t.
the sampling distribution we get:
[Equation]
where, for the sake of clarity, we will replace each logarithmic term as:
[Equation]
Since we are interested in the asymptotic mean, we can notice that for N → ∞ we have W i / k
→ w i / N , where w i = f (cid:48) ( x ) / f
( x ) are the unnormalized importance weights (Ajgl & ˇSimandl, 2011, Section 4.1).
Thus, for N → ∞ , we can see that:
[Equation]
where the random variables T 1 , T 2 , . . .
, T N are identically distributed, so that:
[Equation]
Thus
[Equation]
to compute the expectation
[Equation]
we have to characterize the following probability for any real number r and any x ∼ f (cid:48) :
[Equation]
where we have:
[Equation]
We can rewrite the probability as a binomial:
[Equation]
where P ( S ρ r ,x ) is the probability of x lying into the sphere of radius ρ r , denoted as S ρ r ,x .
Then, we employ the Poisson
Approximation (Hodges & Le Cam, 1960) to this binomial distribution, reducing it to a poisson distribution having parameter:
[Equation]
Therefore
[Equation]
we get:
[Equation]
such that the random variable T x has the pdf:
[Equation]
Finally
[Equation]
we can compute the expectation following the same steps reported in (Singh et al., 2003, Theorem 8):
[Equation]
which for a generic x it yields:
[Equation]
Theorem 3.2.
Let f be a sampling distribution, f (cid:48) a target distribution.
The asymptotic variance of the estimator (cid:98) H k
( f (cid:48) | f ) is given by:
[Equation]
where w = f (cid:48) ( x ) f ( x ) , and C =
N π p/ 2 k Γ( p / 2 +1) is a constant.
Proof.
We consider the limit of the variance of (cid:98) H k
( f (cid:48) | f ) , we have:
[Equation]
where (cid:98) G k
( f (cid:48) | f ) is the estimator without the bias correcting term (7), and T i are the logarithmic terms (8).
Then, since the distribution of the random vector (cid:0) w 1 T 1 , w 2 T 2 , . .
. , w N T N (cid:1) is the same as any permutation of it (Singh et al., 2003):
[Equation]
Assuming that, for N → ∞
[Equation]
the term Cov( w 1 T 1 , w 2 T 2 ) → 0 as its non-IW counterpart (Singh et al., 2003, Theorem 11)
[Equation]
we are interested on the ﬁrst term Var x ∼ f (cid:2) w 1 T 1 (cid:3) .
Especially, we can derive:
[Equation]
where in the following
[Equation]
we will substitute C =
[Equation]
Then, we can write the second momentum as:
[Equation]
while the squared expected value is:
[Equation]
Thus
[Equation]
we have:
[Equation]
Summing it up, we can write the asymptotic order of the variance as:
[Equation]
N
Theorem 5.1.
Let π θ be the current policy and π θ (cid:48) a target policy.
The gradient of the IW estimator (cid:98) H k ( ¯ d T ( θ (cid:48) )
| ¯ d T ( θ ))
w.r
[Equation]
t. θ (cid:48) is given by:
[Equation]
where:
[Equation]
Proof.
We consider the IW entropy estimator (2)
[Equation]
that is
[Equation]
where:
[Equation]
Then
[Equation]
by differentiating Equation (9) w.r
[Equation]
t. θ (cid:48)
[Equation]
we have:
[Equation]
Finally
[Equation]
we consider the expression of ∇ θ (cid:48) w j in Equation (10) to conclude the proof:
[Equation]
|
[Equation]
=
[Equation]
rod
[Equation]
B. Empirical Analysis: Further Details B.1. Environments
For all the environments, we use off-the-shelf implementations from the OpenAI gym library (Brockman et al., 2016) with the exception of GridWorld, which we coded from scratch and we describe in the next paragraph.
We also slightly modiﬁed the MountainCar environment by adding a wall on top of the right mountain to make the environment non-episodic.
In GridWorld, the agent can navigate a map composed of four rooms connected by four hallways, as represented in Figure 4.
At each step the agent can choose how much to move on the x and y axes.
The maximum continuous absolute change in position along any of the axes is 0.2.
Each room is a space of 5 by 5 units, thus, the agent needs around 50 steps to move, along a straight line, from one side of the room to the other.
Any action that leads the agent to collide with a wall is ignored and the agent remains in the previous state (position).
B.2. Class of Policies
In all the experiments, the policy is a Gaussian distribution with diagonal covariance matrix.
It takes as input the environment state features and outputs an action vector a ∼ N ( µ, σ 2 ) .
The mean µ is state-dependent and is the downstream output of a densely connected neural network.
The standard deviation is state-independent and it is represented by a separated trainable vector.
The dimension of µ , σ and a vectors is equal to the action-space dimension of the environment.
B.3. Task-Agnostic Exploration Learning B.3.1. C ONTINUOUS C ONTROL S ET -U P
Here we further comment the experimental set-up for continuous control domains that we have brieﬂy described in
Section 6.1, especially concerning how we select the set of features on which the entropy index is maximized.
First, in the
Ant domain, we maximize the entropy over a 7D space of spatial coordinates (3D) and torso orientation (4D), excluding joint angles and velocities.
This is to obtain an intermediate hurdle, in terms of dimensionality, w.r.t.
the smaller GridWorld and MountainCar, and the most complex Humanoid and HandReach.
Instead, in the latter two domains, we essentially maximize the entropy over the full state space excluding velocities and external forces, so that we have a 24D space both in Humanoid (3D position, 4D body orientation, and all the joint angles) and HandReach (all the joint angles).
We noted that including external forces does not affect the entropy maximization in a meaningful way, as they resulted always zero during training.
The reason why we also discarded velocities from the entropy computation is twofold.
First, we noted that velocity-related features are quite unstable and spiky, so that they can be harmful without normalization, which we avoided.
Secondly, we think that maximizing the entropy over the largest set of features is not necessarily a good idea when targeting generalization (see B.4.3), especially if improving the entropy over some features reduces the entropy over some other (as in the case of positions/angles and velocities).
B.3.2. MEPOL
For what concerns the k-nearest neighbors computation we use the neighbors package from the scikit-learn library (Pedregosa et al., 2011), which provides efﬁcient algorithms and data structures to ﬁrst compute and then query nearest neighbors.
Before starting the main loop of the algorithm we perform some training steps to force the policy to output a zero µ vector.
This is instrumental to obtain a common starting point across all the seeds and can be removed without affecting the algorithm behavior.
As outlined in the pseudcode of Algorithm 1, in each epoch a dataset of particles D τ = { ( τ ti , s i ) } Ni =1 is gathered for the given time horizon T .
We call N traj the number of trajectories, or batch size, used to build the dataset, so that N = N traj ∗ T
[Table]
[Table]
B.3.3. M AX E NT
As already outlined in Section 6.1, we use the original MaxEnt implementation to deal with continuous domains ( https://github.com/abbyvansoest/maxent/tree/master/humanoid ).
We adopt this implementation also for the Ant and MountainCar environments, which were originally presented only as discretized domains ( https://github.com/abbyvansoest/maxent_base and https://github.com/ abbyvansoest/maxent/tree/master/ant ).
This allowed us not only to work in a truly continuous setting, as we do in MEPOL, but also to obtain better results than the discretized version.
We were able to further enhance the
MaxEnt performance by replacing SAC with TRPO.
In the tables below, you can ﬁnd all the parameters and corresponding ranges (or sets) of values over which we searched for their optimal values.
The TRPO parameters are reported employing the same notation as in (Duan et al., 2016).
To estimate the density of the mixture, in Ant and Humanoid, we use a higher time horizon ( T d ) than the optimized one ( T ).
The reason why we do this is that, otherwise, we were not able to obtain reliable density estimations.
The batch size used for the density estimation is denoted as N traj d .
We also report the number of neighbors ( k ), which does not affect the learning process of
MaxEnt, that we used to calculate the entropy of the mixture in the plots of Section 6.1.
The entropy is computed over the same time horizon T and the same batch size N traj used in MEPOL.
Note that the horizon reported for TRPO is the same as the objective horizon T .
The neural policy architectures are not reported as they are the same as in Section B.3.2.
[Table]
, Epanechnikov
[Table] [Table]
No
[Table] [Table]
[Table]
Value
Search In
[Table]
B.3.4. P ARAMETERS S ENSITIVITY
In Figure
[Figure]
we show how the selection of the main parameters of MEPOL impacts on the learning process.
To this end
[Figure]
we consider a set of experiments in the illustrative MountainCar domain
[Figure]
where we vary one parameter at a time to inspect the change in entropy index.
As we can notice, the algorithm shows little sensitivity to the number of neighbors ( k ) considered in the entropy estimation.
Allowing off-policy updates through an higher KL threshold δ positively impacts the learning efﬁciency.
Furthermore, MEPOL displays a good behavior even when we limit the batch-size.
[Figure]
[Table]
B.3.5. S TATE -V ISITATION H EATMAPS
In Figure 6, and Figure 7, we report the MEPOL state-coverage evolution over GridWorld and MountainCar domains.
In
Figure 8, you can see the state-coverage evolution in the Ant domain over a 12 by 12 units space, which is centered in the
Ant starting position.
The well-deﬁned borders in the heatmap are due to trajectories going out of bounds.
These heatmaps were created while running the experiments presented in Section 6.1 by discretizing the continuous state-space.
B.3.6. D ISCRETE E NTROPY
In Figure we report the plots for the evaluation of the entropy on the 2D-discretized state-space from which we have taken the values reported in Figure 2a.
In Ant and Humanoid, the considered state-space is the 2D, discretized, agent’s position (x, y).
These plots were created while running the experiments in Section 6.1.
MEPOL
Random
MaxEnt   ·   sample d
i s c r e t e e n t r opy (a) MountainCar   ·  sample d i s c r e t e e n t r opy (b)
Ant   · 1 .
sample d i s c r e t e e n t r opy (c)
Humanoid
B.4. Goal-Based Reinforcement Learning B.4.1. A LGORITHMS
We use the TRPO implementation from OpenAI’s SpinningUp library https://github.com/openai/ spinningup .
For SAC, we adopt the codebase from https://github.com/microsoft/oac-explore .
We use the original SMM codebase https://github.com/RLAgent/state-marginal-matching , which provides also an implementation of ICM, and Pseudocount.
B.4.2. P ARAMETERS D ETAIL
In Table 6, we report the TRPO parameters used in the tasks, following the same notation in (Duan et al., 2016).
Both the basic agent and the MEPOL agent use the same parameters.
In Table 7, we report the SAC parameters, following the notation in (Haarnoja et al., 2018).
In SMM we use 4 skills.
In SMM, ICM, and Pseudocount we equally weight the extrinsic and the intrinsic components, as we haven’t seen any improvement doing otherwise.
Note that SMM, ICM, and
Pseudocount are built on top of SAC for which we adopt the same parameters as in Table 7.
The neural policy architectures are not reported as they are the same as in Section B.3.2.
[Table]
[Table]
B.4.3. H IGHER -L EVEL AND L OWER -L EVEL P OLICIES
In this section, we discuss the performance achieved by MEPOL policies when facing higher-level tasks, such as 2D navigation (Figure 10a) and standing-up (Figure 10b).
Especially, we can see that higher-level MEPOL policies, which are trained to maximize the entropy over spatial coordinates (x-y in Ant, x-y-z in Humanoid), outperform lower-level
MEPOL policies, which are trained to maximize entropy over spatial coordinates, orientation, and joint angles (as reported in Section 6.1).
This is not surprising, since higher-level policies better match the level of abstraction of the considered tasks.
However, it is worth noting that also lower-level policies achieve a remarkable initial performance, and a positive learning trend, albeit experiencing lower convergence.
[Figure]
B.5. Additional Experiments
In this section, we present two additional experiments.
First, we present another experiment in the HandReach domain in which we maximize the entropy over the 3D position of each of the ﬁngertip, for a total of 15 dimensions, whose performance is reported in Figure 11.
Then, we present MEPOL in the AntMaze domain, in which the objective is to uniformly cover the 2D position of the agent, moving in a maze.
The performance is reported, together with a visual representation of the environment, and a state-visitation evolution, in Figure 12.
[Figure] [Figure]
· 1 . 2 .sample
e n t r opy i nd e x
MEPOL
Random
[Figure]