Learning to Explore a Class of Multiple Reward-Free Environments
Abstract
Several recent works have been dedicated to the pure exploration of a single reward-free environment.
Along this line, we address the problem of learning to explore a class of multiple rewardfree environments with a unique general strategy, which aims to provide a universal initialization to subsequent reinforcement learning problems speciﬁed over the same class.
Notably, the problem is inherently multi-objective as we can trade off the exploration performance between environments in many ways.
In this work, we foster an exploration strategy that is sensitive to the most adverse cases within the class.
Hence, we cast the exploration problem as the maximization of the mean of a critical percentile of the state visitation entropy induced by the exploration strategy over the class of environments.
Then, we present a policy gradient algorithm, MEMENTO, to optimize the introduced objective through mediated interactions with the class.
Finally, we empirically demonstrate the ability of the algorithm in learning to explore challenging classes of continuous environments and we show that reinforcement learning greatly beneﬁts from the pre-trained exploration strategy when compared to learning from scratch.
1. Introduction
The typical Reinforcement Learning (RL, Sutton & Barto, 2018) setting involves a learning agent interacting with an environment in order to maximize a reward signal.
In principle, the reward signal is a given and perfectly encodes the task.
In practice, the reward is usually hand-crafted and designing it to make the agent learn a desirable behavior is often a huge challenge.
This poses a serious roadblock on the way of autonomous learning, as any task requires a costly and speciﬁc formulation, while the synergy between solving one RL problem and another is very limited.
To address this crucial limitation, Jin et al.
(2020) have formulated the reward-free RL problem.
In this setting, the agent is tasked with mastering an environment without rewards, so that the knowledge it acquires can be eventually deployed to solve any RL problem one could specify in this same environment.
Mastering the environment has been formulated in (i) learning to model its transition dynamics (Jin et al., 2020; Tarbouriech et al., 2020b;a; Zhang et al., 2020b), or (ii) learning to explore it with a general, task-agnostic, strategy (Hazan et al., 2019).
Although they overcome the reliance on a reward function, previous solutions to rewardfree RL are still severely environment-speciﬁc.
In this work, we aim to push the generality of reward-free learning even further by addressing the problem of learning to explore multiple environments with a single exploration strategy.
Especially, the agent faces a class of reward-free environments that belong to the same domain but differ in their transition dynamics.
At each turn of the learning process, the agent is drawn into an environment within the class, where it can interact for a ﬁnite number of steps before facing another turn.
The process carries on sequentially for a ﬁnite number of turns.
The ultimate goal of the agent is to learn an exploration strategy that helps to solve any subsequent RL task speciﬁed over the class.
Our contribution to the problem is three-fold: (c1) We frame it into a tractable formulation (Section 3), (c2) we propose a methodology to address it (Section 4), for which (c3) we provide a thorough empirical evaluation (Section 5).
First, we extend a reward-free objective meant for environmentspeciﬁc exploration, which is the Maximum State Visitation
Entropy (MSVE, Hazan et al., 2019).
The underlying intuition is that a general exploration strategy has to visit with high probability any state where the agent might be rewarded in a subsequent RL task.
When dealing with multiple environments, this becomes a multi-objective problem, as one could establish any combination of preferences over the environments.
In this work, instead of na¨ıvely optimizing the mean of the state visitation entropy across the class, we consider its Conditional Value-at-Risk (CVaR, Rockafellar et al., 2000) to prioritize performance in particularly rare or adverse environments.
We propose a policy gradient algorithm
(Deisenroth et al., 2013), Multiple Environments
Maximum ENTropy Optimization (MEMENTO), to optimize the learning objective via mere interactions with the class of environments.
As in recent works (Mutti et al., 2021;
Liu &
Abbeel, 2021a; Seo et al., 2021), the algorithm employs nonparametric methods to deal with state entropy estimation in continuous and high-dimensional environments.
Then, it leverages these estimated values to optimize the CVaR of the entropy by following its policy gradient (Tamar et al., 2015).
Finally, we provide an experimental analysis of the proposed method in a two-stage setting.
First, we show that it is effective in training general exploration strategies over classes of continuous and high-dimensional environments without rewards.
Second, we test the obtained exploration strategies as initialization for a number of RL tasks deﬁned over the same class.
Notably, the trained exploration strategy allows us to solve sparse-rewards tasks that are impractical to learn from scratch, while being robust to the most unfavorable environment thanks to the CVaR objective.
A preliminary theoretical analysis is in Section 6, Related works are in
Appendix A, the proofs of the theorems are in Appendix B.
2. Preliminaries
In this section we report background and notation.
A vector v is in bold, and v i is its i -th entry.
Probability and Percentiles Let X be a random variable distributed according to a cumulative density function (cdf)
F X ( x ) =
Pr ( X ≤ x ) .
We denote with E [ X ] , V ar[ X ] the expected value and the variance of X respectively.
Let α ∈ (0 , 1) be a conﬁdence level, we call the α -percentile (shortened to α %) of the variable X its Value-at-Risk (VaR), which is deﬁned as
[Equation]
Analogously, we call the mean of this same α -percentile the
Conditional Value-at-Risk (CVaR) of X ,
[Equation]
Markov Decision Processes A Controlled Markov
Process (CMP) is a tuple M :=
( S , A , P, D ) , where S is the state space,
A is the action space, the transition model P ( s (cid:48) | a, s ) denotes the conditional probability of reaching state s (cid:48) ∈ S when selecting action a ∈ A in state s ∈ S , and D is the initial state distribution.
The behavior of an agent is described by a policy π ( a | s ) , which deﬁnes the probability of taking acion a ∈ A in s ∈ S .
Let Π be the set of all the stationary policies.
Executing a policy π ∈ Π in a CMP over a time horizon T generates a trajectory τ =
( s 0 ,τ , a 0 ,τ , . .
. , a T − 2 ,τ , s T − 1 ,τ ) with probability
[Equation]
We denote the state-visitation frequencies induced by τ with d τ ( s )
= 1 T (cid:80) T − 1 t =0 1 ( s t,τ = s ) , and we call d M π = [Equation] ∼ p π, M [ d τ ] the marginal state distribution.
We deﬁne the differential entropy (Shannon, 1948) of d τ as H ( d τ ) =
− (cid:82) S
d τ ( s ) log d τ ( s ) d s. For simplicity, we will write H ( d τ ) as a random variable
H τ ∼ δ ( h − H ( d τ ))
p π, M ( τ ) , where δ ( h ) is a Dirac delta.
By coupling a CMP M with a reward function R we obtain a Markov Decision Process (MDP, Puterman, 2014)
M R :=
M∪ R .
Let R ( s, a ) be the expected immediate reward when taking a ∈ A in s ∈ S and let R ( τ )
=
(cid:80) T − 1 t =
R ( s t,τ ) , the performance of a policy π over the MDP M R is deﬁned as
[Equation]
The goal of reinforcement learning (Sutton & Barto, 2018) is to ﬁnd an optimal policy π ∗J ∈ arg max J M R ( π ) through sampled interactions with an unknown MDP M R .
3. Learning to Explore Multiple Environments
Let M =
{M 1 , . . .
, M I } be a class of unknown CMPs, in which every element M i =
( S , A , P i , D ) has a speciﬁc transition model P i , while S , A , D are homogeneous across the class.
At each turn, the agent is able to interact with a single environment M ∈ M .
The selection of the environment to interact with is mediated by a distribution p M over
M , outside the control of the agent.
The aim of the agent is to learn an exploration strategy that is general across all the
MDPs M R one can build upon M .
In a single-environment setting, this problem has been assimilated to learning a policy that maximizes the entropy of the induced state visitation frequencies (Hazan et al., 2019; Lee et al., 2019; Mutti et al., 2021).
One can straightforwardly extend the objective to multiple environments by considering the expectation over the class of CMPs, E M ( π ) =
E M∼ p M τ ∼ p π, M (cid:2)
H τ (cid:3) , where the usual entropy objective over the single environment M i can be easily recovered by setting p M i = 1 .
However, this objective function does not account for the tail behavior of H τ , i.e., for the exploration performance in environments of M that are rare or particularly unfavorable.
This is decidedly undesirable as the agent may be tasked with an MDP built upon one of these adverse environments in the subsequent
RL phase, where even an optimal strategy w.r.t.
E M ( π ) may fail to provide sufﬁcient exploration.
To overcome this limitation, we look for a more nuanced exploration objective that balances the expected performance with the sensitivity to the tail behavior.
By taking inspiration from the riskaverse optimization literature (Rockafellar et al., 2000), we consider the CVaR of the state visitation entropy induced by π over M ,
[Equation]
where α is a conﬁdence level and E 1 M ( π ) :
= E M ( π ) .
The lower we set the value of α , the more we hedge against the possibility of a bad exploration outcome in some M ∈ M .
In the following sections, we propose a method to effectively learn a policy π ∗E ∈ arg max E α M ( π ) through mere interactions with M , and we show how this serves as a pre-training for RL (Figure 1).
4. A Policy Gradient Approach
In this section, we present an algorithm, called Multiple Environments Maximum ENTropy Optimization (MEMENTO), to optimize the exploration objective in (2) through mediated interactions with a class of continuous environments.
MEMENTO operates as a typical policy gradient approach (Deisenroth et al., 2013).
It directly searches for an optimal policy by navigating a set of parametric differentiable policies Π Θ :=
{ π θ : θ ∈ Θ ⊆ R n } .
It does so by repeatedly updating the policy parameters θ in the gradient direction, until a stationary point is reached.
This update has the form
[Equation]
where β is a learning rate, and ∇ θ E α M ( π θ ) is the gradient of (2) w.r.t.
θ .
The following proposition provides the formula of ∇ θ E α M ( π θ ) .
The derivation follows closely the one in (Tamar et al., 2015, Proposition 1), which we have adapted to our objective function of interest (2).
Proposition 4.1.
The policy gradient of the exploration objective E α M ( π θ ) w.r.t.
θ is given by
[Equation]
Algorithm
Input : percentile α , learning rate β
Output : policy π θ 1: initialize θ 2: for epoch = 0 , 1 , . . .
, until convergence do 3: for i = 1 , 2 , . . .
, N do 4: sample an environment M i ∼ p M 5: sample a trajectory τ i ∼ p π θ , M i 6: estimate
H τ i with (4) 7: end for 8: estimate VaR α ( H τ ) with (5) 9: estimate ∇ θ E α M
( π θ ) with (6) 10: update parameters θ ← θ + β (cid:98) ∇ θ E α M
( π θ ) 11: end for
However, in this work we do not assume full knowledge of the class of CMPs M , and the expected value in Proposition 4.1 cannot be computed without having access to p M and p π θ , M .
Instead, MEMENTO computes the policy update via a Monte Carlo estimation of ∇ θ E α M from the sampled interactions { ( M i , τ i ) } Ni =1 with the class of environments M .
The policy gradient estimate itself relies on a Monte Carlo estimate of each entropy value
H τ i from τ i , and a Monte Carlo estimate of VaR α
( H τ ) given the estimated { H τ i } Ni =1 .
The following paragraphs describe how these estimates are carried out, while Algorithm 1 provides the pseudocode of MEMENTO.
Additional details and implementation choices can be found in Appendix C.
Entropy Estimation We would like to compute the entropy
H τ i of the state visitation frequencies d τ i from a single realization { s t,τ i } T − 1 t =
⊂ τ i .
This estimation is notoriously challenging when the state space is continuous and highdimensional S ⊆ R p .
Taking inspiration from recent works pursuing the MSVE objective (Mutti et al., 2021; Liu &
Abbeel, 2021a; Seo et al., 2021), we employ a principled k -Nearest Neighbors ( k -NN) entropy estimator (Singh et al., 2003) of the form
[Equation]
where Γ is the Gamma function, (cid:107) · (cid:107) is the Euclidean distance, and s k -NN t,τ i ∈
τ i is the k -nearest neighbor of s t,τ i .
The intuition behind the estimator in (4) is straightforward: We can suppose the state visitation frequencies
d τ i to have a high entropy as long as the average distance between any encountered state and its k -NN is large.
Despite its simplicity, a Euclidean metric sufﬁces to get reliable entropy estimates in continuous control domains (Mutti et al., 2021).
VaR Estimation and Baseline
The last missing piece to get a Monte Carlo estimate of the policy gradient ∇ θ E α M is the value of VaR α
( H τ ) .
Being H [1] , . .
, H [ N ] the order statistics out of the estimated values { (cid:98) H τ i } Ni =1 , we can na¨ıvely estimate the VaR as
[Equation]
Albeit asymptotically unbiased, the VaR estimator in (5) is known to suffer from a large variance in ﬁnite sample regimes (Kolla et al., 2019), which is aggravated by the error in the upstream entropy estimates, which provide the order statistics.
This variance is mostly harmless when we use the estimate to ﬁlter out entropy values beyond the α %, i.e., the condition H τ ≤ VaR α ( H τ ) in Proposition 4.1.
Instead, its impact is signiﬁcant when we subtract it from the values within the α %, i.e., the term H τ − VaR α ( H τ ) in Proposition 4.1.
To mitigate this issue, we consider a convenient baseline b
= − VaR
α
( H τ ) to be subtracted from the latter, which gives the Monte Carlo policy gradient estimator
[Equation]
where f τ i = (cid:80) T − 1 t =0 ∇ θ log π θ ( a t,τ i | s t,τ i ) .
Notably, the baseline b trades off a lower estimation error for a slight additional bias in the estimation (6).
We found that this baseline leads to empirically good results and we provide some theoretical corroboration over its beneﬁts in Appendix C.1.
5. Empirical Evaluation
In this section, we provide an extensive empirical evaluation of the proposed methodology over the two-phase learning process described in Figure 1, which is organized as follows:
5.1 We show the ability of our method in learning to explore a class of illustrative continuous gridworlds, emphasizing the importance of the percentile sensitivity; 5.2 We discuss how the choice of the percentile of interest affects the exploration strategy; 5.3 We highlight the beneﬁt that the exploration strategy provides to RL on the same class; 5.4 We verify the ability of our method to scale with the size of the class of environments, by considering a class of 10 continuous gridworlds; 5.5 We verify the ability of our method to scale with the dimensionality of the environments in the class, by considering a class of 29D continuous control Ant domains; 5.6 We verify the ability of our method to scale with visual inputs, by considering a class of 147D MiniGrid (Chevalier-Boisvert et al., 2018) domains; 5.7 We show that the exploration strategy learned with our approach is superior for RL w.r.t.
a policy meta-trained with MAML (Finn et al., 2017; Gupta et al., 2018a) on the same class.
A thorough description of the experimental setting is pro- vided
in
Appendix D.
5.1. Learning to Explore Multiple Environments with Percentile Sensitivity
In this section, we consider a class M composed of two different conﬁgurations of a continuous gridworld domain with 2D states and 2D actions, which we call the GridWorld with Slope .
In each conﬁguration, the agent navigates through four rooms connected by narrow hallways, by choosing a (bounded) increment along the coordinate directions.
A visual representation of the setting can be found in Figure 2a, where the shaded areas denote the initial state distribution and the arrows render a slope that favors or contrasts the agent’s movement.
The conﬁguration on the left has a south-facing slope, and thus it is called GridWorld with South slope (GWS).
Instead, the one on the right is called GridWorld with North slope (GWN) as it has a north-facing slope.
This class of environments is unbalanced (and thus interesting to our purpose) for two reasons:
First, the GWN conﬁguration is more challenging from a pure exploration standpoint, since the slope prevents the agent from easily reaching the two bottom rooms; secondly, the distribution over the class is also unbalanced, as it is p M =
[ Pr ( GWS ) , Pr ( GWN )]
= [0 . 8 , 0 . 2] .
In this setting, we compare MEMENTO against Neutral , which is a simpliﬁed version of MEMENTO with α = 1 , 1 to highlight the importance of percentile sensitivity w.r.t.
a na¨ıve approach to the multiple environments scenario.
The methods are evaluated in terms of the state visitation entropy E 1 M induced by the exploration strategies they learn.
In Figure 2, we compare the performance of the optimal exploration strategy obtained by running MEMENTO ( α = 0 .
and Neutral ( α = 1 ) for 150 epochs on the GridWorld with Slope class ( p M = [0 . 8 , 0 . 2] ).
We show that the ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ (a) GridWorld with Slope [Figure] [Figure]
E 1 M (b) p M =
[Figure]
MEMENTO0 .
E 1 M (c) p M = [1 , 0]
[Figure]
MEMENTO
[Figure]
[Figure]
1 − 0 . 0 .
E 1 M (d) p M =
[Figure]
(e) Neutral (left) and MEMENTO (right)
[Figure] [Figure] two methods achieve a very similar expected performance over the class (Figure 2b).
However, this expected performance is the result of a (weighted) average of very different contributions.
As anticipated, Neutral has a strong performance in GWS ( p M =
[1 , 0] , Figure 2c), which is close to the conﬁguration-speciﬁc optimum (dashed line), but it displays a bad showing in the adverse GWN ( p M = [0 , 1] ,
Figure 2d).
Conversely, MEMENTO learns a strategy that is much more robust to the conﬁguration, showing a similar performance in GWS and GWN, as the percentile sensitivity prioritizes the worst case during training.
To conﬁrm this conclusion, we look at the actual distribution that is generating the expected performance in Figure 2b.
In Figure 2e, we provide the empirical distribution of the trajectory-wise performance ( H τ ), considering a batch of 200 trajectories with p M = [0 . 8 , 0 . 2] .
It clearly shows that Neutral is heavytailed towards lower outcomes, whereas MEMENTO concentrates around the mean.
This suggests that with a conservative choice of α we can induce a good exploration outcome for every trajectory (and any conﬁguration), while without percentile sensitivity we cannot hedge against the risk of particularly bad outcomes.
However, let us point out that not all classes of environments would expose such an issue for a na¨ıve, risk-neutral approach (see Appendix D.4 for a counterexample), but it is fair to assume that this would arguably generalize to any setting where there is an imbalance (either in the hardness of the conﬁgurations, or in their sampling probability) in the class.
These are the settings we care about, as they require nuanced solutions (e.g., MEMENTO) for scenarios with multiple environments.
5.2. On the Value of the Percentile
In this section, we consider repeatedly training MEMENTO with different values of α in the GridWorld with Slope domain, and we compare the resulting exploration performance E 1 M as before.
In Figure 2f, we can see that the lower α we choose, the more we prioritize GWN (right bar for every α ) at the expense of GWS (left bar).
Note that this trend carries on with increasing α , ending in the values of Figures 2c, 2d.
The reason for this behavior is quite straightforward, the smaller is α , the larger is the share of trajectories from the adverse conﬁguration (GWN) ending up in the percentile at ﬁrst (Figure 2g), and thus the more
GWN affects the policy update (see the gradient in (6)).
5.3. RL with a General Exploration Strategy
To assess the beneﬁt of the pre-trained strategy, we design a family of MDPs M R , where M ∈ { GWS , GWN } , and R is any sparse reward function that gives 1 when the agent reaches the area nearby a random goal location and 0 otherwise.
On this family, we compare the performance achieved by TRPO (Schulman et al., 2015) with different initializations:
The exploration strategies learned (as in Section 5.1) by MEMENTO ( α = 0 .
and Neutral ( α = 1 ), or a randomly initialized policy (Random).
These three variations are evaluated in terms of their average return J M R , which is deﬁned in (1), over 50 randomly generated goal locations (Figure 3b).
As expected, the performance of TRPO with
Neutral is competitive in the GWS conﬁguration (Figure 3), but it falls sharply in the GWN conﬁguration, where it is not signiﬁcantly better than TRPO with Random.
Instead, the performance of TRPO with MEMENTO is strong on both GWS and GWN.
Despite the simplicity of the domain,
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure]
solving an RL problem in GWN with an adverse goal location is far-fetched for both a random initialization and a na¨ıve solution to the reward-free exploration over multiple environments.
5.4. Scaling to Larger Classes of Environments
In this section, we consider a class M composed of ten different conﬁgurations of the continuous gridworlds presented in Section 5.1 (including the GWN as the worst-case conﬁguration) which we call the MultiGrid domain.
As before, we compare MEMENTO ( α = 0 .
and Neutral ( α = 1 ) on the exploration performance E 1 M achieved by the optimal strategy, in this case considering a uniformly distributed p M .
While the average performance of Neutral is slightly higher across the class (Figure 4a left, left bar), MEMENTO still has a decisive advantage in the worst-case conﬁguration (Figure 4a left, right bar).
Just as in Section 5.3, this advantage transfer to RL, where we compare the average return J M R achieved by TRPO with MEMENTO, Neutral, and Random initializations over 50 random goal locations in the GWN conﬁguration (Figure 4a right).
5.5. Scaling to Increasing Dimensions
In this section, we consider a class M consisting of two
Ant environments, with 29D states and 8D actions.
In the ﬁrst, sampled with probability p M 1 = 0 .
8 , the Ant faces a wide descending staircase ( Ant Stairs Down ).
In the second, the Ant faces a narrow ascending staircase ( Ant Stairs
Up , sampled with probability p M 2 = 0 . 2 ), which is signiﬁcantly harder to explore than the former.
In the mold of the gridworlds in Section 5.1, these two conﬁgurations are speciﬁcally designed to create an imbalance in the class.
As in Section 5.1, we compare MEMENTO ( α = 0 . 2 ) against Neutral ( α = 1 ) on the exploration performance
E 1 M achieved after 500 epochs.
MEMENTO fares slightly better than Neutral both in the worst-case conﬁguration (Fig- ure 4b left, right bar) and, surprisingly, in the easier one (Figure 4b left, left bar).
Then, we design a set of incrementally challenging sparse-rewards RL tasks in the Ant
Stairs Up , which give reward 1 upon reaching a certain step of the staircase.
Also in this setting, TRPO with Memento initialization outperforms TRPO with Neutral and Random in terms of the average return J M R (Figure 4b right).
Note that these sparse-reward continuous control tasks are particularly arduous: TRPO with Neutral and Random barely learns anything, while even TRPO with MEMENTO does not handily reach the optimal average return (1) withinepochs.
5.6. Scaling to Visual Inputs
In this section, we consider a class M of two partiallyobservable MiniGrid (Chevalier-Boisvert et al., 2018) environments, in which the observation is a 147D image of the agent’s ﬁeld of view.
In Figure 4c, we provide a visualization of the domain: The easier conﬁguration (EasyG, left) is sampled with probability p M 1 = 0 .
8 , the adverse conﬁguration (AdvG, right) is sampled with probability p M 2 = 0 .
Two factors make the AdvG more challenging to explore, which are the presence of a door at the top-left of the grid, and reversing the effect of agent’s movements (e.g., the agent goes backward when it tries to go forward).
Whereas in all the previous experiments we estimated the entropy on the raw input features, visual inputs require a wiser choice of a metric.
As proposed in (Seo et al., 2021), we process the observations through a random encoder before computing the entropy estimate in (4), while keeping everything else as in Algorithm 1.
We run this slightly modiﬁed version of
MEMENTO ( α = 0 . 3 ) and Neutral ( α = 1 ) for 300 epochs.
Then, we compare TRPO with the learned initializations (as well as Random) on a series of sparse-reward tasks deﬁned upon the class.
As in previous settings, TRPO with MEMENTO results slightly worse than TRPO with Neutral in the easier conﬁguration (Figure 4d, left), but signiﬁcantly better in the worst-case (Figure 4d, right).
Notably, TRPO from scratch struggles to learn the tasks, especially in the
AdvG (Figure 4d, right).
5.7. Comparison with Meta-RL
In this section, we compare our approach against metatraining a policy with MAML (Finn et al., 2017) on the same GridWorld with Slope ( p M = [0 . 8 , 0 . 2] ) and MultiGrid (uniformly distributed p M ) domains that we have previously presented.
Especially, we consider two relevant baselines.
The ﬁrst is MAML+R, to which we pro- vide full access to the tasks (i.e., rewards) during metatraining.
Note that this gives MAML+R an edge over MEMENTO, which operates reward-free training.
The second is MAML+DIAYN (Gupta et al., 2018a), which operates unsupervised meta-training through an intrinsic reward function learned with DIAYN (Eysenbach et al., 2018).
As in previous sections, we consider the average return J M R achieved by TRPO initialized with the exploration strategy learned by MEMENTO or the meta-policy learned by
MAML+R and MAML+DIAYN.
TRPO with MEMENTO fares clearly better than TRPO with the meta-policies in all the conﬁgurations (Figures 5a, 5b).
Even if it works ﬁne in fast adaptation (see Appendix D.5), MAML struggles to encode the diversity of task distribution into a single meta-policy and to deal with the most adverse tasks in the long run.
Moreover, DIAYN does not speciﬁcally handle multiple environments, and it fails to cope with the larger
MultiGrid class.
6. Preliminary Theoretical Analysis of the Problem
In this section, we aim to theoretically analyze the problem in (2), and especially, what makes a class of multiple CMPs hard to explore with a unique strategy.
This has to be intended as a preliminary discussion on the problem, which could serve as a starting point for future works, rather than a thorough theoretical characterization.
First, it is worth introducing some additional notation.
Lipschitz Continuity Let X, Y be two metric sets with metric functions
d X , d Y .
We say a function f : X → Y is
L f -Lipschitz continuous if it holds for some constant L f d
[Equation]
[Equation]
, x ) ∈ X 2 , where the smallest L f is the Lipschitz constant and the Lipschitz semi-norm is
[Equation]
x (cid:9) .
When dealing with probability distributions we need to introduce a proper metric.
Let p, q be two probability measures, we will either consider the Wasserstein metric (Villani, 2008), deﬁned as d W 1 ( p, q ) = sup
[Equation]
X f [Equation] p − q )
[Equation] [Equation]
f
[Equation]
L ≤ 1 (cid:9) , or the Total Variation (TV) metric, deﬁned as
[Equation]
Intuitively, learning to explore a class M with a policy π is challenging when the state distributions induced by π in different M ∈ M are diverse.
The more diverse they are, the more their entropy can vary, and the harder is to get a π with a large entropy across the class.
To measure this diversity, we are interested in the supremum over the distances between the state distributions ( d M 1 π , . . .
, d M I π ) that a single policy π ∈ Π realizes over the class M .
We call this measure the diameter D M of the class M .
Since we have inﬁnitely many policies in Π , computing D M is particularly arduous.
However, we are able to provide an
[Figure] [Figure]
upper bound to D M deﬁned through a Wasserstein metric.
Assumption 1.
Let d S be a metric on S .
The class M is
L P π -Lipschitz continuous,
[Equation]
where P π ( s | s )
= (cid:82) A π ( a | s ) P ( s | s, a ) d a for P ∈ M , π ∈ Π , L P π is a constant L P π < 1 .
Theorem 6.1.
Let M be a class of CMPs satisfying Ass. 1.
Let d M π be the marginal state distribution over T steps induced by the policy π in M ∈ M .
We can upper bound the diameter D M as
[Equation]
Theorem 6.1 provides a measure to quantify the hardness of the exploration problem in a speciﬁc class of CMPs, and to possibly compare one class with another.
However, the value of D M might result, due to the supremum over Π , from a policy that is far away from the policies we actually deploy while learning, say ( π 0 , . .
, π ∗ E ) .
To get a ﬁner assessment of the hardness of M we face in practice, it is worth considering a policy-speciﬁc measure to track during the optimization.
We call this measure the π -diameter
D M ( π ) of the class M .
Theorem 6.2 provides an upper bound to D M ( π ) deﬁned through a convenient TV metric.
Theorem 6.2.
Let M be a class of CMPs, let π ∈ Π be a policy, and let d M π be the marginal state distribution over T steps induced by π in M ∈ M .
We can upper bound the π -diameter D M ( π ) as
[Equation]
The last missing piece we aim to derive is a result to relate the π -diameter D M ( π ) of the class M (Theorem 6.2) with the actual exploration objective, i.e., the entropy of the state visitations induced by the policy π over the environments in the class.
In the following theorem, we provide an upper bound to the entropy gap induced by the policy π within the class M .
Theorem 6.3.
Let M be a class of CMPs, let π ∈ Π be a policy and D M ( π ) the corresponding π -diameter of
M .
Let d M π be the marginal state distribution over T steps induced by π in M ∈ M , and let σ M ≤ σ M :=
inf s ∈S d M π ( s ) , ∀M ∈ M .
We can upper bound the entropy gap of the policy π within the model class M as sup
[Equation]
7. Conclusions
In this paper, we addressed the problem of learning to explore a class of multiple reward-free environments with a unique general strategy.
First, we formulated the problem within a tractable MSVE objective with percentile sensitivity.
Then, we presented a policy gradient algorithm to optimize this objective.
Finally, we provided an extensive experimental analysis to show its ability in learning to explore and the beneﬁts it brings to subsequent RL problems.
We believe that this paper motivates the importance of designing speciﬁc solutions to the relevant reward-free exploration problem over multiple environments.
As a ﬁnal note, it is worth mentioning an alternative problem formulation in which MEMENTO can be employed with beneﬁt.
Especially, we could replace the class of environments of our setting with a single CMP speciﬁed under uncertainty (Satia & Lave Jr, 1973), and deal with the robust reward-free exploration problem with little or no modiﬁcations to MEMENTO.
A. Related Work
Our work lies at the intersection of reward-free exploration, robust and risk-averse RL, and meta-RL.
The literature that relates the most with our work is the one pursuing a MSVE objective in reward-free settings.
Some methods (Hazan et al., 2019; Lee et al., 2019) focus on learning a mixture of policies that is collectively MSVE optimal, while other (Tarbouriech & Lazaric, 2019; Mutti & Restelli, 2020) casts the MSVE as a dual (or surrogate) linear program in tabular settings.
Successive works tackle MSVE at scale with nonparametric entropy estimation (Mutti et al., 2021; Liu & Abbeel, 2021a;b; Yarats et al., 2021; Seo et al., 2021), or introduce variations to the entropy objective, such as geometry-awareness (Guo et al., 2021) and R´enyi generalization (Zhang et al., 2020a).
To the best of our knowledge, all existing solutions are environment-speciﬁc and do not directly address multiple environments.
Previous work considered CVaR optimization in RL, either to learn a policy that is averse to the risk induced by the volatility of returns (Tamar et al., 2015; Chow & Ghavamzadeh, 2014) or by changes in the environment dynamics (e.g., Rajeswaran et al., 2016).
Here we account for a different source of risk, which is the one of running into a particularly unfavorable environment for the trained exploration strategy.
Finally, the two-stage learning setting we address has clear connections with the meta-RL problem setting (Finn et al., 2017), in which we would call meta-training the reward-free phase, and meta-testing the subsequent RL tasks.
While some methods target exploration in meta-RL (e.g., Xu et al., 2018; Gupta et al., 2018b; Zintgraf et al., 2019), they usually assume access to rewards during meta-training, with the notable exception of (Gupta et al., 2018a).
To the best of our knowledge, none of the existing works combine reward-free meta-training with a multiple-environments setting.
B. Proofs
Proposition 4.1.
The policy gradient of the exploration objective E α M ( π θ ) w.r.t.
θ is given by
[Equation]
Proof.
Let us start from expanding the exploration objective (2) to write
[Equation]
where p π θ , M is the probability density function (pdf) of the random variable H
τ when the policy π θ is deployed on the class of environments M , and the last equality comes from the deﬁnition of CVaR (Rockafellar et al., 2000).
Before computing the gradient of (7), we derive a preliminary result for later use, i.e.,
[Equation]
which follows directly from the Leibniz integral rule, noting that VaR α ( H τ ) depends on θ through the pdf of H τ .
We now take the gradient of (7) to get
[Equation]
where (9) follows from the Leibniz integral rule, and (10) is obtained from (9) through (8), which we can rearrange to write p π θ , M [Equation] VaR α
[Equation]
=
[Equation]
∇
[Equation]
p
[Equation]
h
[Equation]
d h .
All of the steps above are straightforward replications of the derivations by Tamar et al.
(Tamar et al., 2015), Proposition 1.
To conclude the proof we just have to compute the term ∇ θ p π θ , M ( h ) , which is speciﬁc to our setting.
Especially, we note that
[Equation]
where (11) and (12) are straightforward from the deﬁnitions in Section 2, and T is the set of feasible trajectories of length T .
Finally, the result follows by plugging (12) into (10), which gives
[Equation]
Theorem 6.1.
Let M be a class of CMPs satisfying Ass. 1.
Let d M π be the marginal state distribution over T steps induced by the policy π in M ∈ M .
We can upper bound the diameter D M as
[Equation]
Proof.
The proof follows techniques from (Pirotta et al., 2015).
Let us report a preliminary result which states that the function
h f ( s )
= (cid:82) A π ( a | s ) (cid:82) S P ( s | s, a ) d s d a has a Lipschitz constant equal to L P π
(Pirotta et al., 2015, Lemma 3)
[Equation]
where d S is a metric over S and P π ( s | s )
= (cid:82) A π ( a | s ) P ( s | s, a ) d a .
Then, we note that the marginal state distribution over
T steps d M π can be written as a sum of the contributions d M π,t related to any time step t ∈
[ T ] , which is d M π ( s
[Equation]
1 T
T
[Equation]
[Equation]
t =
d M π,t ( s ) .
Hence
[Equation]
we can look at the Wasserstein distance of the state distributions for some t ∈
[ T ] and M (cid:48) , M ∈ M .
We obtain
[Equation]
[Table]
[Equation]
where we plugged the common temporal relation d M π,t
[Equation]
s
[Equation]
[Equation]
d M π
[Equation]
[Equation] s ) π ( a | s ) P ( s
[Equation] | s, a ) d s d a into (15), we sum and subtract (cid:82) S (cid:82) A (cid:82)
S d M [Equation] π,t − 1 ( s ) π ( a | s ) P ( s | s, a ) d s d a d s to get (16), (17), and we apply the inequality in (13) to obtain (18) and then (19).
To get rid of the dependence to the state distributions d M (cid:48) π,t − 1 and d M π,t − 1 , we repeatedly unroll (19) to get
[Equation]
where we note that d W 1 ( d M (cid:48) π, 0 , d M π, 0 )
=
d W 1 ( D (cid:48) , D ) to derive (20), and we assume L P π < 1 (Assumption 1) to get (21) from (20).
As a side note, when the state and action spaces are discrete, a natural choice of a metric is d S ( s (cid:48) , s )
=
1 ( s (cid:48) (cid:54) = s ) and d A = 1 ( a (cid:48) (cid:54) = a ) , which results in the Wasserstein distance being equivalent to the total variation, the constant L P π = 1 , and (cid:80) tj =
L j P π = t .
More details over the Lipschitz constant L P π can be found in (Pirotta et al., 2015).
Finally, we can exploit the result in (21) to write
[Equation]
in which we use (14) to get (22).
The result follows from (23) by assuming the initial state distribution D to be shared across all the CMPs in M , and taking the supremum over P (cid:48) , P ∈ M .
Theorem 6.2.
Let M be a class of CMPs, let π ∈ Π be a policy, and let d M π be the marginal state distribution over T steps induced by π in M ∈ M .
We can upper bound the π -diameter D M ( π ) as
[Equation]
Proof.
The proof follows techniques from (Metelli et al., 2018a), especially Proposition 3.1.
Without loss of generality, we take M (cid:48) , M ∈ M .
With some overloading of notation [Equation] we will alternatively identify a CMP with the tuple M or its transition model P .
Let us start considering the TV between the marginal state distributions induced by π over M (cid:48) , M , we can write
[Equation]
where we use (14) to get (24).
Then, we provide an upper bound to each term of the ﬁnal sum in (25),
i
[Equation]
e.,
[Equation]
30) where we use the temporal relation d M π,t ( s (cid:48) )
= (cid:82) S (cid:82) A d M π,t − 1 ( s ) π ( a | s ) P ( s (cid:48) | s, a ) d s d a to get (26), in which we sum and subtract (cid:82)
S (cid:82)
A (cid:82)
S d M π,t − 1 ( s ) π ( a | s ) P ( s | s, a ) d s d
a d
s to obtain (27) and (28), and we repeatedly unroll (29) to write (30), noting that d TV ( d M (cid:48) π, 0 , d M π, 0 ) =
d TV ( D (cid:48) , D ) .
Finally [Equation] we can plug (30) in (25) to get
[Equation]
in which we have used (14) to obtain (32) from (31).
The ﬁnal result is straightforward from (32) by assuming the initial state distribution D to be shared across all the CMPs in M , and taking the supremum over P (cid:48) , P ∈ M .
Theorem 6.3.
Let M be a class of CMPs, let π ∈ Π be a policy and D M ( π ) the corresponding π -diameter of M .
Let d M π be the marginal state distribution over T steps induced by π in M ∈ M , and let σ M ≤ σ M :=
inf s ∈S d M π ( s ) , ∀M ∈ M .
We can upper bound the entropy gap of the policy π within the model class M as
[Equation]
Proof.
Let us expand the entropy gap of the policy π as sup
[Equation]
[Equation]
in which we sum and subtract (cid:82) S d M (cid:48)
π ( s ) log d M π ( s ) d s to obtain (35) from (34), log d M π ( s ) is upper bounded with log σ M to get (36)
[Equation]
and we use the reverse Pinsker’s inequality D KL ( p || q ) ≤
[Equation]
D TV
[Equation] p [Equation] q )) 2 / inf x ∈X q ( x ) (Csisz´ar & Talata, 2006, p. 1012 and Lemma 6.3) to obtain (11).
Finally, we get the result by upper bounding D TV ( d M (cid:48)
π , d M π ) with the π -diameter
D M ( π ) and σ M with σ M in (37).
C. Algorithm
In this section, we provide additional details about the proposed method (MEMENTO).
A full implementation of the algorithm can be found in the supplementary material.
C.1. The Beneﬁts of the Baseline
In this section, we provide theoretical and empirical motivations to corroborate the use of the baseline b = − VaR α
( H τ ) into the Monte Carlo policy gradient estimator (Section 4, Equation 6).
Thus, we compare the properties of two alternatives policy gradient estimator, with and without a baseline, i.e.,
[Equation]
where f τ i = (cid:80) T − 1 t =0 ∇ θ log π θ ( a t,τ i | s t,τ i ) .
The former ( (cid:98) ∇ θ E α M ) is known to be asymptotically unbiased (Tamar et al., 2015), but it is hampered by the estimation error of the VaR term to be subtracted to each (cid:98)
H τ i in ﬁnite sample regimes (Kolla et al., 2019).
The latter ( (cid:98) ∇ b θ E α M ) introduces some bias in the estimate, but it crucially avoids the estimation error of the
VaR term to be subtracted, as it cancels out with the baseline b .
The following proposition, along with related lemmas, assesses the critical number of samples ( n ∗ ) for which an upper bound to the bias of (cid:98) ∇ b θ E α M is lower to the estimation error of (cid:98) ∇ θ E α M .
Lemma C.1.
The expected bias of the policy gradient estimate (cid:98) ∇ b θ E α M ( π θ ) can be upper bounded as
[Equation]
where U is a constant such that f τ i ≤ U for all τ i .
Proof.
This Lemma can be easily derived by means of E
[Equation]
where (40) follows from (39) by noting that the estimator without the baseline term is unbiased (Tamar et al., 2015), and (41) is obtained by upper bounding f τ with U and noting that E M∼ M
τ
[Equation]
Lemma C.2 (VaR concentration bound from (L.A. et al., 2020)) .
Let X be a continuous random variable with a pdf f X for which there exist η, ∆ > 0 such that f X ( x ) >
η for all x ∈ (cid:2) VaR α ( X ) − ∆ 2 , VaR α ( X ) + ∆ 2 (cid:3) .
Then, for any (cid:15) > 0 we have
[Equation]
where n ∈ N is the number of samples employed to estimate (cid:91) VaR α ( X ) .
Proposition C.3.
Let (cid:98) ∇ θ E α M
π θ ) and (cid:98) ∇ b θ E α M ( π θ ) be policy gradient estimates with and without a baseline.
Let f H be the pdf of H τ , for which there exist η, ∆ > 0 such that f H
( H τ ) >
η for all H τ ∈ (cid:2) VaR α ( H τ ) − ∆2 , VaR α
( H τ ) + ∆2 (cid:3) .
The number of samples n ∗ for which the estimation error (cid:15) of (cid:98) ∇ θ E α M ( π θ ) is lower than the bias of (cid:98) ∇ b θ E α M ( π θ ) with at least probability δ ∈ (0 , 1) is given by
[Equation]
Proof.
The proof is straightforward by considering the estimation error (cid:15) of (cid:98) ∇ θ E α M ( π θ ) equal to the upper bound of the bias of (cid:98) ∇ b θ E α M ( π θ ) from Lemma C.1, i.e., (cid:15) =
U αb .
Then, we set δ = 2 exp (cid:0) − 2 n ∗ η 2 min
U 2 α 2 b 2 , ∆ 2 ) (cid:1) from Lemma C.2, which gives the result through simple calculations.
The Proposition C.3 proves that there is little incentive to choose the policy gradient estimator (cid:98) ∇ θ E α M when the number of trajectories is lower than n ∗ , as its estimation error would exceed the bias introduced by the alternative estimator (cid:98) ∇ b θ E α M .
Unfortunately, it is not easy to compute n ∗ in our setting, as we do not assume to know the distribution of H τ , but the requirement is arguably seldom matched in practice.
Moreover, we can empirically show that the baseline b = − VaR α
( H τ ) might beneﬁt the variance of the policy gradient estimation, at the expense of the additional bias which is anyway lower than the estimation error of (cid:98) ∇ θ E α M .
In Figure(left), we can see that the exploration performance E α M obtained by MEMENTO with and without the baseline is essentially the same in the illustrative GridWorld with Slope domain.
Whereas Figure 7 (right) suggests a slightly inferior variance for the policy gradient estimate employed by MEMENTO with the baseline.
[Figure]
C.2. Importance Weighted Entropy Estimation
As done in (Mutti et al., 2021), we build on the estimator in (4) to consider the case in which the target policy π θ (cid:48) differs from the sampling policy π θ .
The idea is to combine two successful policy-search methods.
The ﬁrst one is POIS (Metelli et al., 2018b), to perform the optimization ofﬂine via importance sampling, allowing for an efﬁcient exploitation of the samples collected with previous policies.
We thus adopt an Importance-Weighted (IW) entropy estimator (Ajgl & ˇSimandl, 2011) of the form (cid:98) H
[Equation]
(42) where ln k − Ψ( k ) is a bias correction term in which Ψ is the Digamma function,
N k
i is the set of indices of the k-NN of s t,τ i , and w j are the normalized importance weights of samples s j,τ i .
To compute these importance weights we consider a dataset D = { s t,τ i } T − 1 t =0 by looking each state encountered in a trajectory as an unweighted particle.
Then, we expand it as
D τ i = { ( τ i,t , s t ) } T − 1 t =0 , where τ i,t = ( s 0 ,τ i , . . .
, s t,τ i ) is the portion of the trajectory that leads to state s t,τ i .
This allows to associate each particle s t,τ i to its importance weight (cid:98) w t and normalized importance weight w t for any pair of target ( π θ (cid:48) )
Output
1: for convergence do 2: do 3: sample an environment M i ∼ p M 4: for j
B do 5: sample a trajectory τ j ∼ p π θ , M i of length T 6: end for 7: end for 8: initialize dataset D = ∅ , off-policy step h =
and θ h
= θ 9: while (cid:98) D KL ( π θ
|| π θ h ) ≤ δ do 10: for j =
, B do 11: estimate H τ j with (42) 12: append (cid:98) H τ j to D 13: end for 14: sort D and split it in D α and D 1 − α 15: compute a gradient step θ h +
=
θ h + β (cid:98) ∇ θ h E
α M
( π θ h ) 16:
h ← h +17: end while 18: θ ← θ h 19: end for and sampling ( π θ ) policies:
[Equation]
The estimator in (42) is then optimized via gradient ascent.
The second policy-search method used during the optimization is TRPO (Schulman et al., 2015), to perform subsequent optimizations within a trust-region around the current policy.
The trust-region constraint is obtained by imposing
[Equation]
where (cid:98) D KL ( π θ (cid:48)
||
π θ ) is a non-parametric IW k-NN estimate of the Kullback-Leibler (KL) divergence (Ajgl & ˇSimandl, 2011).
Its value is computed as in (Mutti et al., 2021), by considering the entire batch of trajectories collected to execute the off-policy optimization steps as a single trajectory.
C.3. Algorithmic Details of MEMENTO
In this section, we provide an extended pseudocode (Algorithm 2) of MEMENTO, along with some additional comments.
Given a probability distribution p M , the algorithm operates by iteratively sampling an environment M i ∈ M drawn according to p M and then sampling B trajectories of length T from it using π θ , where B is the dimension of each mini-batch.
Then, the estimate of the entropy of each mini-batch (cid:98) H τ j is computed by means of the estimator in (42) and appended to the dataset D .
Once obtained the ﬁnal dataset D , we can straightforwardly derive a risk-sensitive policy update by just subsampling from it, so that to keep only the realizations below the α -percentile.
This can be easily done by sorting D in ascending order and considering only the αN ﬁrst mini-batches.
Then, we can compute the gradient as follows:
[Equation]
The operations carried out once all the trajectories have been sampled are executed in a fully off-policy manner, in which we repeat the same steps until the trust-region boundary is reached or until the number of off-policy iterations exceeds a speciﬁed limit.
The reason why we introduce an additional parameter B , instead of considering one trajectory at a time, is due to the fact that a signiﬁcant amount of samples (see the parameters in Table 1) is needed to obtain a reliable estimate of the entropy, noting that the entropy estimator is only asymptotically unbiased.
D. Experiments
In this section, we report an extensive description of the conducted experiments, with the corresponding hyperparameter values and some additional plots and experiments.
D.1. Environments
We use three different environments in our experiments.
The ﬁrst one is a custom implementation of a gridworld, coded from scratch.
The second one is an adapted version of the rllab Ant-Maze environment (Duan et al., 2016).
D.1.1. G RID W ORLD WITH S LOPE
In GridWorld with Slope (2D states, 2D actions), the agent can move inside a map composed of four rooms connected by four narrow hallways, by choosing at each step how much to move on the x and y axes.
The side of the environment measures 2 units and the maximum viable space of the agent at each step is 0 .
Thus, the agent needs around 10 steps to go from one side to the other on a straight line.
When the agent collides with the external borders or with the internal walls, it is re-positioned according to a custom function.
This is done not only to make the interaction more realistic, but also to limit the possibility to have a negative inﬁnite entropy resulting from the k-NN computation, which can occur when the samples are too close and the value of the parameter k is not high enough.
This precaution is particularly useful in our scenario, due to the presence of a slope, and especially in the adversarial conﬁguration GWN, because of the initial position of the agent, which is sampled in a small square in the top-right corner.
It is easy to see that in the ﬁrst epochs in the GWN environment, the agent would repeatedly collide with the top-border, leading in general to a much more lower entropy w.r.t.
to GWS.
The slope is applied only in the upper half of the environment, since we found this to be a good trade-off between the intention of maintaining a difference in terms of risk among the two conﬁgurations and the overall complexity of the exploration.
Indeed, we noted that by applying the slope to the whole GridWorld, the advantage in terms of exploration entailed by the risk-averse approach is even higher, but it struggles to explore the bottom states of the environment with a reasonable number of samples.
The slope is computed as s ∼ N ( ∆ max 2 , ∆ max 20 ) , where ∆ max = 0 .
2 is the maximum step that the agent can perform.
D.1.2. M ULTI G RID
In MultiGrid , everything works as in GridWorld with Slope , but we indeed have 10 conﬁgurations.
These environments differ for both the shape and the type of slope to which they are subject to.
The adversarial conﬁguration is still GWN, but the slope is computed as s ∼ N ( ∆ max 2 . 6 , ∆ max 20 ) , where ∆ max = 0 .
The other 9 gridworlds have instead a different arrangement of the walls (see the heatmaps in Figure 10) and the slope, computed as s ∼ N ( ∆ max 3 . 2 , ∆ max 20 ) with ∆ max = 0 . 2 , is applied over the entire environment.
Two conﬁgurations are subject to south-facing slope, three to east-facing slope, one to south-east-facing slope and three to no slope at all.
D.1.3. A NT S TAIRS
We adopt the Ant-Maze environment (29D states, 8D actions) of rllab (Duan et al., 2016) and we exploit its malleability to build two custom conﬁgurations which could ﬁt our purposes.
The adverse conﬁguration consists of a narrow ascending staircase ( Ant Stairs Up ) made up of an initial square (the initial position of the Ant), followed by three blocks of increasing height.
The simpler conﬁguration consists of a wide descending staircase ( Ant Stairs Down ), made up of 3 × 3 blocks of decreasing height and a ﬁnal 1 × 3 ﬂat area.
Each block has a side length slightly greater than the Ant size.
A visual representation of such settings is provided in Figure 11.
During the learning to explore phase, E α M is maximized over the x,y spatial coordinates of the ant’s torso.
D.1.4. M INI G RID
We use the MiniGrid suite (Chevalier-Boisvert et al., 2018), which consists of a set of fast and light-weighted gridworld environments.
The environments are partially observable, with the dimension of the agent’s ﬁeld of view having size 7 × 7 × 3 .
Both the observation space S and the action space
A are discrete, and in each tile of the environment there can be only one object at the same time.
The set of objects is O = { wall, floor, lava, door, key, ball, box, goal } .
The agent can move inside the grid and interact with these objects according to their properties.
In particular, the actions comprise turning left, turning right, moving forward, picking up an object, dropping an object and toggling, i.e., interacting with the objects (e.g., to open a door).
We exploit the suite’s malleability to build two custom environments.
The simpler one has a size of 18 × 18 , and it simply contains some sparse walls.
The adverse conﬁguration is smaller, 10 × 10 , and is characterized by the presence of a door at the top of a narrow hallway.
The door is closed but not locked, meaning that the agent can open it without using a key.
Moreover, we modify the movement of the agent so that the direction is given by the bottom of the triangle instead of the top.
The intuition is that by doing this we are essentially changing the shape of the agent, hence causing the policy to struggle in the exploration.
As regards the training procedure, everything remains the same, except for two differences.
The ﬁrst difference is that the k -NN computation is performed on the representation space generated by a ﬁxed random encoder.
Note that this random encoder is not part of the policy.
It is randomly initialized and not updated during the training in order to produce a more stable entropy estimate.
In addition, before computing the distances, we apply to its output a random Gaussian noise (cid:15) ∼ N (0 . 001 , 0 . 001) truncated in [0 , 0 . 001] .
We do this to avoid the aliasing problem, which occurs when we have many samples (more than k ) in the same position, thus having zero distance and producing a negative inﬁnite entropy estimate.
The homogeneity of the MiniGrid environments in terms of features make this problem more frequent.
The second difference is the addition of a bootstrapping procedure for the easy conﬁguration, meaning that we use only a subset of the mini-batches of the easy conﬁguration to update the policy.
Especially, we randomly sample a number of mini-batches that is equal to the dimension of the D α dataset so that Neutral uses the same number of samples of MEMENTO.
The reason why we avail this method is to avoid a clear advantage for Neutral in learning effective representations, since it usually access more samples than MEMENTO.
Note that it is not a stretch, since we are essentially balancing the information available to the two algorithms.
D.2. Class of Policies
In all the experiments but one the policy is a Gaussian distribution with diagonal covariance matrix.
It takes as input the environment state features and outputs an action vector a ∼ N ( µ, σ 2 ) .
The mean µ is state-dependent and is the downstream output of a densely connected neural network.
The standard deviation is state-independent and it is represented by a separated trainable vector.
The dimension of µ , σ , and a vectors is equal to the action-space dimension of the environment.
The only experiment with a different policy is the MiniGrid one, for which we adopt the architecture recently proposed by (Seo et al., 2021).
Thus, we use a random encoder made up of 3 convolutional layers with kernel 2, stride 1, and padding 0, each activated by a ReLU function, and with 16, 32 and 64 ﬁlters respectively.
The ﬁrst ReLU is followed by a 2D max pooling layer with kernel 2.
The output of the encoder is a 64 dimensional tensor, which is then fed to a feed-forward neural network with two fully-connected layers with hidden dimension 64 and a Tanh activation function.
D.3. Hyperparameter Values D.3.1. L EARNING TO E XPLORE
In Table 1, we report the parameters of MEMENTO and Neutral that are used in the experiments described in Section 5.1,
Section 5.2, Section 5.4 and Section 5.5.
[Table]
D.3.2. R EINFORCEMENT L EARNING
In Table
[Table]
we report the TRPO parameters that are used in the experiments described in Section 5.3, Section 5.4, Section 5.
and Section 5.
[Table]
[Table]
D.3.3. M ETA -RL
In Table 3 and Table
[Table]
we report the MAML and DIAYN parameters that are used in the experiments described in
Section 5. [Table] in order to meta-train a policy on the GridWorld with Slope and MultiGrid domains. For MAML
[Table]
we adopted the codebase at https://github.com/tristandeleu/pytorch-maml-rl [Table] while for DIAYN, we used the original implementation
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
D.4.
Counterexample: When Percentile Sensitivity Does Not Matter
[Figure]
In this section [Figure] we provide a convenient example to conﬁrm the fact that there are classes of environments in which we would not need any particularly smart solution for the multiple environments problem, beyond a na¨ıve, risk-neutral approach.
We consider two GridWorld environments that differ for the shape of the traversable area, sampled according to p M = [0 . 8 , 0 . 2] , and we run MEMENTO with α = 0 .
35 and Neutral ( α = 1 )
, obtaining the two corresponding exploration policies.
In
Figure 8 we show the performance (measured by E 1 M ) obtained by executing those policies on each setting.
Clearly, regardless of what conﬁguration we consider, there is no advantage deriving from the use of a risk-averse approach as
MEMENTO, meaning that the class of environments M is balanced in terms of hardness of exploration.
D.5. Further Details on Meta-RL Experiments
[Figure]
In this section, we provide additional details on the experiments of Section 5.7.
Especially, we show that MAML does perform well on its own objective, which is to learn a fast-adapting policy during meta-training (Figure 9a).
Instead, in
Figure 9b we highlight the performance measure of DIAYN (Eysenbach et al., 2018).
In particular, the more log q φ ( s | z ) grows with the learning epochs, the better is the intrinsic reward we feed to MAML+DIAYN.
Clearly, DIAYN struggles to deal with the larger MultiGrid class of environments, which explains the inferior performance of MAML+DIAYN in this domain.
D.6. Additional Visualizations
In this section, we provide some additional visualizations, which are useful to better understand some of the domains used in the experiments of Section 5.
In Figure 10 we report the state-visitation frequencies achieved by MEMENTO (Figure 10a) and Neutral (Figure 10b) in each conﬁguration of the MultiGrid domain.
Clearly, MEMENTO manages to obtain a better exploration in the adversarial conﬁguration w.r.t.
Neutral, especially in the bottom part of the environment, which is indeed the most difﬁcult part to visit.
On the other environments, the performance is overall comparable.
In Figure 11 we show a render of the Ant Stairs domain, illustrating both the environments used in the experiments of Section 5.5.
Note that the front walls are hidden to allow for a better visualization.
E. Future Directions
First, it is worth mentioning an alternative setting in which MEMENTO can be employed with beneﬁt (with little or no modiﬁcations).
This is the the robust reward-free exploration problem, in which we just have to replace the class of environments with a single CMP speciﬁed under uncertainty (Satia & Lave Jr, 1973).
Secondly, in this work we focused on a speciﬁc solution for an essentially multi-objective problem, by establishing a preference over the environments through the
CVaR objective.
Instead, a future direction could pursue learning a direct approximation of the Pareto frontier (Parisi et al., 2016) of the exploration strategies over multiple environments.
Another promising direction is to assume some control over the class distribution during the learning to explore process, either by an external supervisor or by the agent itself (Metelli et al., 2018a).
Lastly, future work may establish regret guarantees for the reward-free exploration problem over multiple environments, in a similar ﬂavor to the reward-free RL problem in a single environment (Jin et al., 2020).
[Figure] [Figure]