Learning General World Models in a Handful of Reward-Free Deployments
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Building generally capable agents is a grand challenge for deep reinforcementlearning (RL).
To approach this challenge practically, we outline two key desiderata:1) to facilitate generalization, exploration should be task agnostic; 2) to facilitatescalability, exploration policies should collect large quantities of data withoutcostly centralized retraining.
Combining these two properties, we introduce thereward-free deployment efﬁciency setting, a new paradigm for RL research.
Wethen present CASCADE , a novel approach for self-supervised exploration in thisnew setting.
CASCADE seeks to learn a world model by collecting data with apopulation of agents, using an information theoretic objective inspired by Bayesian
Active Learning.
CASCADE achieves this by speciﬁcally maximizing the diversityof trajectories sampled by the population through a novel cascading objective. Weshow a tabular version of CASCADE theoretically improves upon na¨ıve approachesthat do not account for population diversity.
We then demonstrate that CASCADEcollects diverse task-agnostic datasets and learns agents that generalize zero-shotto novel, unseen downstream tasks on Atari, MiniGrid and the DM Control Suite.
Introduction
Reinforcement learning (RL, [103]) has achieved a number of impressive feats over the past decade,with successes in games [68, 13, 98, 112] robotics [76] and the emergence of real world applications[11, 24].
Indeed, now that RL has successfully mastered a host of individual tasks, the communityhas begun to focus on the grand challenge of building generally capable agents [89, 107, 67, 5].
In this work, we take steps towards building generalist agents at scale , where we outline two keydesiderata.
First, in order for agents to become generalists that can adapt to novel tasks, we eschewthe notion of restricting agent learning to task-speciﬁc reward functions.
Instead, we believe it is vitalto focus on the reward-free problem setting [82, 27], whereby agents must discover novel skills andbehaviors for themselves.
1 Consider the problem of learning to control robotic arms: we may alreadyhave some expert ofﬂine data to learn from, which will likely cover only a subset of the entire rangeof possible behaviors.
Therefore, to learn additional general skills, it is imperative to further collectnovel and diverse data without a pre-speciﬁed reward function.
Second, to ensure scalability, we should have access to a large ﬂeet of robots that we can deployto gather this data for a large number of timesteps [59], without costly and lengthy centralizedretraining during this crucial phase [66].
This has recently been referred to as deployment efﬁciency ,falling between the typical online/ofﬂine RL dichotomy (see Figure 1).
Limiting deployments notonly reduces the overhead in retraining exploration policies, but also limits the potential costs andrisks present when deploying new policies [106, 52], an important consideration in many real world
(cid:38)(cid:82)(cid:79)(cid:79)(cid:72)(cid:70)(cid:87)(cid:3)(cid:39)(cid:68)(cid:87)(cid:68)(cid:86)(cid:72)(cid:87) (cid:68) (cid:86) (cid:55)(cid:85)(cid:68)(cid:76)(cid:81)(cid:3)(cid:40)(cid:91)(cid:83)(cid:79)(cid:82)(cid:85)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:51)(cid:82)(cid:79)(cid:76)(cid:70)(cid:92) (cid:171)
[Table]
(cid:19) (cid:20)(cid:16)(cid:21)(cid:19) (cid:20)(cid:19)(cid:19)(cid:86)(cid:16)(cid:20)(cid:48)(cid:86) (cid:68) (cid:86) (cid:68) (cid:86) (cid:171) settings, such as robotics [47], education [64] and healthcare [28].
Combining these two desiderata,we introduce the reward-free deployment efﬁciency setting, a new paradigm for deep RL research.
To maximize deployment efﬁciency we now must train exploration policies capable of collecting largequantities of interesting and, crucially, useful data.
To tackle this new problem we use world models[33], which allow agents to plan and/or train policies without interacting with the true environment.
World models have already been shown to be highly effective for deployment efﬁciency [66], ofﬂine
RL [117, 48, 8, 87] and self-supervised exploration [97, 95].
Furthermore, world models offer thepotential for increasing agent generalization capabilities [7, 36, 67, 16, 8, 20, 10, 57], one of thefrontiers of RL research [79, 50].
However, since existing self-supervised methods for learning worldmodels are designed to collect only a few transitions with a single exploration policy, they likelyproduce a homogenous dataset when deployed at scale, which does not optimally improve the model.
[Figure] gain.
If we train a population of agents in-
dependently, at deployment time they will all follow the trajectory to #1, producing a homogenous dataset.
However, if we consider the diversity of the data then we will produce agents that reach both #1 and #2.
Instead, drawing analogies from Bayesian Active Learning[41, 51], we introduce a new information theoretic objective that maximizes the information gain from an entiredataset collected by a population of exploration agents (see
Figure 2).
We call our method C oordinated A ctive S ample
C ollection vi a D iverse E xplorers or CASCADE .
We providetheoretical justiﬁcation for CASCADE , which emphasizesthe importance of collecting data with diverse agents.
Inaddition, we provide a rigorous empirical evaluation acrossthree challenging domains that shows CASCADE can discover a rich dataset from a handful of deployments.
Wesee that CASCADE produces general exploration strategiesthat are equally adept at both “deep” exploration problemsand diverse behavior discovery.
This makes it possible totrain agents capable of zero-shot transfer when rewards areprovided at test time in a variety of different settings.
To summarize, our contributions are as follows: 1) We introduce a novel problem setting, RewardFree Deployment Efﬁciency , designed to train generalist agents in a scalable fashion; 2) We propose
CASCADE , a theoretically motivated model-based RL agent designed to gather diverse, highlyinformative data, inspired by Bayesian Active Learning; 3)
We demonstrate that CASCADE is capableof improved zero-shot transfer , and provide analysis that shows CASCADE theoretically improvessample efﬁciency over other na¨ıve methods that do not ensure sample diversity.
Problem Statement
Reinforcement learning (RL) considers training an agent to solve a Markov Decision Process (MDP),represented as a tuple M = {S , A , P, R, ⇢ ,  } , where s 2 S and a 2 A are the set of states and 70 actions respectively, P ( s| s, a ) is a probability distribution over next states given a previous stateand action,
R ( s, a, s 0 ) !
r is a reward function mapping a transition to a scalar reward, ⇢ is aninitial state distribution and  is a discount factor.
A policy ⇡ acting in the environment produces a trajectory ⌧ = { s 1 , a 1 , . . .
, s H , a H } for an episode with horizon H .
Since actions in the trajectoryare sampled from a policy, we can then deﬁne the RL problem as ﬁnding a policy ⇡ that maximizesexpected returns in the environment, i.e. ⇡ ?
=
arg max ⇡ E ⌧ ⇠ ⇡ [ R ( ⌧ )] .
We seek to learn policies that can transfer to any MDP within a family of MDPs.
This can beformalized as a Contextual MDP [50], where observations, dynamics and rewards can vary given acontext.
In this paper we consider settings where only the reward varies, thus, if the test-time contextis unknown at training time we must collect data that sufﬁciently covers the space of possible rewardfunctions .
Finally, to facilitate scalability, we operate in the deployment efﬁcient paradigm [66],whereby policy learning and exploration are completely separate, and during a given deployment ,we gather a large quantity of data without further policy retraining (c.f. online approaches like DER[110], which take multiple gradient steps per exploration timestep in the real environment).
Takentogether, we consider the reward-free deployment efﬁciency problem.
This differs from previous workas follows: 1) unlike previous deployment efﬁciency work, our exploration is task agnostic; 2) unlikeprevious reward-free RL work, we cannot update our exploration policy ⇡ EXP during deployment.
Thus, the focus of our work is on how to train ⇡ EXP ofﬂine such that it gathers heterogeneous andinformative data which facilitate zero-shot transfer to unknown tasks.
In this paper we make use of model-based RL (MBRL), where the goal is to learn a model of theenvironment (or world model [94]) and then use it to subsequently train policies to solve downstreamtasks.
To do this, the world model needs to approximate both P and R .
Typically, the model willbe a neural network, parameterized by , hence we denote the approximate dynamics and rewardfunctions as P and R , which produces a new “imaginary” MDP, M = ( S , A , P , R , ⇢ ) .
Wefocus on Dyna-style MBRL [102], whereby we train a policy ( ⇡ ✓ parameterized by ✓ ) with model-free RL solely using “imagined” transitions inside M .
Furthermore, we can train the policy on a single GPU with parallelized rollouts since the simulator is a neural network [53].
The general form of allmethods in this paper is shown in Algorithm 1, with the key difference being step 5: We aim to update⇡ EXP in the new imaginary MDP M such that it continues to collect a large, diverse quantity ofreward-free data.
Note that ⇡ EXP need not be a single policy, but could also refer to a collection ofpolicies that we can deploy (either in parallel or in series), such that ⇡ 2 ⇡ EXP .
We focus on learning world models from high dimensional sensory inputs such as pixels [33, 75, 46],where at each timestep we are given access to an observation o t rather than a state s t .
A seriesof recent works have shown tremendous success by mapping the observation to a compact latentstate z t [38, 37, 39].
In this paper we will make use of the model from DreamerV2 [39], which hasbeen shown to produce highly effective policies in a variety of high dimensional environments.
Theprimary component of DreamerV2 is a Recurrent State Space Model (RSSM) that uses a learnedlatent state to predict the image reconstruction, reward r t and discount factor  t .
Aside from thereward head, all components of the model are trained jointly, in similar fashion to variational encoders(VAEs, [49, 91]).
For zero-shot evaluation, we follow [95] and only train the reward head at test timewhen provided with labels for our pre-collected data, which is then used to train a behavior policy.
Thus, it is critical that our dataset is sufﬁciently diverse to enable learning novel, unseen behaviors.
Coordinated Active Sample Collection
The aim of this work is to train a population of B exploration policies { ⇡ ( i )EXP } Bi =1 such that theycollectively acquire data which maximally improves the accuracy of a world model .
To achieve this,we take inspiration from the information theoretic approach in Plan2Explore [95], but crucially focuson maximizing information gain over entire trajectories rather than per state-action, and hence drop the conditional dependence on state and action (see App. C.1 for why this distinction is important):
[Equation]
where d ⇡ M is the distribution of states visited by the policy ⇡ in the imaginary MDP M .
This objective produces ⇡ EXP , a policy whose visitation distribution has a high entropy when computed overmodel samples, but has low entropy for each individual MDP model (i.e., high epistemic/reducableuncertainty).
To make this objective more general, we represent the trajectory data collected by apolicy with a “summary” embedding space [78, 71].
Let  :  !
⌦ be a summary function mappingtrajectories into this embedding space.
P  ⇡ [ M ] denotes the embedding distribution generated bypolicy ⇡ in imaginary MDP M .
We can now write the objective from Eq.
1 as follows:
[Equation]
This more general framework allows us to consider multiple representations for trajectories, in asimilar fashion to behavioral characterizations in Quality Diversity algorithms [84].
For the restof this discussion, we will use the ﬁnal state embedding as our summary representation, whereby ( ⌧ ) =
h H , since in the case of the RSSM, the ﬁnal latent is a compact representation of the entiretrajectory collected by the policy, analogous to the ﬁnal hidden state in an RNN [93, 101].
3.1 A Cascading Objective with Diverse Explorers
We now consider a population-based version of
Equation.
2, using
B agents:
[Equation]
where ⇡ B = ⇡ (1) , · · · , ⇡ ( B ) and
Q
Bi =1 P  ⇡ ( i ) [ M ] is the product measure of the policies’ embedding 134 distributions in
M .
By deﬁnition, the conditional entropy factorizes as:
[Equation]
It is now possible to show that maximum information gain is achieved with an diverse set of agents:
Lemma 1.
When all models M in the support of the model posterior are deterministic and tabular,and the space of policies ⇧ consists only of deterministic policies, there always exists a solution{ ⇡ ( i )EXP } Bi =1 satisfying ⇡ ( i )EXP 6 = ⇡ ( j ) EXP 8 i 6 = j .
Moreover, there exists a family of tabular MDP models,such that the maximum cannot be achieved by setting ⇡ EXP ( i )
=
⇡ for a ﬁxed ⇡ .
The proof of Lemma 1 is in Appendix C.1.1.
Since the mutual information objective 3 is submodular, a greedy algorithm yields a (1  1 e ) approximation of the optimum (where e is Euler’s number) [72].
Leveraging this insight, let us assume thatwe already have a set of policies ⇡ (1) , · · · , ⇡ ( i  1) ; we then select the next policy ⇡ ( i ) based on thefollowing greedy objective:
[Equation]
{+) = ⇡ ( j ) 8 j  i  
+}
A  H@ i
Y j =
P  ⇡ ( j ) [ M ]
M , ˜ ⇡ ( j ) =
{+⇡ ( j ) 8 j  i  
+}
A
Which can be factorized in similar fashion to Equation 4 (See Appendix C).
3.2 A Tractable Objective for Deep RL
Inspired by [95], we make a couple of approximations to derive a tractable objective for { ⇡ EXP } Bi =1in the deep RL setting.
First, we assume that the embedding distributions are Gaussian with means that depend on the policies and sampled worlds, and variances that depend on the worlds,i.e. P  ⇡ [ M = w ] = N ( µ ( w, ⇡ ) , ⌃ ( w )) .
In this case, H ( P  ⇡ [ M ] |M = w ) = ⇢ ( w ) , and
Eq.
5 reduces to solving ⇡ ( i ) =
arg max ˜ ⇡
( i )
⇧ H ⇣Q ij =1 P  ˜ ⇡ ( j ) [ M ]  ˜ ⇡ ( j ) =
⇡ ( j ) 8 j  i ⌘ fora policy that maximizes the resulting joint entropy of the embedding distribution when added tothe policy population.
This produces the following surrogate objective, maximizing a quadraticcascading disagreement:
[Equation]
5 where D ( i  1) is a dataset of imagined trajectories sampled from policies ⇡ (1) , · · · , ⇡ ( i  1) in themodel, and ImagDiv is short for Imagined Diversity.
Finally, following [95, 9], we also add a perstate model-disagreement/information gain component to each policy’s reward to encourage a richerlandscape for data acquisition: InfoGain( ⇡ ) =
E ⌧ ⇠ P ⇡ [ M ] hP
[Equation]
s
[Equation]  ( s, a [Equation] i where  ( · , · ) is thevariance across the ensemble latent state predictions (for details see App. B).
Taken together, theseobjectives form our approach, which we call C oordinated A ctive S ample C ollection vi a D iverse
E xplorers or CASCADE .
CASCADE trains agents to optimize: 1) a diversity term ( ImagDiv ) thattakes into account the behaviors of the other agents in the population; 2) an information gain term( InfoGain ) that encourages an individual agent to sample states that maximally improve the model:
[Equation]
i
[Equation]
where  is a weighting hyperparameter that trades off whether we favor individual model informationgain or population diversity.
Finally, we train the B agents in parallel using (policy) gradient descentover ✓ , which makes it possible to achieve the same wall clock time as training a single agent [22].
3.3 Theoretical Motivation
We now seek to provide a tabular analogue to CASCADE which provides a theoretical grounding forour approach.
In App.
D we outline the pseudo-code of CASCADE - TS , a greedy Thompson Samplingalgorithm [4] that produces the i -th exploration policy in a tabular enviornment using “imaginary”data gathered by running policies ⇡ (1) , · · · , ⇡ ( i  1) in the model .
We can then show the following:
Lemma 2.
For the class of Binary Tree MDPs, the CASCADE - TS algorithm satisﬁes,
[Equation]
where T ( ✏ , · ) are the expected number of rounds of deploying a population of B policies necessaryto learn the true model up to ✏ accuracy; SinglePolicyBatch plays a ﬁxed policy B times in eachround; Sequential does not have a population, and instead interleaves updates and executions of asingle policy B times within each round.
The proof is in App.
D. We see that CASCADE - TS achieves provable efﬁciency gains over a na¨ıvesampling approach that does not ensure diversity in its deployed agents.
This provable gain isachieved by discouraging the i -th policy away from imaginary state-action pairs sampled by theprevious i  1 policies by using imaginary counts.
Now returning to CASCADE , we can see the importance of leveraging the imaginary data gathered inthe model by the previous i  1 policies when training the i -th exploration policy.
Concretely, encouraging policy ⇡ ( i ) to induce high disagreement with the embeddings produced by { ⇡ (1) , · · · , ⇡ ( i  1) }is analogous to the imaginary count bonus term of CASCADE - TS in Line 10 of Alg. 2, which avoidsredundant data collection during deployment.
Experiments
In our experiments we test whether CASCADE facilitates the learning of generalist agents by evaluatingtheir zero-shot transfer to unseen tasks, given a limited number of reward-free deployments.
In allexperiments, agents cannot train online in the environment, and instead execute a ﬁxed exploration policy to gather new data during each deployment.
We test against three baselines.
The ﬁrst is randomexploration, which is used in the majority of RL publications.
Our next baseline is Plan2Explore[95] ( P 2 E ), which trains a single exploration policy that optimizes for expected information gaininside a world model.
We then include a population-based version of P 2 E , which we call Population
Plan2Explore ( PP 2 E ).
PP 2 E trains a set of randomly initialized agents that independently seek tomaximize expected information gain.
All methods make use of a DreamerV2 world model [39] anduse the same hyperparameters for model and agent training (more details in App. B).
We test all methods in two separate settings: Firstly Exploring Worlds (Sec 4.1), where we seekto collect data in challenging exploration environments.
We test how well the explorers cover thestate space and discover sparse rewards (also known as “deep exploration” [77, 74]).
Secondly, weconsider Exploring Behaviors (Sec 4.2) where we consider collecting data in continuous controlenvironments.
In both settings we test zero-shot transfer as follows: 1) we provide reward labels tothe model; 2) we train a separate reward head; 3) we train a task speciﬁc behavior policy and test it inthe environment.
This tests whether our model is general enough to facilitate learning downstreampolicies for previously unknown tasks [95, 54, 115].4.
1 Exploring Worlds
In many cases we may not know a priori which states in an environment are highly rewarding.
Thus,if we wish to use our world model to subsequently train agents to solve tasks, it is crucial that weaccurately model as much of state space as possible.
Here we consider exploring worlds in threedifferent environments: MiniGrid [19], Atari [12] and Crafter [36].
In both settings, we start with arandom deployment before beginning reward-free exploration.
We begin with MiniGrid, a set of sparse-reward, partially observable navigation environmentscommonly used to test state-of-the-art exploration methods [88, 17, 30, 120].
MiniGrid environmentsare procedurally generated, which provides an additional generalization challenge [45, 92].
Weconsider the canonical FourRooms and the more challenging MultiRoom .
In each case we are limitedto 5 deployments of 200k transitions, for a total of 1M environment interactions.
2 We score thereward-free exploration in two ways: 1) state coverage, where we evaluate the percentage of the statesvisited by the exploration policies after each deployment on the same set held out test levels rewarding episodes, where we show the number of rewarding episodes collected during the trainingprocess.
Note that these two objectives are not useful in isolation, as solved levels may only include anarrow set of observations, while state coverage alone could be maximized without performing deep,goal-seeking exploration in the environment.
We show the results in Figure 3, where we see the CASCADE exploration policies cover the statespace far more effectively than all baselines.
In FourRooms , both CASCADE and PP 2 E ﬁnd the goalan equal number of times, while CASCADE ﬁnds almost double the number of rewarding episodes inthe MultiRoom environment.
We also test zero-shot performance, where CASCADE achieves almost100% success rate.
Interestingly, in both settings the Random policy does cover a large proportionof the state space, but ﬁnds fewer rewarding episodes and subsequently does not have a sufﬁcientquantity to train a policy to solve the task (and accordingly achieves 0% success rate).
Next we consider three Atari games: Montezuma’s Revenge , Frostbite and Hero , often used asbarometers for exploration capabilities [26].
We use 15 reward-free deployments, each consisting of200k steps, and use the ﬁnal model to train a behavior policy with task reward. We test these agents ina zero-shot fashion, with results shown in the ﬁrst three columns of Figure 4.
We see that CASCADEgets statistically signiﬁcantly stronger zero-shot returns compared to the baselines. Furthermore, thebaseline methods display no clear trend, with random exploration sometimes offering the strongestperformance. On the other hand, we see that CASCADE performs consistently well, providing furtherevidence of it being a strong general exploration strategy. See App.
A.2 for more detailed analysis. Finally, we consider the Crafter environment [36], a procedurally generated grid world based on thegame of Minecraft. This more complex setting requires the agent to master a variety of compositionalskills to fully explore all possible behaviors.
Online P 2 E represents the state of the art reward-freeagent, achieving a “Crafter Score” of 2.1. Making the problem more challenging, we deploy only 20distinct exploration policies, each collecting 50k timesteps, which leads to weaker performance for
P 2 E vs. the reported result in [39]. However, as we see in Figure 4, adding a population of diverseexplorers recovers the majority of this performance. We ﬁnd this result encouraging as the behaviorsrequired to achieve a higher Crafter score are non-trivial. We believe this opens the door to furthergains with more active tuning of the behavioral representation used in CASCADE .
4.2 Exploring Behaviors
Next we wish to test is whether our agents sufﬁciently explore a range of behaviors in a continuouscontrol setting, using the DM Control Suite [109] (DMC).
In these experiments, we operate frompixels and assume access to an initial dataset of 200k transitions.
Then, we conduct two furtherreward-free deployments, each collecting 200k transitions.
We consider the possibility of improvinggenerality from arbitrary ofﬂine data, using common datasets from the ofﬂine RL literature: random ,medium and expert . When using random data, this represents learning from scratch.
We focus onthe walker environment and evaluate our agents on the four tasks proposed in URLB [55].
After each deployment, we test the generality of the world model by training a separate agent fromscratch for each of the four individual tasks: stand , walk , run , flip .
This demonstrates whether theworld model is capable of producing an imaginary MDP that facilitates learning speciﬁc behaviors.
To compare the performance of CASCADE and the baselines, in Figure 5 we show the aggregatedstatistics from the RLiable library [3] for a given number of deployments.
To do this, we combinethe results across all datasets ( random , medium , expert ) and all downstream tasks ( stand , walk ,run , flip ) and compute the following: 1) the Inter-Quartile Mean (IQM) of the normalized scores(the robust statistic recommended in [3]); 2) the Probability of Improvement, the likelihood of onemethod outperforming another on a new, unseen task.
We show the results for each deployment in Figure 5.
As we see, CASCADE shows clear and consistent gains over the next best baseline ( PP 2 E )throughout 15 deployments, showing statistical signiﬁcance under the Mann-Whitney U test [65].
Note that here the single agent P2E performs poorly, likely due to only collecting data with a singlemode of exploratory behavior. See Figure 6 in the Appendix for a full breakdown of the performance.
4.3 Discussion and Limitations
We have shown that CASCADE is the most effective method in training generally capable agentsacross a wide range of environments.
Indeed, CASCADE provides the best of both worlds: deepexploration by seeking the frontier of the current knowledge (via InfoGain), while covering thespace of possible behaviors (via ImagDiv).
Each of our baselines lacked in one of these two keyproperties.
Random exploration fared poorly when exploring worlds, which requires deep exploration.
However, it can be a highly effective method for exploring behaviors.
This is likely why boostingentropy in the action space is so effective for robotics tasks [35], but less so in deep exploration tasks[74].
Furthermore, the information theoretic baselines were effective in Exploring Worlds, but thedisappointing performance for P 2 E in DMC is likely caused by a lack of data diversity, while the PP 2 E populations were not sufﬁciently heterogeneous to match the coverage of CASCADE .
In terms of limitations, the main weakness in current methods is the absolute level of performance.
Despite outperforming baselines in DMC, the policies learned by CASCADE are far from optimal.
Thisis likely due to the challenge of the setting (reward-free deployment efﬁciency) and we believe thatfuture work in the community will be crucial to further improve these methods.
Finally, CASCADEdoes require training additional agents that may provide additional computational expense.
In thissetting we assume the time between deployments is arbitrary, but this may not be the case in allsettings and future work could consider ways to make this process more efﬁcient and scalable.
Related Work
At the core of our paper is the notion of training generalist agents, which have recently been ofincreased interest in RL [89, 67, 107, 108].
In particular, we focus on the reward-free paradigm, sincein many cases it may not be possible to know a priori all possible tasks we may wish our agent to solve[44].
With this in mind, there has been a surge of interest in methods that can learn in a self-supervisedfashion, to build representations that lead to fast (or even zero-shot) adaptation to future tasks.
These methods range from competence-based (i.e. learning “skills”) [1, 32, 27, 58, 61, 99] to data-basedthat aim to maximize data diversity [62, 116].
In this work we take inspiration from knowledge-based approaches, which typically seek to learn aworld model [9, 82, 83, 29, 105, 15, 96].
In particular we build on Plan2Explore [95] which trains anexploration agent by maximizing an objective resembling information gain inside the world model.
Plan2Explore was the ﬁrst work that showed it is possible to transfer a model learned without rewardsin a zero-shot fashion.
However, it differed from our work in that Plan2Explore solely considers theonline RL setting, deploying a new exploration policy every timestep.
Other recent work examinedthe effectiveness of reward-free RL for collecting datasets for subsequent ofﬂine RL [115, 54].
Onceagain however, this work trains the exploration agent online.
By contrast, we only deploy a handfulof exploration agents and collect a larger quantity of data with each one.
Our setting resembles a reward-free version of the deployment efﬁciency paradigm, ﬁrst introducedin [66].
By contrast, their method focuses on policy optimization in the supervised RL paradigm,whereas we focus on exploration and zero-shot generalization in the unsupervised RL setting.
Followup work has remained focused on supervised policy optimization [100], whereas we are focused ontraining generalist agents in the reward-free paradigm.
To address the challenges of this novel problem setting, we draw inspiration from a variety of ﬁelds.
In particular, several works have considered exploration with a population of agents [104, 25, 63], butthese have only considered task-speciﬁc online RL.
Our choice of a diverse population of agents takesinspiration from the ﬁeld of Quality Diversity (QD, [84, 23]), where it has been shown that diversitycan boost exploration in RL [21].
Recently, [60] showed it is possible to train diverse populationsinside a world model to acquire new skills.
As far as we are aware, we are the ﬁrst to consider acascading objective for training diverse populations, which is typically done either synchronously[81] or sequentially [121, 118].
Also related are works that speciﬁcally train policies that are able to maximally cover the statespace of a given MDP [2, 69].
However, these works focus on the more theoretically amenable (andless general) low-rank MDP setting, and hence do not demonstrate any experiments on larger scaleproblems.
Furthermore, we aim to achieve state-space coverage by encouraging policy diversity ,which is distinct from the approaches taken in these works.
We make use of an information theoreticobjective, which has been shown to be highly effective for exploration in RL [70, 42, 40].
Mostsimilar to our work [119] consider the reward free RL setting, and show that maximizing entropy canlead to improved exploration that can facilitate transfer to arbitrary reward functions.
By contrastwe consider the limited deployment setting and use model-based RL.
Finally, our work relates tocollective intelligence [34, 114], since each member of the population contributes a small piece to thepuzzle that produces a general world model.
Conclusion
We introduced the reward-free deployment efﬁciency paradigm—an important problem setting forlearning generally capable agents in a scalable fashion.
To address the challenges of this setting, weproposed CASCADE , a theoretically motivated approach that leverages a diverse population of selfsupervised exploration agents.
We showed in a variety of experiments that CASCADE produces generalexploration strategies , adept at both deep exploration and gathering sufﬁciently hetereogeneousdatasets that facilitate the learning of diverse downstream behaviors.
For future work, we couldconsider alternative means for computing diversity, for example using optimization landscapes[80] or explicit behavioral diversity [81].
These could also be combined with representations thatdirectly incorporate the behaviors of the policy [14] to further improve the design of the “summary”embedding space.
Furthermore, we would be interested in considering transformer models [111] fortrajectory modelling [18, 43] that ought to scale more gracefully with larger batches of data [86].
Recent work has shown that this class of models can compose unseen complex multi-task behaviorsat test time from ﬁxed ofﬂine datasets [31], as well as leverage signiﬁcantly large and diverse ofﬂinedatasets to improve behavioral modeling [90], highly relevant to both aforementioned desiderata forthe reward-free deployment efﬁciency paradigm.