Semantic Exploration from Language Abstractions and Pretrained Representations
(s
Abstract
Effective exploration is a challenge in reinforcement learning (RL).
Novelty-basedexploration methods can suffer in high-dimensional state spaces, such as continuouspartially-observable 3D environments.
We address this challenge by deﬁningnovelty using semantically meaningful state abstractions, which can be found inlearned representations shaped by natural language.
In particular, we evaluatevision-language representations, pretrained on natural image captioning datasets.
We show that these pretrained representations drive meaningful, task-relevantexploration and improve performance on 3D simulated environments.
We alsocharacterize why and how language provides useful abstractions for explorationby considering the impacts of using representations from a pretrained model,a language oracle, and several ablations.
We demonstrate the beneﬁts of ourapproach in two very different task domains—one that stresses the identiﬁcation andmanipulation of everyday objects, and one that requires navigational exploration inan expansive world—as well as two popular deep RL algorithms: Impala and R2D2.
Our results suggest that using language-shaped representations could improveexploration for various algorithms and agents in challenging environments.
Introduction
Exploration is one of the central challenges of reinforcement learning (RL).
A popular way to increasean agent’s tendency to explore is to augment trajectories with intrinsic rewards for reaching novelenvironment states.
However, the success of this approach depends critically on which states areconsidered novel, which can in turn depend on how environment states are represented.
The literature on novelty-driven exploration describes several approaches to deriving state representations [6].
One popular method employs random features and represents the state by embeddingthe visual observation with a ﬁxed, randomly initialized target network [RND; 5].
Another methoduses learned visual features, taken from an inverse dynamics model [NGU; 2].
These approacheswork well in classic 2D environments like Atari but it is less clear whether they are as effective inhigh-dimensional, partially-observable settings such as 3D environments.
For instance, in 3D settings, different viewpoints of the same scene may map to distinct visual states/features, despite beingsemantically similar.
The difﬁculty of identifying a good mapping between visual state and featurespace is exacerbated by the fact that useful state abstractions are highly task dependent.
For example, a task involving tool use requires object-affordance abstractions, whereas navigation does not.
Thus, acquiring state representations that support effective exploration is a chicken-and-egg problem—knowing whether two states should be considered similar requires the type of understanding that anagent can only acquire after effectively exploring its environment.
To overcome these challenges, we propose to endow agents with prior knowledge, in the formof abstractions derived from large vision-language models [e.g. 38] that are pretrained on image captioning data.
We hypothesize that representations acquired by vision-language pretraining driveeffective, semantic exploration in 3D environments, because the representations are shaped by theunique abstract nature of natural language.
Several aspects of natural languagesuggest that it could be useful to direct novelty-based exploration.
First,language is inherently abstract: language links superﬁcially distinct, butcausally-related situations by describing them similarly, and contrasts between causally-distinct states by describing them differently, thus outlining useful concepts [27, 26].
Second,humans use language to communicate important information efﬁciently,without overspecifying [18, 19].
Thus,human language omits distracting irrelevant information and focuses onimportant aspects of the world.
Forexample, it is often observed that anagent rewarded for seeking novel experience would be attracted forever toa TV with uncontrollable and unpredictable random static [6].
However, ahuman would likely caption this scene“a TV with no signal” regardless of theparticular pattern; thus an agent exploring with language abstractions would quickly leave the TV behind.
Figure 1 shows anotherconceptual example of how language abstractions can accelerate exploration.
We ﬁrst perform motivating proof-of-concept experiments using a language oracle.
We show thatlanguage is a useful abstraction for exploration not only because it coarsens the state space, but alsobecause it coarsens the state space in a way that reﬂects the semantics of the environment.
We thendemonstrate that our results scale to environments without a language oracle using pretrained visionmodels, which are only supervised with language during pretraining.
We create language-augmentedvariants of two popular exploration methods from the literature, Never Give Up (NGU; Badia et al.[2]) and Random Network Distillation (RND; Burda et al.
We compare the performance of theoriginal and language-augmented algorithms on object manipulation, search, and navigation tasksin two challenging 3D environments simulated in Unity: Playroom (a house containing toys andfurniture) and City (a large-scale urban setting).
Our results show that language-based explorationwith pretrained vision-language representations improves sample efﬁciency on Playroom tasks by18-70%.
It also doubles the visited areas in City, compared to baseline methods.
We show thatlanguage-based exploration is effective for both IMPALA [15] and R2D2 [23] agents.
Related Work
Exploration in RL Classical exploration strategies include (cid:15) -greedy action selection [48], statecounting [46, 3, 29, 28, 2], curiosity driven exploration [41], and intrinsic motivation methods [33].
Our work is part of this last class of methods, where the agent is given an intrinsic reward for visitingdiverse states over time [32].
Intrinsic rewards can be derived from various measures: novelty[40, 52, 5, 53], prediction-error [35, 2], ensemble disagreement [9, 36, 45, 43, 16, 47], or informationgain [21].
One family of methods gives intrinsic reward for following a curriculum of goals [7, 10, 37].
Others use novelty measures to identify interesting states from which they can perform additionallearning [14, 51].
These methods encourage exploration in different ways, but they all rely on visualstate representations that are learned jointly with the policy.
Although we focus on novelty-basedintrinsic reward and demonstrate the beneﬁts of language in NGU and RND, our methodology isrelatively agnostic to the exploration method.
We suggest that many other exploration methods couldbe improved by using language abstractions and pretrained embeddings to represent the state space. [Figure]
Pretraining representations for RL Pretraining has been used in RL to improve the representations of the policy network.
Self-supervised representation learning techniques distill knowledge fromexternal datasets to produce downstream features that are helpful in virtual environments [13, 50].
Some recent work shows beneﬁts from pretraining on more general, large-scale datasets.
Pretrained
CLIP features have been used in a number of recent robotics papers to speed up control and navigation tasks.
These features can condition the policy network [24], or can be fused throughout the visualencoder to integrate semantic information about the environment [34].
Pretrained language models can also provide useful initializations for training policies to imitate ofﬂine trajectories [39, 25].
These successes demonstrate that large pretrained models contain prior knowledge that can be usefulfor RL.
While the existing literature uses pretrained embeddings directly in the agent, we insteadallow the policy network to learn from scratch, and only use pretrained embeddings to guide exploration.
We imagine that future work may beneﬁt from combining both approaches.
Language for exploration Some recent works have used language to guide agent learning, byeither using language subgoals for exploration/planning or providing task-speciﬁc reward shaping[44, 30, 11, 17].
Schwartz et al.
[42] use a custom semantic parser for VizDoom and show thatrepresenting states with language, rather than vision, leads to faster learning.
Chaplot et al.
[8]tackle navigation in 3D by constructing a semantic map of the environment from pretrained SLAMmodules and language-deﬁned object categories.
Work concurrent to ours by Mu et al.
[31] showshow language, in the form of hand-crafted BabyAI annotations, can help improve exploration in 2Denvironments.
These works demonstrate the value of language abstractions: the ability to ignoreextraneous noise and highlight important environment features.
However, these prior methods rely onenvironment-speciﬁc semantic parsers or annotations.
In contrast, by exploiting powerful pretrainedvision-language models, our approach can be applied to any visually-naturalistic environment,including 3D settings, which have not been widely studied in prior exploration work.
Our methodcould even potentially improve exploration for physical robots, but we leave that for future work.
Method
We consider a goal-conditioned Markov decision process deﬁned by a tuple ( S , A , G , P, R e , γ ) ,where S is the state space, A is the action space, G is the goal space, P : S ×
A → S speciﬁes theenvironment dynamics, R e : S × G → R e is the extrinsic reward, and γ is the discount factor.
States t is presented to the agent as a visual observation O V .
In some cases, in order to calculate intrinsic reward, we use a language oracle O : S
→ L that provides natural language descriptions of the state,
O L .
Note that O L is distinct from the language instruction g ∈ G , which is sampled from a goaldistribution at the start of an episode—the agent never observes O L .
We later remove the need for alanguage oracle by using pretrained models.
We use goal-conditioned reinforcement learning to produce a policy π g ( · | O V ) that maximizes theexpected reward E [ (cid:80)
Ht =
γ t ( r et + βr it )] , where H is the horizon, r et is the extrinsic reward, r it isthe intrinsic reward, and β is a tuned hyperparameter.
The intrinsic reward is goal-agnostic and iscomputed with access to either O V or O L .
Our approach builds on two popular exploration algorithms: Never Give Up (NGU; Badia et al.[2]) and Random Network Distillation (RND; Burda et al.
These algorithms were chosen todemonstrate the value of language under two different exploration paradigms.
While both methodsreward visiting novel states, they differ on several dimensions: the novelty horizon (episodic versuslifetime), how the history of past visited states is retained (non-parametric versus parametric), andhow states are represented (learned controllable states versus random features).
[Table]
Intrinsic reward is derived from the prediction error between the trainable network and frozen target function.
[Table]
ˆ f LD f C :
O V → O L
O L from language oracle
3.1 Never Give Up (NGU)
To more clearly isolate our impact, we focus only on the episodic novelty component of the NGUagent [2].
State representations along the trajectory are written to a non-parametric episodic memorybuffer.
The intrinsic reward reﬂects how novel the current state is relative to the states visited so farin the episode.
Novelty is a function of the L2 distances between the current state and the k -nearestneighbor representations stored in the memory buffer.
Intrinsic reward is higher for larger distances.
Full details can be found in the original paper; however, we make two key simpliﬁcations.
While
Badia et al.
[2] proposes learning a family of policy networks that are capable of different levelsof exploration, we train one policy network that maximizes reward r = r e + βr i for a ﬁxedhyperparameter β .
We also ﬁx the long-term novelty modulator α to be 1, essentially removing it.
The published baseline method, which we refer to as Vis-NGU , uses a controllable state taken froman inverse dynamics model.
The inverse dynamics model is trained jointly with the policy, but thetwo networks do not share any parameters.
The intrinsic reward relies on directly comparing state representations from the buffer, so ourapproach focuses on modifying the embedding function to inﬂuence exploration (Table 1).
LangNGU uses a frozen pretrained language encoder to embed the oracle caption O L .
We comparelanguage embeddings from BERT [12], CLIP [38], Small-ALM, and Med-ALM.
The ALMs (ALign
Models) are trained with a contrastive loss on the ALIGN dataset [22].
Small-ALM uses a 26Mparameter ResNet-50 image encoder [20]; Med-ALM uses a 71M parameter NFNet [4].
The languagebackbones are based on BERT and are all in the range of 70-90M parameters.
We do not ﬁnetune onenvironment-speciﬁc data; this preserves the real world knowledge acquired during pretraining anddemonstrates its beneﬁt without requiring any environment-speciﬁc captions.
LSE-NGU does not use the language oracle.
Instead, it uses a frozen pretrained image encoderto embed the visual observation O V .
We use the image encoder from CLIP or ALM, which aretrained on captioning datasets to produce outputs that are close to the corresponding languageembeddings.
The human-generated captions structure the visual embedding space to reﬂect featuresmost pertinent to humans and human language, so the resulting representations can be thought of as Language Supervised Embeddings (LSE).
The primary beneﬁt of LSE-NGU is that it can be applied toenvironments without a language oracle or annotations.
CLIP and ALM are trained on real-world data,so they would work best on realistic 3D environments.
However, we imagine that in future work thepretraining process or dataset could be tailored to maximize transfer to a desired target environment.
3.2 Random Network Distillation (RND)
Our RND-inspired family of methods rewards lifetime novelty.
Generically, the intrinsic reward isderived from the prediction error between a trainable network and some target value generated by afrozen function (Table 2).
The trainable network is learned jointly with the policy network, althoughthey do not share any parameters.
As the agent trains over the course of its lifetime, the prediction errorfor frequently-visited states decreases, and the associated intrinsic reward consequently diminishes.
Intuitively, the weights of the trainable network implicitly store the state visitation counts.
For clarity, we refer to the baseline published by Burda et al.
[6] as Vis-RND .
The trainable networkf V :
O V → R k maps the visual state to random features.
The random features are produced bya ﬁxed, randomly initialized network ˆ f V .
Both f V and ˆ f V share the same architecture: a ResNetfollowed by a MLP.
The intrinsic reward is the mean squared error (cid:107) f V ( O V ) − ˆ f V
( O V ) (cid:107) 2 .
In network distillation ( ND ), the target function is not random, but is instead a pretrained text or imageencoder from CLIP/ALM.
The trainable network f learns to reproduce the pretrained representations.
To manage inference time, f is a simpler network than the target (see Appendix A.2).
The intrinsic lossis the mean squared error between f and the large pretrained network.
Like the respective Lang-NGUand LSE-NGU counterparts, text-based ND requires a language oracle, but image-based ND does not.
In Section 5.1 we compare against two additional methods to motivate why language is a usefulabstraction.
The ﬁrst, Lang-RND , is a variant in which the trainable network f L : O L → R k maps theoracle caption to random features.
The intrinsic reward is the mean squared error between the outputsof f L and ﬁxed ˆ f L with random initialization.
Both f L and ˆ f L networks are of the same architecture.
The second method, language distillation ( LD ), is loosely inspired by RND in that the novelty signalcomes from a prediction error.
However, instead of learning to produce random features, the trainablenetwork learns to caption the visual state, i.e. f C :
O V → O L .
The network architecture consists ofa CNN encoder and LSTM decoder.
The intrinsic reward is the negative log-likelihood of the oraclecaption under the trainable model f C .
In LD, the exploration dynamics not only depend on howfrequently states are visited but also the alignment between language and the visual world.
We testwhether this caption-denoted alignment is necessary for directing semantic exploration by comparing
LD to a variant with shufﬂed image-language alignment ( S-LD ) in Section 5.1.
Experimental Setup
4.1 Environments
Previous exploration work benchmarked algorithms on video games, such as 2D grid-world MiniHackand Montezuma’s Revenge, or 3D ﬁrst-person shooter Vizdoom.
In this paper, we focus on ﬁrst-person
Unity-based 3D environments that are meant to mimic familiar scenes from the real world (Figure 2).
Playroom Our ﬁrst domain, Playroom [1, 49], is a randomly-generated house containing everydayhousehold items (e.g. bed, bathtub, tables, chairs, toys).
The agent’s action set consists of 46 discreteactions that involve locomotion primitives and object manipulation, such as holding and rotating.
We study two settings in Playroom.
In the ﬁrst setting, the agent is conﬁned to a single room with3-5 objects and is given a lift or put instruction.
At the start of an episode, the set of objects aresampled from a larger set of everyday objects (i.e. a candle, cup, hairdryer).
Object colors and sizesare also randomized, adding superﬁcial variance to different semantic categories.
The instructionstake the form: "Lift a <object>" or "Put a <object> on a {bed, tray}" .
With a liftgoal, the episode ends with reward 1 or 0 whenever any object is lifted.
With a put goal, the episodeends with reward 1 when the condition is fulﬁlled.
This setting tests spatial rearrangement skills.
In the second setting, the agent is placed in a house with 3-5 different rooms, and is given a findinstruction of the form "Find a {teddy bear, rubber duck}" .
Every episode, the house israndomly generated with the teddy and duck hidden amongst many objects, furniture, and decorations.
The target objects can appear in any room— either on the ﬂoor, on top of tables, or inside bookshelves.
The agent is randomly initialized and can travel throughout the house and freely rearrange objects.
The episode ends with reward 1 when the agent pauses in front of the desired object.
The find taskrequires navigation/search skills and tests the ability to ignore the numerous distractor objects.
City Our second domain, City, is an expansive, large-scale urban environment.
Each episode, a newmap is generated; shops, parks, and buildings are randomly arranged in city blocks.
Daylight is simulated, such that the episode starts during the morning and ends at nighttime.
The agent is randomlyinitialized and is instructed to “explore the city.”
It is trained to maximize its intrinsic reward andcan take the following actions: move_{forward,backward,left,right} , look_{left,right} ,and move_forward_and_look_{left,right} .
We divide up the map into a 32 × 32 grid and trackhow many unique bins are visited in an episode.
Additionally, City does not provide explicit visual or verbal signage to disambiguate locations.
Assuch, systematic exploration is needed to maximize coverage.
In contrast to Playroom, City testslong horizon exploration.
A Playroom episode lasts only 600 timesteps, whereas a City episode lasts11,250 and requires hundreds of timesteps to fully traverse the map even once.
The City covers a270-by-270 meter square area, which models a 2-by-2 grid of real world blocks.
4.2 Captioning Engine
We equip the environment with a language oracle that generates language descriptions of the scene,
O L , based on the Unity state, s (Figure 2).
In Playroom, the caption describes if and how the agentinteracts with objects and lists what is currently visible to it.
In City, O L generally describes theobject that the agent is directly looking at, but the captions alone do not disambiguate the agent’slocations.
Since these captions are generated from a Unity state, these descriptions may not be asvaried or rich as a human’s, but they can be generated accurately and reliably, and at scale.
4.3 Training Details
At test time, the agent receives image observation
O V and language-speciﬁed goal g .
The policynetwork never requires caption
O L to act.
During training, the exploration method calculates theintrinsic reward from O L or O V .
We show that language-based exploration is compatible with both policy gradient and Q-learningalgorithms.
We use Impala [15] on Playroom and R2D2 on City [23].
Q-learning is more suitable forthe City, because the action space is more restricted compared to the one needed for Playroom tasks.
For both environments, the agent architecture consists of an image ResNet encoder and a language
LSTM encoder that feed into a memory LSTM module.
The policy and value heads are MLPs thatreceive the memory state as input.
If the exploration method requires additional networks, such as thetrainable network in RND or inverse dynamics model in NGU, they do not share any parameters withthe policy or value networks.
Hyperparameters and additional details are found in Appendix A.
Results
5.1 Motivation: Language is a Meaningful Abstraction
We share the intuition with other work [e.g. 31] that language can improve exploration.
We design a setof experiments to show how and why this may be the case.
Our analysis follows the desiderata outlinedby Burda et al. [5]—prediction-error exploration ought to use a feature space that ﬁlters irrelevantinformation (compact) and contains necessary information (sufﬁcient).
Burda et al.
[5] speciﬁcallystudies RND and notes that the random feature space, the outputs of the random network, may notfully satisfy either condition.
As such, we use the language variants of RND to frame this discussion.
We hypothesize that language abstractions are useful, because they (1) create a coarser state space and(2) divide the state space in a way that meaningfully aligns with the world (i.e. using semantics).
First,if language provides a coarser state space, then the random feature space becomes more compact,leading to better exploration.
We compare Lang-RND to Vis-RND to test this claim.
Lang-RNDlearns the lift task 33% faster and solves the put task as Vis-RND starts to learn (Figure 3a).
Second, we ask whether semantics – that is how language divides up the state space – is critical foreffective exploration.
We use LD to test this hypothesis, precisely because the [Figure] [Figure] [Figure]
The states are grouped together based on similarities in the visual random feature space and assigned a label.
Exploring in this shufﬂed space is less effective than exploring with the semanticallymeaningful abstractions shown in Figure 1.
We compare LD to a shufﬂed variant SLD, where we replace the particular semantic state abstraction that language offers witha statistically-matched randomized abstraction(Figure 4).
S-LD is similar to LD; the intrinsic reward is the prediction error of the captioning network.
However, instead of targeting thelanguage oracle output, the S-LD trainable network produces a different target caption (cid:102)
O L thatmay not match the image.
(cid:102)
O L is produced by aﬁxed, random mapping ˆ f S : O V → (cid:102) O L .
ˆ f S isconstrained such that the marginal distributions
P ( O L ) ≈ P ( (cid:102) O L ) are matched under trajectories produced by policy π LD .
See Appendix A.4for full details on the construction of S-LD.
Thus, whereas the LD captions parcel up statespace in a way that reﬂects the abstractions thatlanguage offers, the randomized mapping ˆ f Sparcels up state space in a way that abstracts overrandom features of the visual space (Figure 4).
We control for the compactness and coarsenessof the resulting representation by maintainingthe same marginal distribution of captions.
If semantics is crucial for exploration, then weexpect to see LD outperform S-LD.
This indeedholds experimentally (Figure 3b).
We can alsoview these results under the Burda et al.
[5] framework.
The S-LD abstractions group togethervisually similar, but semantically distinct states.
A single sampled caption likely fails to capture thegroup in a manner that is representative of all the encompassing states.
In other words, ˆ f S producesa compact feature space that may not be sufﬁcient.
This may explain why S-LD learns faster than Vis-RND on the simpler lift task but fails on the more complex put and find tasks.
The S-LDexperiments imply that language abstractions are helpful for exploration because they expose notonly a more compact, but also a more semantically meaningful state space.
5.2 Pretrained Vision-Language Representations Improve Exploration
Having shown how language can be helpful for exploration, we now incorporate pretrained visionlanguage representations into NGU and RND to improve exploration.
Such representations (e.g. fromthe image encoder in CLIP/ALM) offer the beneﬁts of explicit language abstractions, without theneed to rely on a language oracle.
We also compare language-shaped representations to pretrained
ImageNet embeddings to isolate the effect of language.
To keep the number of experiments tractable,we only perform a full comparison on the Playroom tasks.
[Figure]
City We ﬁrst compare how representations affect performance in a pure exploration setting.
With no extrinsic reward, the agent is motivatedsolely by the NGU intrinsic reward to explorethe City.
We report how many unique areas theagent visits in an episode in Figure 5.
Whileoptimizing coverage only requires knowledgeof an agent’s global location rather than genericscene understanding, vision-language representations are still useful simply because meaningful exploration is inherently semantic.
LangNGU, which uses text embeddings of O L , visitsan area up to 3 times larger.
LSE-NGU achieves2 times the coverage even without querying alanguage oracle (Appendix Figure S3).
Playroom We next show that pretrainedvision-language representations signiﬁcantlyspeed up learning across all Playroom tasks (Figure 6).
The LSE-NGU and Lang-NGU agentsimprove sample efﬁciency by 50-70% on thelift and put tasks and 18-38% on the findtask, depending on the pretraining model used.
The ND agents are signiﬁcantly faster than VisRND, learning 41% faster on the find task.
Wealso measure agent-object interactions.
Nearlyall LSE-NGU and Lang-NGU agents learn tofoveate on and hold objects within 40k learningupdates, whereas Vis-NGU agent takes at least60k updates to do so with the same frequency(Appendix Figure S5).
Although LSE-NGU and image-based ND agents do not access a language oracle, they are similarly effective as their annotation-dependent counterparts in the Playroom tasks (Appendix Figure S4), suggesting that our method could be robust to the availability of a language oracle.
To demonstrate the value of rich language, we compare LSE-NGU agents to a control agent thatinstead uses pretrained ImageNet embeddings from a 70M NFNet [4].
ImageNet embeddingsoptimize for single-object classiﬁcation, so they confer some beneﬁt to the most object-focused tasks,lift and put .
However, ImageNet embeddings hurt exploration in the find task, where agentsencounters more complex scenes (Figure 6b).
By contrast, the language-shaped representations arewell-suited for not only describing simple objects, but also have capacity for multi-object, complexscenes.
Of course, current CLIP-style models can be further improved in their ability to understandmulti-object scenes, which may explain why the beneﬁts are less pronounced for the find task.
However, as the performance of pretrained vision-language models improve, we expect to see thosebeneﬁts transfer to this method and drive even better exploration.
Discussion
We have shown that language abstractions and pretrained vision-language representations improve thesample efﬁciency of existing exploration methods.
This beneﬁt is seen across on-policy and off-policyalgorithms (Impala and R2D2), different exploration methods (RND and NGU), different 3D domains [Figure] [Figure] [Figure]
[Figure] [Figure] (Playroom and City), and task speciﬁcations (lifting/putting, searching, and intrinsicallymotivated navigation). Furthermore, we carefully designed control experiments to understand howlanguage contributes to better exploration.
Our results are consistent with cognitive perspectives onhuman language—language is powerful because it groups together situations according to semanticsimilarity.
In terms of the desiderata that Burda et al.
[5] present, language is both compact andsufﬁcient.
Finally, we note that using pretrained vision-language representations to embed imageobservations enables more effective exploration even if language is not available during agent training.
This is vital for scaling to environments that do not have a language oracle or annotations.
Limitations and future directions
We highlight several avenues for extending our work.
First,additional research could provide a more comprehensive understanding of how language abstractionsaffect representations.
This could include comparing different types of captions offering varyinglevels of detail, or task-dependent descriptions.
Second, it would be useful to investigate how toimprove pretrained vision-language representations for exploration by ﬁnetuning on relevant datasets.
The semantics of the dataset could even be tailored to task-speciﬁc abstractions to increase thequality of the learnt representations.
Such approaches would potentially allow applying our methodto environments that are farther from the pretraining distribution, such as Atari.
Finally, we note alimitation with this approach is that current pretrained vision-language models are susceptible todomain shift on virtual environments and have been shown to be less effective on multi-object scenes.
Thus, future pretrained models that are larger or otherwise improved would presumably improve thequality of these representations and lead to even more effective exploration. Checklist [Table] For all authors...
[Table]
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
[Table]
contributions and scope?
[Yes]
[Table]
(b) Did you describe the limitations of your work?
[Yes] See Section 6 paragraph on
[Table]
Limitations and future directions.
[Table]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
[Table]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
[Table]
imental results (either in the supplemental material or as a URL)?
[Yes] We plan to
[Table]
open-source the agent-training code and all environments not previously released with
the camera-ready version.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
[Table]
were chosen)?
[Yes] See Appendix Section A.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars and error bands are included in all results
ﬁgures and are deﬁned in the ﬁgure legends.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We trained on a 2 × 2 TPUv2with 256 parallel CPU actors.
Training took 8-36 hours to reach 200k learner updates.
These and other training details are discussed in Appendix Section A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] .
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]