Semantic Exploration from Language Abstractions and Pretrained Representations
Anonymous Author(s)
Abstract
Effective exploration is a challenge in reinforcement learning (RL).
Novelty-basedexploration methods can suffer in high-dimensional state spaces, such as continuouspartially-observable 3D environments.
We address this challenge by deﬁningnovelty using semantically meaningful state abstractions, which can be found inlearned representations shaped by natural language.
In particular, we evaluatevision-language representations, pretrained on natural image captioning datasets.
We show that these pretrained representations drive meaningful, task-relevantexploration and improve performance on 3D simulated environments.
We alsocharacterize why and how language provides useful abstractions for explorationby considering the impacts of using representations from a pretrained model, alanguage oracle, and several ablations.
We demonstrate the beneﬁts of our approachwith on- and off-policy RL algorithms and in two very different task domains—one that stresses the identiﬁcation and manipulation of everyday objects, and onethat requires navigational exploration in an expansive world.
Our results suggestthat using language-shaped representations could improve exploration for variousalgorithms and agents in challenging environments.
Introduction
Exploration is one of the central challenges of reinforcement learning (RL).
A popular way to increasean agent’s tendency to explore is to augment trajectories with intrinsic rewards for reaching novelenvironment states.
However, the success of this approach depends critically on which states areconsidered novel, which can in turn depend on how environment states are represented.
The literature on novelty-driven exploration describes several approaches to deriving state representations [6].
One popular method employs random features and represents the state by embeddingthe visual observation with a ﬁxed, randomly initialized target network [Random Network Distillation; 5].
Another method uses learned visual features, taken from an inverse dynamics model [Never Give Up; 2].
These approaches work well in classic 2D environments like Atari but it is less clearwhether they are as effective in high-dimensional, partially-observable settings such as 3D environments.
For instance, in 3D settings, different viewpoints of the same scene may map to distinct visualstates/features, despite being semantically similar.
The difﬁculty of identifying a good mappingbetween visual state and feature space is exacerbated by the fact that useful state abstractions arehighly task dependent.
For example, a task involving tool use requires object-affordance abstractions,whereas navigation does not.
Thus, acquiring state representations that support effective explorationis a chicken-and-egg problem—knowing whether two states should be considered similar requiresthe type of understanding that an agent can only acquire after effectively exploring its environment.
To overcome these challenges, we propose to give agents access to additional prior knowledge duringtraining, in the form of abstractions derived from large vision-language models [e.g. 39] that are pretrained on image captioning data.
We use these pretrained models to derive a intrinsic rewardthat reﬂects meaningful novelty.
We hypothesize that representations acquired by vision-languagepretraining drive effective, semantic exploration in 3D environments, because the representations areshaped by the unique abstract nature of natural language.
Several aspects of natural languagesuggest that it could be useful to direct novelty-based exploration.
First,language is inherently abstract: language links superﬁcially distinct, butcausally-related situations by describing them similarly, and contrasts between causally-distinct states by describing them differently, thus outlining useful concepts [27, 26].
Second,humans use language to communicate important information efﬁciently,without overspecifying [18, 19].
Thus,human language omits distracting irrelevant information and focuses onimportant aspects of the world.
Forexample, it is often observed that anagent rewarded for seeking novel experience would be attracted forever toa TV with uncontrollable and unpredictable random static [6].
However, ahuman would likely caption this scene“a TV with no signal” regardless of theparticular pattern; thus an agent exploring with language abstractions would quickly leave the TV behind.
Figure 1 shows anotherconceptual example of how language abstractions can accelerate exploration.
We ﬁrst perform motivating proof-of-concept experiments using a language oracle.
We show thatlanguage is a useful abstraction for exploration not only because it coarsens the state space, but alsobecause it coarsens the state space in a way that reﬂects the semantics of the environment.
We thendemonstrate that our results scale to environments without a language oracle using pretrained visionmodels, which are only supervised with language during pretraining.
This work strives to enhance therepresentations used in novelty-based exploration, rather than compare various exploration methods.
We consider two popular novelty-based exploration methods from the literature, Never Give Up (NGU; Badia et al. [2]) and Random Network Distillation (RND; Burda et al.
[6]), and compare them totheir language-augmented variants, Lang-NGU/LSE-NGU and Lang-RND.
We evaluate performanceand sample efﬁciency on object manipulation, search, and navigation tasks in two challenging 3Denvironments simulated in Unity: Playroom (a house containing toys and furniture) and City (alarge-scale urban setting).
Our results show that language-based exploration with pretrained visionlanguage representations improves sample efﬁciency on Playroom tasks by 18-70%.
It also doublesthe visited areas in City, compared to baseline methods.
We show that language-based exploration iseffective for both on-policy (IMPALA [15]) and off-policy (R2D2 [23]) agents.
Related Work
Exploration in RL Classical exploration strategies include ✏ -greedy action selection [49], statecounting [47, 3, 30, 28, 2], curiosity driven exploration [42], and intrinsic motivation methods [34].
Our work is part of this last class of methods, where the agent is given an intrinsic reward for visitingdiverse states over time [33].
Intrinsic rewards can be derived from various measures: novelty[41, 53, 5, 54], prediction-error [36, 2], ensemble disagreement [9, 37, 46, 44, 16, 48], or informationgain [21].
One family of methods gives intrinsic reward for following a curriculum of goals [7, 10, 38].
Others use novelty measures to identify interesting states from which they can perform additionallearning [14, 52].
These methods encourage exploration in different ways, but they all rely on visualstate representations that are learned jointly with the policy.
Although we focus on novelty-based [Figure] intrinsic reward and demonstrate the beneﬁts of language in NGU and RND, our methodology isrelatively agnostic to the exploration method.
We suggest that many other exploration methods couldbe improved by using language abstractions and pretrained embeddings to represent the state space.
Pretraining representations for RL Pretraining has been used in RL to improve the representations of the policy network.
Self-supervised representation learning techniques distill knowledge fromexternal datasets to produce downstream features that are helpful in virtual environments [13, 51].
Some recent work shows beneﬁts from pretraining on more general, large-scale datasets.
Pretrained
CLIP features have been used in a number of recent robotics papers to speed up control and navigation tasks.
These features can condition the policy network [24], or can be fused throughout the visualencoder to integrate semantic information about the environment [35].
The goal of these works is toimprove perception in the policy.
Pretrained language models can also provide useful initializationsfor training policies to imitate ofﬂine trajectories [40, 25].
These successes demonstrate that largepretrained models contain prior knowledge that can be useful for RL.
While the existing literatureuses pretrained embeddings directly in the agent, we instead allow the policy network to learn fromscratchm and only utilize pretrained embeddings to guide exploration during training (Figure S2).
We imagine that future work may beneﬁt from combining both approaches.
Language for exploration Some recent works have used language to guide agent learning, byeither using language subgoals for exploration/planning or providing task-speciﬁc reward shaping[45, 31, 11, 17].
Schwartz et al.
[43] use a custom semantic parser for VizDoom and show thatrepresenting states with language, rather than vision, leads to faster learning by simplifying policyinputs.
Chaplot et al.
[8] tackle navigation in 3D by constructing a semantic map of the environmentfrom pretrained SLAM modules, language-deﬁned object categories, and agent location.
Thisapproach lends itself well to navigation, but it is unclear how it would extend easily to other generictasks, such as manipulation.
Work concurrent to ours by Mu et al.
[32] shows how language, inthe form of hand-crafted BabyAI annotations, can help improve exploration in 2D environments.
These works demonstrate the value of language abstractions: the ability to ignore extraneous noiseand highlight important environment features.
However, these prior methods rely on environmentspeciﬁc semantic parsers or annotations, which may limit the settings to which they can be applied.
In contrast, by exploiting powerful pretrained vision-language models, our approach can be appliedto any visually-naturalistic environment, including 3D settings, which have not been widely studiedin prior exploration work.
We additionally do not require any language from the environment itself.
Our method could even potentially improve exploration for physical robots, but we leave that forfuture work.
Method
We consider a goal-conditioned Markov decision process deﬁned by a tuple ( S , A , G , P, R e ,  ) ,where S is the state space, A is the action space, G is the goal space, P : S ⇥ A !
S speciﬁes theenvironment dynamics, R e : S ⇥ G ! R e is the extrinsic reward, and  is the discount factor.
States t is presented to the agent as a visual observation O V .
In some cases, in order to calculate intrinsicreward, we use a language oracle O : S !
L that provides natural language descriptions of the state, 130 O
L .
Note that O L is distinct from the language instruction g 2 G , which is sampled from a goaldistribution at the start of an episode—the agent never observes O L .
We later remove the need for alanguage oracle by using pretrained models.
We use goal-conditioned reinforcement learning to produce a policy ⇡ g ( · | O V ) that maximizes theexpected reward E [ P
Ht =
t ( r et +  r it )] , where H is the horizon, r et is the extrinsic reward, r it isthe intrinsic reward, and  is a tuned hyperparameter.
The intrinsic reward is goal-agnostic and iscomputed with access to either O V or O L .
Note that neither O L nor pretrained embeddings are usedby the policy, and thus we only use them during training to compute the intrinsic reward (Figure S2).
Our approach builds on two popular exploration algorithms: Never Give Up (NGU; Badia et al.[2]) and Random Network Distillation (RND; Burda et al.
These algorithms were chosen todemonstrate the value of language under two different exploration paradigms.
While both methodsreward visiting novel states, they differ on several dimensions: the novelty horizon (episodic versuslifetime), how the history of past visited states is retained (non-parametric versus parametric), andhow states are represented (learned controllable states versus random features).
[Table]
Intrinsic reward is derived from the prediction error between the trainable network and frozen target function.
[Table]
3.1 Never Give Up (NGU)
[Table] To more clearly isolate our impact, we focus only on the episodic novelty component of the NGU [Table] agent [2].
State representations along the trajectory are written to a non-parametric episodic memorybuffer.
The intrinsic reward reﬂects how novel the current state is relative to the states visited so farin the episode.
Novelty is a function of the L2 distances between the current state and the k -nearestneighbor representations stored in the memory buffer.
Intrinsic reward is higher for larger distances.
Full details can be found in the original paper; however, we make two key simpliﬁcations.
While
Badia et al.
[2] proposes learning a family of policy networks that are capable of different levelsof exploration, we train one policy network that maximizes reward r = r e +  r i for a ﬁxedhyperparameter  .
We also ﬁx the long-term novelty modulator ↵ to be 1, essentially removing it.
The published baseline method, which we refer to as Vis-NGU , uses a controllable state taken froman inverse dynamics model.
The inverse dynamics model is trained jointly with the policy, but thetwo networks do not share any parameters.
The intrinsic reward relies on directly comparing state representations from the buffer, so ourapproach focuses on modifying the embedding function to inﬂuence exploration (Table 1).
LangNGU uses a frozen pretrained language encoder to embed the oracle caption O L .
We comparelanguage embeddings from BERT [12], CLIP [39], Small-ALM, and Med-ALM.
The ALMs (ALign
Models) are trained with a contrastive loss on the ALIGN dataset [22].
Small-ALM uses a 26Mparameter ResNet-50 image encoder [20]; Med-ALM uses a 71M parameter NFNet [4].
The languagebackbones are based on BERT and are all in the range of 70-90M parameters.
We do not ﬁnetune on environment-speciﬁc data; this preserves the real world knowledge acquired during pretraining anddemonstrates its beneﬁt without requiring any environment-speciﬁc captions.
LSE-NGU does not use the language oracle.
Instead, it uses a frozen pretrained image encoderto embed the visual observation O V .
We use the image encoder from CLIP or ALM, which aretrained on captioning datasets to produce outputs that are close to the corresponding languageembeddings.
The human-generated captions structure the visual embedding space to reﬂect featuresmost pertinent to humans and human language [29], so the resulting representations can be thoughtof as Language Supervised Embeddings (LSE).
The primary beneﬁt of LSE-NGU is that it can beapplied to environments without a language oracle or annotations.
CLIP and ALM are trained onreal-world data, so they would work best on realistic 3D environments.
However, we imagine thatin future work the pretraining process or dataset could be tailored to maximize transfer to a desiredtarget environment.
3.2 Random Network Distillation (RND)
Our RND-inspired family of methods rewards lifetime novelty.
Generically, the intrinsic reward isderived from the prediction error between a trainable network and some target value generated by afrozen function (Table 2).
The trainable network is learned jointly with the policy network, althoughthey do not share any parameters.
As the agent trains over the course of its lifetime, the prediction errorfor frequently-visited states decreases, and the associated intrinsic reward consequently diminishes.
Intuitively, the weights of the trainable network implicitly store the state visitation counts.
For clarity, we refer to the baseline published by Burda et al.
[6] as Vis-RND .
The trainable networkf V : O V !
R k maps the visual state to random features.
The random features are produced bya ﬁxed, randomly initialized network ˆ f V .
Both f V and ˆ f V share the same architecture: a ResNetfollowed by a MLP.
The intrinsic reward is the mean squared error k f V ( O V )  ˆ f V
( O V ) k 2 .
In network distillation ( ND ), the target function is not random, but is instead a pretrained text or imageencoder from CLIP/ALM.
The trainable network f learns to reproduce the pretrained representations.
To manage inference time, f is a simpler network than the target (see Appendix A.2).
The intrinsic lossis the mean squared error between f and the large pretrained network.
Like the respective Lang-NGUand LSE-NGU counterparts, text-based ND requires a language oracle, but image-based ND does not.
In Section 5.1 we compare against two additional methods to motivate why language is a usefulabstraction.
The ﬁrst, Lang-RND , is a variant in which the trainable network f L : O L ! R k maps theoracle caption to random features.
The intrinsic reward is the mean squared error between the outputsof f L and ﬁxed ˆ f L with random initialization.
Both f L and ˆ f L networks are of the same architecture.
The second method, language distillation ( LD ), is loosely inspired by RND in that the novelty signalcomes from a prediction error.
However, instead of learning to produce random features, the trainablenetwork learns to caption the visual state, i.e. f C : O V !
O L .
The network architecture consists ofa CNN encoder and LSTM decoder.
The intrinsic reward is the negative log-likelihood of the oraclecaption under the trainable model f C .
In LD, the exploration dynamics not only depend on howfrequently states are visited but also the alignment between language and the visual world.
We testwhether this caption-denoted alignment is necessary for directing semantic exploration by comparing
LD to a variant with shufﬂed image-language alignment ( S-LD ) in Section 5.1.
Experimental Setup
4.1 Environments
Previous exploration work benchmarked algorithms on video games, such as 2D grid-world MiniHackand Montezuma’s Revenge, or 3D ﬁrst-person shooter Vizdoom.
In this paper, we focus on ﬁrst-person
Unity-based 3D environments that are meant to mimic familiar scenes from the real world (Figure 2).
Playroom Our ﬁrst domain, Playroom [1, 50], is a randomly-generated house containing everydayhousehold items (e.g. bed, bathtub, tables, chairs, toys).
The agent’s action set consists of 46 discreteactions that involve locomotion primitives and object manipulation, such as holding and rotating.
We study two settings in Playroom.
In the ﬁrst setting, the agent is conﬁned to a single room with3-5 objects and is given a lift or put instruction.
At the start of an episode, the set of objects aresampled from a larger set of everyday objects (i.e. a candle, cup, hairdryer).
Object colors and sizesare also randomized, adding superﬁcial variance to different semantic categories.
The instructionstake the form: "Lift a <object>" or "Put a <object> on a {bed, tray}" .
With a liftgoal, the episode ends with reward 1 or 0 whenever any object is lifted.
With a put goal, the episodeends with reward 1 when the condition is fulﬁlled.
This setting tests spatial rearrangement skills.
In the second setting, the agent is placed in a house with 3-5 different rooms, and is given a findinstruction of the form "Find a {teddy bear, rubber duck}" .
Every episode, the house israndomly generated with the teddy and duck hidden amongst many objects, furniture, and decorations.
The target objects can appear in any room— either on the ﬂoor, on top of tables, or inside bookshelves.
The agent is randomly initialized and can travel throughout the house and freely rearrange objects.
The episode ends with reward 1 when the agent pauses in front of the desired object.
The find taskrequires navigation/search skills and tests the ability to ignore the numerous distractor objects.
City Our second domain, City, is an expansive, large-scale urban environment.
Each episode, a newmap is generated; shops, parks, and buildings are randomly arranged in city blocks.
Daylight is simulated, such that the episode starts during the morning and ends at nighttime.
The agent is randomlyinitialized and is instructed to “explore the city.”
It is trained to maximize its intrinsic reward andcan take the following actions: move_{forward,backward,left,right} , look_{left,right} ,and move_forward_and_look_{left,right} .
We divide up the map into a 32 ⇥ 32 grid and trackhow many unique bins are visited in an episode.
Additionally, City does not provide explicit visual or verbal signage to disambiguate locations.
Assuch, systematic exploration is needed to maximize coverage.
In contrast to Playroom, City testslong horizon exploration.
A Playroom episode lasts only 600 timesteps, whereas a City episode lasts11,250 and requires hundreds of timesteps to fully traverse the map even once.
The City covers a270-by-270 meter square area, which models a 2-by-2 grid of real world blocks.
4.2 Captioning Engine
We equip the environment with a language oracle that generates language descriptions of the scene,
O L , based on the Unity state, s (Figure 2).
In Playroom, the caption describes if and how the agentinteracts with objects and lists what is currently visible to it.
In City, O L generally describes theobject that the agent is directly looking at, but the captions alone do not disambiguate the agent’slocations.
Since these captions are generated from a Unity state, these descriptions may not be asvaried or rich as a human’s, but they can be generated accurately and reliably, and at scale.
4.3 Training Details
At test time, the agent receives image observation
O V and language-speciﬁed goal g .
The policynetwork never requires caption
O L to act.
During training, the exploration method calculates theintrinsic reward from O L or O V .
We show that language-based exploration is compatible with both policy gradient and Q-learningalgorithms.
We use Impala [15] on Playroom and R2D2 on City [23].
Q-learning is more suitable forthe City, because the action space is more restricted compared to the one needed for Playroom tasks.
For both environments, the agent architecture consists of an image ResNet encoder and a language
LSTM encoder that feed into a memory LSTM module.
The policy and value heads are MLPs thatreceive the memory state as input.
If the exploration method requires additional networks, such asthe trainable network in RND or inverse dynamics model in NGU, they do not share any parameterswith the policy or value networks.
Figure S2 is a visualization of an Impala agent that uses languageaugmented exploration.
Hyperparameters and additional details are found in Appendix A.
Results
5.1 Motivation: Language is a Meaningful Abstraction
[Figure]
The states are grouped together based on similarities in the visual random feature space and assigned a label.
Exploring in this shufﬂed space is less effective than exploring with the semanticallymeaningful abstractions shown in Figure 1.
We share the intuition with other work [e.g.32] that language can improve exploration.
Wedesign a set of experiments to show how andwhy this may be the case.
Our analysis follows the desiderata outlined by Burda et al.[5]—prediction-error exploration ought to use afeature space that ﬁlters irrelevant information(compact) and contains necessary information(sufﬁcient).
Burda et al.
[5] speciﬁcally studies RND and notes that the random feature space,the outputs of the random network, may not fullysatisfy either condition.
As such, we use the language variants of RND to frame this discussion.
We hypothesize that language abstractions areuseful, because they (1) create a coarser statespace and (2) divide the state space in a way thatmeaningfully aligns with the world (i.e. usingsemantics).
First, if language provides a coarserstate space, then the random feature space becomes more compact, leading to better exploration.
We compare Lang-RND to Vis-RND totest this claim.
Lang-RND learns the lift task33% faster and solves the put task as Vis-RNDstarts to learn (Figure 3a).
Second, we ask whether semantics – that is howlanguage divides up the state space – is criticalfor effective exploration.
We use LD to test this hypothesis, precisely because the exploration in LDis motivated by modeling the semantic relationship between language and vision.
We compare LD to a shufﬂed variant S-LD, where we replace the particular semantic state abstractionthat language offers with a statistically-matched randomized abstraction (Figure 4).
S-LD is similarto LD; the intrinsic reward is the prediction error of the captioning network.
However, instead oftargeting the language oracle output, the S-LD trainable network produces a different target captionf
O L that may not match the image.
f [Equation]
is produced by a ﬁxed, random mapping [Equation] f [Equation]
is constrained such that the marginal distributions P ( O L )
⇡ P ( f O L ) are matched under trajectoriesproduced by policy ⇡ LD .
See Appendix A.4 for full details on the construction of S-LD.
Thus, whereas the LD captions parcel up state space in a way that reﬂects the abstractions thatlanguage offers, the randomized mapping ˆ f S parcels up state space in a way that abstracts overrandom features of the visual space (Figure 4).
We control for the compactness and coarseness of theresulting representation by maintaining the same marginal distribution of captions.
If semantics is crucial for exploration, then we expect to see LD outperform S-LD.
This indeed holdsexperimentally (Figure 3b).
We can also view these results under the Burda et al.
[5] framework.
The S-LD abstractions group together visually similar, but semantically distinct states.
A single sampledcaption likely fails to capture the group in a manner that is representative of all the encompassing states.
In other words, ˆ f S produces a compact feature space that may not be sufﬁcient.
This may explainwhy S-LD learns faster than Vis-RND on the simpler lift task but fails on the more complex putand find tasks.
The S-LD experiments imply that language abstractions are helpful for explorationbecause they expose not only a more compact, but also a more semantically meaningful state space.
5.2 Pretrained Vision-Language Representations Improve Exploration
Having shown how language can be helpful for exploration, we now incorporate pretrained visionlanguage representations into NGU and RND to improve exploration.
Such representations (e.g. fromthe image encoder in CLIP/ALM) offer the beneﬁts of explicit language abstractions, without theneed to rely on a language oracle.
We also compare language-shaped representations to pretrained
ImageNet embeddings to isolate the effect of language.
To keep the number of experiments tractable,we only perform a full comparison on the Playroom tasks.
The groundtruth agent represents state in NGU as the global coordinate of the agent location. The dashed line indicates coverage of a uniform random policy. Error bars indicate standard error of the mean, overreplicas. See Appendix Table S4 for absolute coverage numbers.
City We ﬁrst compare how representations affect performance in a pure exploration setting.
With no extrinsic reward, the agent is motivatedsolely by the NGU intrinsic reward to explorethe City.
We report how many unique areas theagent visits in an episode in Figure 5.
Whileoptimizing coverage only requires knowledgeof an agent’s global location rather than genericscene understanding, vision-language representations are still useful simply because meaningful exploration is inherently semantic.
LangNGU, which uses text embeddings of O L , visitsan area up to 3 times larger.
LSE-NGU achieves2 times the coverage even without querying alanguage oracle (Appendix Figure S5).
Playroom We next show that pretrainedvision-language representations signiﬁcantlyspeed up learning across all Playroom tasks (Figure 6).
The LSE-NGU and Lang-NGU agentsimprove sample efﬁciency by 50-70% on thelift and put tasks and 18-38% on the findtask, depending on the pretraining model used.
The ND agents are signiﬁcantly faster than VisRND, learning 41% faster on the find task.
Wealso measure agent-object interactions.
Nearlyall LSE-NGU and Lang-NGU agents learn tofoveate on and hold objects within 40k learningupdates, whereas Vis-NGU agent takes at least60k updates to do so with the same frequency(Appendix Figure S7).
Although LSE-NGU and image-based ND agents do not access a language oracle, they are similarly effective as their annotation-dependent counterparts in the Playroom tasks (Appendix Figure S6), suggesting that our method could be robust to the availability of a language oracle. [Figure]
To demonstrate the value of rich language, we compare LSE-NGU agents to a control agent thatinstead uses pretrained ImageNet embeddings from a 70M NFNet [4].
ImageNet embeddingsoptimize for single-object classiﬁcation, so they confer some beneﬁt to the most object-focused tasks,lift and put .
However, ImageNet embeddings hurt exploration in the find task, where agentsencounters more complex scenes (Figure 6b).
By contrast, the language-shaped representations arewell-suited for not only describing simple objects, but also have capacity for multi-object, complexscenes.
Of course, current CLIP-style models can be further improved in their ability to understandmulti-object scenes, which may explain why the beneﬁts are less pronounced for the find task.
However, as the performance of pretrained vision-language models improve, we expect to see thosebeneﬁts transfer to this method and drive even better exploration.
Discussion
We have shown that language abstractions and pretrained vision-language representations improve thesample efﬁciency of existing exploration methods.
This beneﬁt is seen across on-policy and off-policyalgorithms (Impala and R2D2), different exploration methods (RND and NGU), different 3D domains(Playroom and City), and various task speciﬁcations (lifting/putting, searching, and intrinsicallymotivated navigation).
Furthermore, we carefully designed control experiments to understand howlanguage contributes to better exploration.
Our results are consistent with cognitive perspectives onhuman language—language is powerful because it groups together situations according to semanticsimilarity.
In terms of the desiderata that Burda et al.
[5] present, language is both compact and sufﬁcient.
Finally, we note that using pretrained vision-language representations to embed imageobservations enables more effective exploration even if language is not available during agent training.
This is vital for scaling to environments that do not have a language oracle or annotations.
Limitations and future directions
We highlight several avenues for extending our work.
First,additional research could provide a more comprehensive understanding of how language abstractionsaffect representations.
This could include comparing different types of captions offering varyinglevels of detail, or task-dependent descriptions.
Second, it would be useful to investigate how toimprove pretrained vision-language representations for exploration by ﬁnetuning on relevant datasets.
The semantics of the dataset could even be tailored to task-speciﬁc abstractions to increase thequality of the learnt representations.
Such approaches would potentially allow applying our methodto environments that are farther from the pretraining distribution, such as Atari.
Finally, we note alimitation with this approach is that current pretrained vision-language models are susceptible todomain shift on virtual environments and have been shown to be less effective on multi-object scenes.
Thus, future pretrained models that are larger or otherwise improved would presumably improve thequality of these representations and lead to even more effective exploration.