Pre-Trained Language Models for Interactive Decision-Making
Anonymous Author(s)
AffiliationAddress email
Abstract
Language model (LM) pre-training is useful in many language processing tasks.
But can pre-trained LMs be further leveraged for more general machine learning problems?
We propose an approach for using LMs to scaffold learning andgeneralization in general sequential decision-making problems.
In this approach,goals and observations are represented as a sequence of embeddings, and a policynetwork initialized with a pre-trained LM predicts the next action.
We demonstratethat this framework enables effective combinatorial generalization across different environments and supervisory modalities.
We begin by assuming access toa set of expert demonstrations, and show that initializing policies with LMs andfine-tuning them via behavior cloning improves task completion rates by 43.6%in the VirtualHome environment.
We then examine how our framework may beused in environments without pre-collected expert data.
To do this, we integratean active data gathering procedure into pre-trained LMs.
The agent iterativelylearns by interacting with the environment, relabeling the language goal of past“failed” experiences, and updating the policy in a self-supervised loop.
The active data gathering procedure also enables effective combinatorial generalization,outperforming the best baseline by 25.1%.
Finally, we explain these results byinvestigating three possible factors underlying the effectiveness of the LM-basedpolicy.
We find that sequential input representations (vs. fixed-dimensional featurevectors) and favorable weight initialization are both important for generalization.
Surprisingly, however, the format of the policy inputs encoding (e.g. as a naturallanguage string vs. an arbitrary sequential encoding) has little influence.
Together,these results suggest that language modeling induces representations that are usefulfor modeling not just language, but also goals and plans; these representations canaid learning and generalization even outside of language processing.
Introduction
Language models (LMs) play a key role in machine learning approaches to natural languageprocessing tasks [8].
This includes tasks that are not purely linguistic, and require nontrivial planningand reasoning capabilities [24, 12]: for example, instruction following, vision-language navigation,and visual question answering.
Indeed, some of these tasks are so distant from language modelingthat one can ask whether pre-trained LMs can be used as a general framework even for tasks thatinvolve no language at all.
If so, how might these capabilities be accessed in a model trained only toprocess and generate natural language strings?
In this paper, we study these questions through the lens of embodied decision-making , investigatingthe effectiveness of LM pre-training as a general framework for learning policies across a variety ofenvironments.
We propose LID , a framework that uses Pre-Trained L anguage Models for I nteractive
Language Model
Task-specific Model
Tokenization
Next action a t +
Observation o t History h t Goal g
Tokenization
Tokenization
F 𝜃
F 𝜃 F 𝜃
F 𝜃
F 𝜃 F 𝜃
F 𝜃
F 𝜃 F 𝜃
VirtualHome
BabyAI
Graph partial observation:
Goal predicates: Inside(pancake,stove):
Grid partial observation: Language goal:
Put the green box next to the purple box agent agent
D ecision-Making.
As shown in Figure 1 (right), we encode the inputs to a policy—includingobservations, goals, and history—as a sequence of embeddings.
These embeddings are passed to apolicy network initialized with the parameters of a pre-trained LM, which is fine-tuned to predictactions.
This framework is generic, accommodating goals and environment states represented asnatural language strings, image patches, or scene graphs.
We find that imitation learning using pre-trained LMs as policy initializers improves in-domainperformance and enables strong generalization over novel tasks.
For i.i.d.
training and evaluation tasks,this approach yields 20% more successful policies than other baseline methods in VirtualHome [29].
For combinatorial generalization to out-of-distribution tasks, i.e . tasks involving new combinationsof goals, states or objects, LM pre-training confers even more benefits: it improves task completionrates by 43.6% for tasks involving novel tasks (see Figure 3).
These results hold for a variety ofenvironment representations: encoding states as natural language strings, when possible, improvesthe data-efficiency of training, but even LMs fine-tuned on random environment encodings generalizecombinatorially to new goals and states when trained on large enough datasets.
We further examine how our method may be used in environments, where the expert data is notavailable and an agent must actively gather data from the surrounding environment.
To do this, weintegrate an A ctive D ata G athering ( ADG ) procedure into pretrained LMs as shown in Figure 2.
Theproposed method with ADG consists of three parts.
First, exploration collects trajectories using amix of random actions and actions generated by the current policy.
Exploration is insufficient inthis high dimensional problem and most of the trajectories will likely fail to achieve the end goal.
A key insight is that even the failed trajectories contain useful sub-trajectories that solve certainsub-goals, and we relabel their goal in the hindsight relabeling stage.
The relabeled goal describeswhat was achieved in the extracted sub-trajectory.
Policy update samples relabeled trajectories toupdate the policy.
The active data gathering procedure allows us to train the LM-policy withoutpre-collected expert data.
It also outperforms the Reinforcement Learning (RL) methods on embodieddecision-making tasks and enables more effective generalization to novel tasks.
We hypothesize and investigate three possible factors underlying the effectiveness of pre-trained
LM for generalization in policy learning: 1) input encoding scheme; 2) sequential input representations; and 3) favorable weights initialization.
We investigate 1) by encoding the policy inputs asdifferent types of sequences.
Different input encoding schemes have only a negligible impact on theperformance: the effectiveness of language modeling is not limited to utilizing natural strings, but infact extends to arbitrary sequential encodings.
We study 2) by encoding observations with a singlevector embedding, thereby removing its sequential structure.
This operation significantly degradesthe model’s performance on novel tasks.
Finally, we investigate 3) by learning the parameters of thepolicy from scratch.
The success rate after removing the pre-trained LM weights drops by 11 .
To summarize, our work has four main contributions.
• First, we propose to use pre-trained LMs as a general scaffold for interactive decision-making
across a variety of environments by converting all policy inputs into sequential data.• Second, the proposed method demonstrates that language modeling improves combinatorial
generalization in policy learning : initializing a policy with a pre-trained LM substantiallyimproves out-of-distribution performance on novel tasks.
• Third, we integrate an active data gathering procedure into the proposed approach to furtherenable policy learning on environments without using pre-collected expert
data.• Finally, we perform several analyses to explain the generalization capabilities of pre-trained LMs,finding that natural strings are not needed to benefit from LM pre-training, but the sequential inputencoding and weight pre-training are important.
These results point to the effectiveness of the proposed framework with pre-trained LMs as a generalpurpose framework to promote structured generalization in interactive decision-making.
Related Work
In recent years, word and sentence representations from pre-trained LMs [27, 8, 31] have becomeubiquitous in natural language processing [45, 28].
Some of the most successful applications ofpre-training lie at the boundary of natural language processing and other domains, as in instructionfollowing [12] and language-guided image retrieval [22].
Learning representations of language.
From nearly the earliest days of the field, natural languageprocessing researchers observed that representations of words derived from distributional statistics inlarge text corpora serve as useful features for downstream tasks [7, 10].
The earliest versions of theserepresentation learning schemes focused on isolated word forms [25, 26].
However, recent years haveseen a number of techniques for training (masked or autoregressive) language models to producecontextualized word representations (which incorporate information neighboring words in sentencesand paragraphs) via a variety of masked-word prediction objectives [8, 43].
Applications of pre-trained LMs.
LMs can be fine-tuned to perform language processing tasksother than language modeling by casting those tasks as word-prediction problems.
Successful usesof representations from pre-trained models include syntactic parsing [19] and language-to-codetranslation [41]; successful adaptations of LM prediction heads include machine translation [45],sentiment classification [5] and style transfer [18].
A number of tasks integrate language and othermodalities, including visual question answering and image captioning [44].
Recent works find thatimage representations can be injected directly into LMs’ embedding layers [39].
LM pre-training beyond language The possibility that LMs might encode non-linguistic informationuseful for other downstream tasks is suggested by a number of recent studies [34, 16, 21].
Severalrecent papers [33, 17, 14] consider questions related to the ones investigated here. [5] show that the
GPT-3 model is capable of performing a limited set of arithmetic and string manipulation tasks; [23]show that pre-trained LMs require very little fine-tuning to match the performance of task-specificmodels on several image classification and numerical sequence processing tasks.
[36] show that LMscan serve as an effective backbone for hierarchical policies that express plans as natural languagestrings [2, 4].
In this paper, we focus on building a general framework for decision-making tasksusing pre-trained LMs, even when language is not provided as an input or output.
Decision-Making and Language Modeling
3.1 POMDPs and Policy Learning
We explore the application of LMs to general sequential decision-making tasks in partially observedenvironments.
These tasks may be formalized as partially observable Markov decision processes(POMDPs).
A POMDP is defined by a set of states S , a set of observations O , a set of actions A ,and a transition model T ( s t +1 | s t , a t ) .
Importantly, in a POMDP setting, the observation o t onlycaptures a portion of the underlying state s t , and an optimal decision-making strategy (a policy ) mustincorporate both the current observation and the previous history of observations and actions.
In ourexperiments, policies are parametric models π ϕ ( a t +1 | g, h t , o t ) that select actions given the goals g ,history information h t , and partial observations o t of the current state s t .
In Figure 1 (right), we show a high-level overview of the proposed method.
We first convert all policyinputs into a sequence and provide them as input to a transformer encoder.
Representations from thisencoder model are then passed to a task-specific decoder that predicts actions.
We collect a dataset of
N training trajectories D
= { d 1 , · · · , d N } , where each individual trajectory consists of a goal and a 126 sequence of observations and actions: d i = { g i , o i 1 , a i 1 , · · · , o iT i , a iT i } , where T i is the length of thetrajectory.
We then train the policy to maximize the probability p ( { a i } Ni =1 |D ) of actions we want to 128 achieve a i = { a i 1 , . . .
, a iT i } across trajectories using the cross-entropy loss:
− ln p ( { a i } Ni =1 |D ) = −
N (cid:88) i =
T i (cid:88) t = ln p ( a it +1 |
g i , h it , o it ) .
(1) where h it represents the history information before time t .
3.2 Language models as policy initializers
Our experiments focus on autoregressive , transformer-based LMs [40].
These models are trainedto fit a distribution p ( y ) over a text sequence y =
{ y 1 , y 2 , . . .
y n } via the chain rule p ( y )
=
p ( y 1 ) (cid:81) ni
=2 p ( y i | y 1 , . . .
, y i − 1 ) .
Each term on the right hand side is parameterized by a transformernetwork, which accepts the conditioned tokens as input.
Each token passes through a learnedembedding layer F θ , then the full conditioned sequence is fed into the LM.
In our work, we use astandard LM, GPT-2, to process the input sequence rather than to predict future tokens.
Both POMDP decision-making and language modeling are naturally framed as sequence predictiontasks, where successive words or actions/observations are predicted based on a sequence of previouswords or actions/observations.
This suggests that pre-trained LMs can be used to initialize POMDPpolicies by fine-tuning them to model high-reward or expert trajectories, as described below.
Approach
We evaluate the effectiveness of pre-trained LMs in solving decision-making tasks across environments.
We use BabyAI [15] and VirtualHome [29] to evaluate the proposed method.
Whileboth environments feature complex goals, the nature of these goals, as well as the state and actionsequences that accomplish them, differ substantially across environments (Figure 1 (left)).
4.1 Policy Network
We first examine whether pre-trained LMs provide effective initializers when states and actionhistories are represented as natural language strings.
We encode the inputs to the policy—includingobservations, goals, and action histories—as sequences of words.
These word sequences are passedto the LM (using its pre-trained word embedding layer F θ ) and used to obtain contextualized tokenrepresentations.
These token representations are averaged and used to predict actions.
We design apolicy network following the general policy framework proposed in Figure 1.
Environment encodings in VirtualHome.
In VirtualHome, each goal consists of a sequence of predicates and multiplicities, and is translated into a templated English sentence ( e.g . “ Inside(apple,fridge):2 ” becomes “put two apples inside the fridge”).
To encode the agent’s partial observation,we extract a list of currently visible objects, their states ( e.g . “open, clean”), and 3D world coordinates.
We use a fully-connected layer to encode the 3D information and generate a feature representation ofeach object in the observation.
To encode history, we store information about all previous actions andconvert them into templated English sentences ( e.g .
“I have put the plate on the kitchen table and theapple inside the fridge”).
Environment encodings in BabyAI.
The observation by default is a 7 × 7 grid.
We convert theobservation into 7 × 7 text descriptions, e.g . “purple ball”, “grey wall”, “open door”, and combinethem into a long sentence.
We then convert the history actions into text descriptions, e.g .
“turn left”and “go forward”.
We combine the language instruction (without modification) with the observationand history text descriptions, and feed them to the pre-trained LM.
We note that the policy network described above does not strictly require that these encodings takethe form of natural language strings—other encodings of the environment as a sequence also work(see Section 7).
This framework could be also generalized to support pixel-based observations usingdiscretization schemes like the one employed in the Vision Transformer [9].
Action prediction.
We pool the outputs of the pre-trained LM into a “context feature” that is used topredict the next action.
In training, we maximize the probabilities of actions we want to achieve.
Ininference, we select the spaces action spaces
4.2 Data Gathering
We first examine LID through imitation learning on data collected by experts in Section 4.2.1.
Wethen show that integrating an active data gathering procedure into LID enables policy learning withoutusing expert data in Section 4.2.2.
We use VirtualHome as an example to explain the data gathering.
4.2.1 Policy Learning with Expert Data
The policy model is first initialized from a pre-trained LM and then fine-tuned on data collectedby experts.
We build on the VirtualHome environment to collect a set of expert trajectories usingregression planning [20] and create a VirtualHome-Imitation Learning dataset .
Given a taskdescribed by goal predicates, the planner generates an action sequence to accomplish this task (See
Appendix F.1 ).
The planner has access to privileged information, such as information about thepre-conditions and effects of each action, allowing an agent to robustly perform tasks in partiallyobservable environments and generate expert trajectories for training and evaluation.
4.2.2 Policy Learning with Active Data Gathering
[Figure]
Collecting expert data is sometimes challenging.
It may require privileged information of the environment or human annotations, which can be time-consuming anddifficult to scale.
A promising way to scaleup supervision is Hindsight Experience Replay (HER) [3], which allows agents tolearn from orders of magnitude more datawithout supervision
However, existing HER methods [ 11] focus on simple taskswith small state/action space and full observability.
They cannot tackle more complicated embodied decision-making tasks,requiring nontrivial planning and reasoningor natural language understanding.
LIDwith the active data gathering ( LID-ADG )can be used in solving tasks in such environments.
As shown in Figure 2, LID-ADG consists of three components, i.e . exploration , hindsight relabeling ,and policy update .
The key idea is to gradually improve the task success rate by asking the agentto iteratively explore the environment, relabel failure samples, and update its policy using imitationlearning.
In the exploration stage, we first randomly sample a goal g and an initial state s o .
Wethen use a mix of random actions and actions generated by the current policy π ϕ ( a t +1 | g, h t , o t ) toobtain the next action a t +1 .
We repeat this process until this episode ends.
We collect M trajectoriesand store them in the replay buffers.
The generated actions in the early stages can barely finish thegiven task.
However, even the failed trajectories contain useful sub-trajectories that solve certainsub-goals.
In the hindsight relabeling stage, we extract useful sub-trajectories and relabel a goal g ′for each of them.
We design a goal relabel function f l that generates a goal based on the sequence ofobservations and actions using hand-designed templates.
In practice, we implement the goal relabelfunction as a program (see Appendix F.2 ).
The hindsight relabeling stage allows sample-efficientlearning by reusing the failure cases.
During policy update , the agent samples the data from thereplay buffers and updates its policy network π ϕ .
By interleaving the exploration, hindsight relabeling, and policy update, LID-ADG can graduallyimprove the policy without requiring pre-collected expert data.
In embodied environments withlarge action spaces, sparse rewards, and long-horizon planning, RL methods often struggle to obtainstable policy gradients during training.
Our method enables sample-efficient learning from the sparserewards by relabeling new goals for the bad samples that the agent fails to achieve.
In addition,
LID-ADG leverages the stability of supervised learning in the policy update stage, enabling it tooutperform RL approaches on a wide range of decision-making tasks.
Experiment Setup
We evaluate the proposed method and baselines on VirtualHome and BabyAI.
[Figure] [Figure]
[Table]
5.1 VirtualHome
VirtualHome is a 3D embodied environment featuring partial observability, large action spaces, andlong time horizons.
We evaluate policies’ performance from three aspects: (1) performance onin-distribution tasks; (2) generalization to novel scenes; and (3) generalization to novel tasks.
In-Distribution.
The predicate types and their counts in the goal are randomly sampled from thesame distribution as the training data.
The objects are initially placed in the environment according tocommon-sense layouts; ( e.g . plates appear inside the kitchen cabinets rather than the bathtub).
Novel
Scenes.
The objects are placed in random positions in the initial environment without common-senseconstraints ( e.g . apples may appear inside the dishwasher).
Novel Tasks.
The components of all goalpredicates are never seen together during training ( e.g . both plates and fridge appear in training goals,but Inside(plate, fridge) only appears in the test set.
(See Appendix G for more details.)
We evaluate the success rates of different methods on each test set.
A given episode is scored assuccessful if the policy completes its entire goal within the maximum allowed steps of the environment.
On each of the 3 test subsets, we use 5 different random seeds and test 100 tasks under each seed.
Thus there are 1500 examples used to evaluate each model.
5.2 BabyAI
BabyAI is a 2D grid world environment for instruction following.
The observation in BabyAI isa 7 × 7 × 3 grid describing a partial and local egocentric view of the state of the environment.
We evaluate the methods on four representative tasks:
GoToRedBall , GoToLocal , PickupLoc , and
PutNextLocal .
Performing well on the test set requires the models to generalize to new environmentlayouts and goals, resulting in new combinations of tasks not seen in training.
For each method, wecompute its success rates over 500 episodes on each task.
Experiments
We first show results of the proposed method and baselines for embodied decision-making tasks usingexpert data in Section 6.1.
We then show our results when using actively gathered data in Section 6.2.
6.1 Embodied Decision Making with Pre-trained Language Model (LID)
6.1.1 Results on VirtualHome
We compare the proposed method with a variety of baselines using different policy architectures.
LID-Text (Ours) is the proposed method that converts all environments inputs into text descriptions.
The pre-trained LM is fine-tuned for decision-making (conditioned on goals, observations, andhistories) as described in Section 4.1.
Recurrent Network .
We compare our method with a recurrent baseline using an LSTM [13] toencode the history information.
The hidden representation from the last time step together with thegoal, and the current observation are used to predict the next action.
MLP and MLP-1 .
We perform additional comparisons with baselines that do not use the recurrentnetwork and pre-trained LMs.
MLP and MLP-1 take the goal, histories, and the current observation asinput and send them to the multilayer perceptron neural network (MLP) to predict actions.
MLP-1 has three more average-pooling layers than MLP that average the features of tokens in the goal, historyactions, and the current observation, respectively, before sending them to the MLP layer.
Quantitative results .
Each method is trained on 20 K demos from the VirtualHome-Imitation
Learning dataset, and then evaluated on the three test subsets: In-Distribution , Novel Scenes , and
Novel Tasks .
In Figure 3, LID -Text (Ours) , which initializes the policy with a pre-trained LM,has higher success rates than other methods.
This difference is most pronounced in the Novel
Tasks setting, where test tasks require combinatorial generalization across goals that are never seenduring training.
Here, LID -Text (Ours) dramatically ( 43 . 6% ) improves upon all baselines.
Suchcombinatorial generalization is necessary to construct general purpose agents, but is often difficult forexisting approaches.
Our results suggest that pre-trained LMs can serve as a computational backbonefor combinatorial generalization.
6.1.2 Results on BabyAI
We use the standard training and test data provided by [15].
In BabyAI, performing well on theunseen test tasks with new environment layouts and goals requires approaches having combinatorialreasoning capabilities.
In Table 1, we report the success rate of models trained on different numberof demos.
[15] is the method used in the original paper.
LID-Text (Ours) is the proposed methodthat converts policy inputs into a text sequence.
Given enough training data, i.e . 10K demos, bothmethods achieve high success rates, but LID -Text (Ours) outperforms [15] given fewer training data,indicating the proposed method can generalize well to novel language tasks using fewer training data.
6.2 Pre-trained Language Model with Active Data Gathering (LID-ADG)
We compare LID-ADG , the proposed LM framework for decision-making using actively gathereddata (Section 4.2.2), to a variety of baselines that do not use pre-collected expert data on VirtualHome.
Random.
The agent selects the next action randomly from the valid action space at that state.
Goal-Object.
The agent randomly selects an object that in the goal and in the valid action spaceto interact with.
For example, given a goal of “ Inside(apple, fridge) :1”, this baseline mightchoose “grab apple”, “open fridge”, or other actions containing “apple” or “fridge”.
Online RL.
Wecompare with PPO [35], one of the most commonly used online RL methods.
For fair comparison,we equip PPO with the same main policy network as the proposed method.
Our implementation isbased on Stable Baselines3 [32].
Hindsight Experience Replay.
We compare with DQN+HER usedin [3] and modify its main policy network to be the same as the proposed method.
Quantitative results .
We compare LID-ADG with baselines on VirtualHome in Table 2.
Eachexperiment is performed 5 times with different random seeds.
The Random baseline is always 0,indicating the tasks in VirtualHome cannot be easily solved by a random policy.
Goal-Object isbetter than Random because Goal-Object has access to objects in the goal and it samples actionsfrom a much smaller action space.
The online RL baseline, PPO , fails to solve tasks in VirtualHomefeatured by partially observation, large state/action space, and long-term horizon.
DQN+HERworks well on simple tasks on 2D environments, but they cannot tackle VirtualHome tasks neither,requiring nontrivial planning and reasoning.
LID-ADG does not require expert data and can solvethe complicated tasks in 3D embodied environments which cannot be easily achieved using RL.
Policy initializer and data provider.
LID-ADG can further be used to initialize the weights for finetuning RL policies and to gather data for offline learning.
As shown in Table 2, directly training RL,e.g . PPO, fails to solve tasks in VirtualHome.
However, after using the policy trained by LID-ADGto initialize the PPO policy, we may effectively learn an interactive policy with good performance.
In Table 3, PPO (Init from LID-ADG) is initialized from LID-ADG and further fine-tuned tosolve the tasks in VirtualHome.
After initialization, PPO improves its success rate by 53 .
7% on the In-Distribution setting (See PPO results in Table 2 and Table 3).
In addition, LID-ADG can providedata for offline learning.
LID-ADG saves the relabeled data in replay buffers.
We train Decision
Transformer (DT) [6] using the data collected by LID-ADG.
See DT (LID-ADG Data) in Table 3.
Analysis: Understanding the Sources of Generalization
The pre-trained LM policy, fine-tuned on either expert data or actively gathered data, exhibits effectivecombinatorial generalization.
Is this simply because LMs are effective models of relations between
[Table] [Table]
natural language descriptions of states and actions [1], or because they provide a more generalframework for combinatorial generalization in decision-making?
We hypothesize and investigatethree possible factors to understand the sources of such combinatorial generalization.
7.1 Input Encoding Scheme
We first hypothesize that converting environment inputs into natural language contributes to thecombinatorial generalization as the LMs are trained on language data.
We explore the role of naturallanguage by investigating three alternative ways of encoding policy inputs to our model without usingnatural language strings: two in VirtualHome, and one in BabyAI.
BabyAI results are in Appendix A.
[Table]
[Table]
Index encoding in VirtualHome.
Differentfrom LID -Text
(Ours) that convert policy inputsinto natural language strings, LID -Index (Ours)converts policy inputs into indexes.
LID -Index(Ours) retains the discrete, serial format of thegoal, history, and observation, but replaces eachword with an index, and the embedding layerfrom the pre-trained LM with a new embeddinglayer trained from scratch.
For example, grabapple is mapped to “5 3” based on the positionsof grab and apple in the vocabulary set.
Unnatural string encoding in VirtualHome.
LID -Unnatural
(Ours) replaces the natural languagetokens (e.g. converting the goal “ On(fork, table):1 ” as put one fork on the table ) with randomones (e.g. converting On(fork, table) as brought wise character trees fine yet ).
This is done byrandomly permuting the entire vocabulary, mapping each token to a new token.
Such a permutationbreaks the semantic information in natural strings.
LID -Index
(Ours) and LID -Unnatural
(Ours) have the same policy network as LID -Text (Ours) .
Allof them are fine-tuned on the expert data.
Each of them is trained on different numbers of demos.
Theaveraged results using 5 different random seeds on the Novel Tasks setting are reported in Table 4.
Given few training data, e.g. 100 demos, all the models perform poorly, with success rates lower than10% .
LID -Text (Ours) achieves higher success rates than LID -Index
(Ours) and LID -Unnatural(Ours) when the training data is increased, e.g .
LID -Text (Ours) is around 4% higher than LID -Index(Ours) and LID -Unnatural
(Ours) when there are 500 training demos.
However, when the trainingdataset is further enlarged, e.g . 20K demos, success rates of all approaches reach similar performance.
Such a result indicates that the effectiveness of pre-trained LMs in compositional generalization is notunique to natural language strings, but can be leveraged from arbitrary encodings, although adaptingthe model to arbitrary encodings may require more training data.
7.2 Sequential Input Representation
[Table]
Next, we explore whether the generalization iscaused by the sequential processing mechanisms intransformer-based LMs.
We investigate whether the
LM pre-trained policy will still be effective when theinput encoding is not sequential.
No-Seq uses thenon-sequential inputs.
It encodes the goal as a singlevector by averaging the features of all goal embeddings.
The history and observation features are obtained in the same way.
All the features are then sentto the pre-trained LM to predict actions.
As shown in
N ov e l
T a s k s
[Figure]
o T o L o c a l
[Figure]
I nD i s t
r i bu t i o n
[Figure]
Table 5, removing the sequential structure significantly hurts the performance on Novel Tasks .
No-Seqachieves good performance on test tasks that are closer to training tasks, but cannot generalize well tomore challenging unseen tasks.
Thus, the combinatorial generalization of pre-trained LMs may beattributed in part to the transformer’s ability to process sequential input representations effectively.
7.3 Favorable Weight Initialization
Finally, we investigate if the favorable weight initialization from LM pre-training enables effectivegeneralization of the proposed model.
No-Pretrain does not initialize the policy using the pre-trained
LM, but instead training the policy on the expert data from scratch.
In Table 5, we find that removingthe pre-trained weights can fit the in-domain data and thus performs well on the In-Distributionsetting.
However, its success rate is 11 .
2% lower than the proposed model on the Novel Tasks setting,indicating the pre-trained weights are important for effective generalization, but not necessary foreffective data fitting.
We further test a baseline, No-FT , that keeps the pre-trained weights of thelanguage model but freezes them while training the rest model on our expert data.
Freezing the pretrained weights without fine-tuning significantly hurts the performance on both settings, suggestingthat fine-tuning of the transformer weights is essential for effective combinatorial generalization.
Together, these results suggest that sequential input representations (vs. fixed-dimensional featurevectors) and favorable weight initialization are both important for generalization, however, theinput encoding schemes (e.g. as a natural language string vs. an arbitrary encoding scheme) haslittle influence.
These results point to the potential broader applicability of pre-trained LMs as acomputational backbone for compositional embodied decision making, where arbitrary inputs, suchas language, images, or grids, may be converted to sequential encodings.
Qualitative Results
In Figure 4, we show examples of LID -Text (Ours) completing tasks in VirtualHome and BabyAI.
We show two successful examples from VirtualHome on the In-Distribution and Novel Tasks settings,and two successful examples from BabyAI on solving the GoToLocal and PickupLoc tasks.
We onlyshow short trajectories or extract a sub-trajectory for saving space.
See Appendix B for more results.
Conclusion and Broader Impact
In this paper, we introduced LID, a general approach to sequential decision-making that convertsgoals, histories, and observations into sequences and processes them using a policy initialized witha pre-trained LM.
We integrated an active data gathering procedure into the proposed method toenable policy learning without using expert data.
Our analysis showed that input representation andfavorable weight initialization both contribute to the generalization while the input encoding schemehas little influence.
One drawback of the active data gathering is that it relies on hand-designed rulesfor task relabeling.
More generally, a potential disadvantage of the proposed approach is that biasesof the pre-trained LMs may influence its behavior, and further study of LID-based models’ bias isrequired before they may be deployed in sensitive downstream applications.
Nevertheless, our resultsdemonstrate that LID enables effective combinatorial generalization across different environments,and highlight the promise of LM pre-training for more general decision-making problems.
Appendix
In this appendix, we first show the convolutional encoding in BabyAI in Appendix A. We then providemore qualitative results in Appendix B.
We describe the environment details in Appendix C and theimplementation details of the proposed model in Appendix D.
We show the algorithm of interactiveevaluation in Section E and the data gathering procedure in Appendix F.
The goal predicates used in
VirtualHome test subsets are shown in Appendix G.
Finally, we visualize the attention weights inlanguage models in Appendix H.
A Convolutional encoding in BabyAI
In the main paper Section 7.1, we explore the role of natural language by investigating two alternativeways of encoding policy inputs in VirtualHome.
In this section, we show the third way of encodingpolicy inputs in BabyAI.
We test a new model, LID-Conv (Ours) , that converts environment inputs into convolutionalembeddings .
We pass the 7 × 7 × 3 grid observation in BabyAI to convolutional layers and obtaina 7 × 7 × d feature map, where d is the feature dimension.
We flatten the feature map and get asequence of features to describe the observation.
The rest of the model is the same as LID -Text(Ours) .
Table 6 shows the results of policies using the text encoding and convolutional encoding .
LID -Text (Ours) and LID -Conv
(Ours) have similar results given enough training data, but LID -Text(Ours) is slightly better when there are fewer training data.
This conclusion is coincident with theresults on VirtualHome.
Different input encoding schemes have only a negligible impact on model performance: the effectiveness of pre-training is not limited to utilizing natural strings, but in fact extends to arbitrary sequentialencodings.
[Table]
B More Qualitative Results
Failure case analysis.
In Figure 5, we show some failure cases of the proposed method.
We observed [Table] two main types of failure cases: grounding error and policy error.
For failures caused by the groundingerror, the agent interacts with a wrong object that is not related to the given goal, e.g .
the agent putscutlets instead of the salmon inside the fridge.
For failures caused by the policy error, the agentcannot find the target objects or does not interact with them.
The proposed method that convertspolicy inputs into sequential encodings and feeds them to the general LM framework can accomplishdecision-making tasks efficiently, however, there are still challenging tasks that the policy fails toaccomplish.
Larger LMs, e.g . GPT-3 [5], may improve the success rate of those challenging tasks.
Goal: Inside (chicken, oven): 1; Close (oven);
TurnON (oven) P o li c y
E rr o r
G r o und i n g
E rr o r
Goal: Inside (salmon, fridge): 1; Inside (sundae, fridge):
Walk to Stove
Open Stove
Walk to Bedroom
Walk to Cabinet
Grab Cutlets
Open Fridge
Put Cutlets inside the Fridge
[Figure]
to Bedroom
Goal
[Figure] [Figure]
kitchen table):
Goal
[Figure]
[Figure]
, bed )
[Figure]
C Environments
We use BabyAI [15] and VirtualHome [29] to evaluate the proposed method.
While both environments feature complex goals, the nature of these goals, as well as the state and action sequences thataccomplish them, differ substantially across environments.
C.1 VirtualHome
VirtualHome is a 3D realistic environment featuring partial observability, large action spaces, andlong time horizons.
It provides a set of realistic 3D homes and objects that can be manipulated toperform household organization tasks.
Goal Space.
For each task, we define the goal as a set of predicates and multiplicities.
For example,
Inside(apple, fridge):2 ; Inside(pancake, fridge):1 ; means “put two apples and onepancake inside the fridge”.
In each task, the initial environment (including initial object locations),the goal predicates, and their orders and multiplicities are randomly sampled.
There are 59 differenttypes of predicates in total.
Observation Space.
The observation in VirtualHome by default is a graph describing a list of objectsand their relations in the current partial observation.
Each object has an object name, a state, e.g .open, close, clean , and 3D coordinates.
Action Space.
Agents can navigate in the environment and interact with objects.
To interact with anobject, the agent must predict an action name and the index of the interested object, e.g . Open(5) toopening the object with index (5).
The agent can only interact with objects that are in the currentobservation or execute the navigation actions, such as Walk(bathroom) .
For some actions, suchas open , the agent must be close to the object.
There are also strict preconditions for actions, e.g .the agent must grab an object before it can put the object on a target position.
As a result of theseconstraints, the subset of actions available to the agent changes at every timestep.
We evaluate the success rates of different methods on VirtualHome.
A given episode is scored assuccessful if the policy completes its entire goal within T steps, where T = 70 is the maximumallowed steps of the environment.
C.2 BabyAI
BabyAI is a 2D grid world environment designed to evaluate instruction following.
Different from
VirtualHome, the observation in BabyAI by default is a grid describing a partial and local egocentricview of the state of the environment.
This grid shows 7 × 7 tiles in the direction the agent is facing.
Each tile contains at most one object, encoded using 3 integer values: one storing the type of object,one storing its color, and a state for doors indicating whether it is open, closed or locked.
The goalsin BabyAI are described using language instructions, e.g . “put the blue key next to the purple ball”.
BabyAI has 7 actions, e.g .
“turn left”, “pick up”, and “drop”.
[Table]
Pre-Trained Language Models for Interactive Decision-Making observation into a sequence of image patches as in Vision Transformer
(Dosovitskiy et al., 2020)
[Table]
We choose to convert the observations
[Table]
goal
[Table]
into
[Table]
and pass them
[Table]
pre
[Table]
language model.
We take the text tokens as an example
[Table]
explain the
[Table]
method
[Table]
Goal.
Each goal consists of a sequence of predicates and multiplicities, and is translated into a templated English sentence ( e.g . “ Inside(apple, fridge):2 ” becomes “put two apples inside the fridge”).
Observation.
To encode the agent’s partial observation, we extract a list of currently visible objects, their states ( e.g . “open, clean”), and3Dworldcoordinates.
Weuseafully-connectedlayerto encode the 3D information and generate a feature representation of each object in the observation.
See Appendix B.1 for more details.
History.
We store information about all previous actions and convert them into templated English sentences ( e.g .
“I have put the plate on the kitchen table and the apple inside the fridge”).
Action prediction.
VirtualHome features a large action space, and the set of valid actions changes as the agent moves through and interacts with the environment.
Each action consists of 1 verb and 1 object.
In training, we maximize the probabilities of the verb and object selected by an expert action.
To produce the verb probabilities, we pool the outputs of the pre-trained LM and pass this “context feature” through a fully-connected layer as shown in Figure 2.
To determine the object probabilities, the model computes the inner product of the context feature with the LM output vectors corresponding to each object in the observation.
This design automatically ﬁlters out objects that are not in the observation and hence cannot be interacted with.
In inference, we select the valid action with the highest joint probability 1 .
[Equation]
4.2. Policy Learning on BabyAI
Wefurtherevaluatetheeffectivenessoftheproposedmethodon BabyAI (Hui et al., 2020).
BabyAI contains program- matically generated natural language instructions, e.g. “put the green ball next to the box on your right” and requires the agent to navigate the world and move objects to target locations.
BabyAIprovidesdemosfor19tasksofincreasing difﬁculty for testing imitation learning and reinforcement learning algorithms.
In our experiments, we compare the
[Table]
constrained
[Table]
valid
[Table] [Table]
of
[Table]
product
[Table]
(of verb
[Table]
) is
[Table]
the mode of their joint distribution.
Tokenization
Context feature f c
History h t <walk> [kitchen] …<put> [apple] <in> [cabinet]
[Table]
entropy loss
[Table]
loss
Average Pooling
F ! F !
F !
Tokenization INSIDE(pancake, fridge):2 …ON(apple, table):1 Goal g
Tokenization agent kitchen apple fridge cabinet Partial observation o t
Object embeddings … … …
F ! F !
F !
F ! F !
FC FC
[Table]
(",$,%)
FC ("
[Figure]
5. Experiments 5.1. Experiments on VirtualHome 5.1.1. E VALUATION M ETRICS
We build three test sets that evaluate policies’ ability from three aspects: (1) performance on in-distribution tasks; (2) generalization to novel scenes; and (3) generalization to novel tasks.
(See Appendix E for more details.)
In-Distribution.
The predicate types and their counts are randomly sampled based on the same distribution as the training data.
There are 2 ⇠ 10 predicates in each task.
The objects are initially placed in the environment according to common-senselayouts; ( e.g .platesappearinsidethekitchen cabinets rather than the bathtub).
Even goal predicates are
[Figure]
D More implementation Details of the Proposed Method in VirtualHome
In Appendix D.1, we provide more details of the model architecture used in the main paper Section 4.1.
We then introduce the training detail in Appendix D.2.
D.1 Model
architecture details in VirtualHome
In this section, we provide more details of the policy network we used in VirtualHome.
Our policymodel consists of three parts, i.e . inputs, the pre-trained LM, and outputs.
As shown Figure 6, weencode the inputs to the policy—including goal g , history h t , and the current partial observation o t —as sequences of embeddings.
These embeddings are passed to the LM (using its pre-trained embeddinglayer F θ ) and used to obtain contextualized token representations.
These token representations areaveraged to generate a context feature f c , which is then passed to fully-connected layer to predict thenext action a t +1 .
The output action in VirtualHome consists of a verb and an object.
For brevity, weomit the time subscript t from now on.
In VirtualHome, the partial observation o of the environment state can be represented as a list ofobjects in the agent’s view.
We represent each object by its name, e.g . “oven”, a state description,e.g . “open, clean”, and position both in the world and relative to the agent.
In this part, we providemore details of how LID-Text (Ours) encodes the name, state, and position of each object in theobservation.
Figure 7 shows the model architecture we used to encode the observation.
Name encoding.
For each object node, we serialize its object name as an English phrase s o .
Weextract its tokens and features using the tokenizer and the embedding layer of the pre-trained LM,respectively.
Since one object name might generate several English tokens using the tokenizer fromthe pre-trained LM, e.g .
the tokens of “kitchencabinet” is [15813 , 6607 , 16212 , 500] , we take theaveraged features of all the tokens in the object name and obtain a “name” feature f o, name i for eachobject node as shown in Figure 7.
State encoding.
Some objects have a state description, e.g .
“oven: open, clean”.
There are six typesof object states in the environment: “clean”, “closed”, “off”, “on”, “open”, and “none”.
For eachobject node, we use a binary vector to represent its state.
Taking the “oven” as an example, if the oven f c
Observation o t History h t Goal g
F 𝜃 F 𝜃
FC FC
FC
Name (e.g. “kitchen oven”)
Statevector
Positionvector
FC FC
FC F 𝜃
FC
Token sequence
F 𝜃
Avg pool
Concatenate
For each object in o t
FC (𝑥,𝑦,𝑧) (𝑥,𝑦,𝑧)
Tokenization
ReLU is open and clean, its state vector would be [1 , 0 , 0 , 0 , 1 , 0] .
This state vector is then passed through afully-connected layer to generate a state feature f o, state i of object o i .
Position encoding.
To encode the position information of each object o i , we take their worldcoordinates { o i,x , o i,y , o i,z } and their spatial distance to the agent { a x , a y , a z } to generate a position 650 vector [ o i,x , o i,y , o i,z , o i,x − a x , o i,y − a y , o i,z − a z ] .
This position vector is then passed through twofully-connected layers with a ReLU layer in the middle to generate a position feature f o, position i ofobject o i .
The final feature f oi of each object node is obtained by passing the concatenation of its namefeature f o, name i , state feature f
o, state i , and position feature f o, position i through a fully-connect layer.
Theobservation at a single step can be written as a set of features { f o 1 , · · · , f oN } , where N is the number 656 of objects in the current observation.
D.2 Training details
Our proposed approach and baselines are trained on Tesla 32GB GPUs.
We train every single modelon 1 Tesla 32GB GPU.
All experiments used the AdamW optimizer with the learning rate of 10 − 5 .
We utilize a standard pre-trained language model, GPT-2, in our experiments.
GPT-2 is trained on the
Webtext dataset [31] using the Huggingface library [42].
E Interactive Evaluation
The algorithm for interactive evaluation is shown in Algorithm 1.
F Data Gathering Details in VirtualHome
In this section, we provide more data gathering details in VirtualHome for training the decisionmaking policies.
We introduce the expert data collection and active data gathering in Appendix F.1and Appendix F.2, respectively.
F.1 Expert Data Collection
VirtualHome-Imitation Learning Dataset.
To train the models, we collect a set of expert trajectoriesin VirtualHome using regression planning (RP) [20].
We follow the implementation of the regressionplanner used in [30].
Given a task described by goal predicates, the planner generates an actionsequence to accomplish this task.
As shown in Figure 8, the agent has a belief about the environment,i.e . an imagined distribution of object locations.
As the agent explores the environment, its beliefof the world becomes closer to the real world.
At every step, the agent updates its belief based onthe latest observation (see [30]), finds a new plan using the regression planner, and executes the firstaction of the plan.
If the subtask (described by the goal predicate) has been finished, the agent willselect a new unfinished subtask, otherwise, the agent will keep doing this subtask until it finishes.
Similarly to previous work [38, 37, 30], we generate training data using a planner that has accessto privileged information, such as full observation of the environment and information about thepre-conditions and effects of each action.
The planner allows an agent to robustly perform tasks inpartially observable environments and generate expert trajectories for training and evaluation.
Wegenerate 20 , 000 trajectories for training and 3 , 000 trajectories for validation.
Each trajectory has agoal, an action sequence, and the corresponding observations after executing each action. [Equation] [Figure]
Given: a goal relabel function f l ;
Initialize: policy π ϕ ; goal set G ; training replay buffer R train = {} ; validation replay buffer R val = {} ; for iteration=1, N do for example=1, M do Sample a goal g from G and an initial state s o ; for t = 0 , T do Sample an action from policy π ϕ ( a t +1 | g, h t , o t ) or sample an action randomly;
Execute a t +1 and obtain a new observation o t +1 ; end
Store the trajectory ( o 0 , a 0 , · · · , o T , g ) in the replay buffer R train or R val ; end
Relabel each failure trajectory d = ( o 0 , a 0 , · · · , o T ) in the replay buffers and get new goal g ′ =
f l ( d ) ; Put new goals g ′ in the goal set G ; for k = 1 , K do repeat Sample data from R train and update policy π ϕ ; until training episode ends ; Get validation accuracy using data from
R val ; end π ϕ =
π val_best end
F.2 Active Data Gathering
The algorithm for active data gathering is shown in Algorithm 2.
The hindsight relabeling stage isthe key component for active data gathering.
In this part, we provide more implementation details ofhow we relabel “failed” trajectories with new goals in the hindsight relabeling stage.
For each “failed” trajectory, we extract its useful sub-trajectories and relabel a task goal g ′ for it.
We design a goal relabel function f l that generates a goal based on the sequence of observationsand actions.
To do this, we first use a hand-designed program to detect what tasks are contained ina “failed” trajectory.
This program find useful tasks based on the keywords in the action list.
Forexample in Figure 9, the program knows the trajectory containing a task of “ On(apple, kitchentable) : 1” based on the action “ [ put ] < apple >< kitchentable > ”.
The selected sub-trajectories are not always optimal.
We thus design a rule to filter out bad trajectories,i.e . for trajectories with the same goal, selecting the “shorter” ones.
One example is shown in
Figure 10.
Suppose that there are two trajectories having the same goal, e.g . “ On(apple, kitchentable) : 1”.
The first trajectory has actions that are redundant or not related to the task, such as“ [ walk ] < bathroom > ” and “ [ walk ] < kitchen > ” while the second trajectory is more optimalgiven the goal.
We select the second trajectory and delete the first trajectory from the replay buffer.
Note that the “shorter” does not mean fewer actions, but fewer actions that are not related to the task.
The hindsight relabeling stage allows sample-efficient learning by reusing the failure cases.
Therelabeled data are used to train policies in the policy update stage.
G Test Sets in VirtualHome
In this section, we provide more details of each test set.
We first introduce the test sets used forevaluating the proposed model trained on expert data, i.e . LID, in Section G.1.
We then show the testsets used for evaluating the proposed model with active data gathering, i.e . LID-ADG, in Section G.2.
G. LID Test Sets
In Section 6.1, we compared the proposed method and baselines trained on expert data.
In Table 7,we provide a detailed description of each test subset, including the count of goal predicate types andthe number of goal predicates in each task.
The In-Distribution setting has 37 goal predicates intotal and each task has 2 ∼ 10 goal predicates.
The tasks are drawn from the same distribution as the [Figure]
Useful sub-trajectory for hindsight relabeling
[Figure]
Action generated by the current policy or random exploration:
[Figure] [Figure] training tasks. The Novel Scenes setting also has 37 goal predicates and each task has 2 ∼ 10 goalpredicates.
The objects are randomly placed in the initial environment.
The Novel Tasks setting has22 goal predicates in total and each task has 2 ∼ 8 goal predicates.
The tasks are never seen duringtraining.
G. LID-ADG Test Sets
As we have mentioned in the main paper Section 9, one limitation of active data gathering is that itrelies on hand-designed rules for task relabeling.
In addition, it is sometimes challenging to defineeffective rules to extract useful sub-trajectories and get high-quality hindsight labels, especially whentrajectories are long and tasks become more complex.
Thus we only relabel short sub-trajectories,where the goal consists of a single goal predicate, e.g . “ On(apple, kitchen table) :1”.
Duringtesting, we evaluate the success rate of approaches on solving such tasks as well, i.e .
the count of thegoal predicate equals to 1.
The types of goal predicates are still the same as Section G.1, i.e . 37 goalpredicates in the In-Distribution setting and the Novel Scenes setting, and 22 goal predicates in the
Novel Tasks setting.
[Table]
H Visualization of Attention Weights
To better understand how does LM pre-trained policies make decisions, we visualize the attentionweights from the self-attention layers of GPT-2 [40] in Figure 11 and Figure 12.
We show theattention weights from the input to the output of LID-Text (Ours) .
The order of tokens in the inputand ouput is observation, goal, and history.
Figure 11 illustrates the attention weights of a layer named “Head 3 Layer 2”.
We show attentionweights on two different tasks.
We find that “Head 3 Layer 2” can capture objects in the goalpredicates, such as “wineglass” and “cutleryfork” in the left figure, and “pancake” and “chicken” inthe right figure (the figures are cropped for visualization).
Figure 12 illustrates the attention weights of layers named “Head 1 Layer 2” (left) and “Head 4 Layer11” (right).
Given the goal predicates, history, and the current observation, the policy predict the nextaction as “grab milk”.
We find that “Head 1 Layer 2” is able to capture objects in the goal predicates,such as “milk”, “pancake”, and “chicken” while “Head 4 Layer 11” focuses on the interacted objectin the predicted action, such as “milk”.
The attention weights from different self-attention layers are significantly different—some selfattention layers assign high attention weight to objects in the goal predicates while some layers focuson the interacted object.
There are also some layers that do not have interpretable meanings.
Theattention weights just provide us an intuition of how does the internal language model works, morequantified results are reported in the main paper.
[Figure]
[Figure]