Learning Language-Conditioned Robot Behavior from Ofﬂine Data and Crowd-Sourced Annotation
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We study the problem of learning a range of vision-based manipulationtasks from a large ofﬂine dataset of robot interaction.
In order to accomplish this,humans need easy and effective ways of specifying tasks to the robot.
Goal imagesare one popular form of task speciﬁcation, as they are already grounded in therobot’s observation space.
However, goal images also have a number of drawbacks:they are inconvenient for humans to provide, they can over-specify the desiredbehavior leading to a sparse reward signal, or under-specify task information inthe case of non-goal reaching tasks.
Natural language provides a convenient andﬂexible alternative for task speciﬁcation, but comes with the challenge of ground- ing language in the robot’s observation space.
To scalably learn this groundingwe propose to leverage ofﬂine robot datasets (including highly sub-optimal, autonomously collected data) with crowd-sourced natural language labels .
With thisdata, we learn a simple classiﬁer which predicts if a change in state completesa language instruction.
This provides a language-conditioned reward functionthat can then be used for ofﬂine multi-task RL.
In our experiments, we ﬁnd thaton language-conditioned manipulation tasks our approach outperforms both goalimage speciﬁcations and language conditioned imitation techniques by more than25%, and is able to perform visuomotor tasks from natural language, such as “openthe right drawer” and “move the stapler”, on a Franka Emika Panda robot.
Introduction
[Figure]
We are motivated by the goal of generalist robotswhich can be commanded to complete a diverse rangeof manipulation tasks.
Doing so requires humans tobe able to effectively specify tasks for the robot tosolve.
One popular approach to task speciﬁcation isthrough goal-states, which by deﬁnition are groundedin the robot’s observation space, making them a natural choice for self-supervised techniques [1, 2, 3, 4].
However, goal-state speciﬁcation comes with a number of drawbacks, including (a) human effort requiredin generating a goal state to provide the robot, (b) taskover-speciﬁcation resulting in a sparse reward signal(e.g.
a goal image for the task of pushing a singleobject also speciﬁes positions of all other objects andthe robot itself), and (c) task under-speciﬁcation (e.g.moving to the right indeﬁnitely).
Natural languagepresents a promising alternative form of speciﬁcation,providing an easy way for humans to communicatetasks.
Moreover, natural language can ﬂexibly represent non-goal reaching tasks and tasks with varyingdegrees of speciﬁcity, such as grasping any markerfrom a desk with several markers, while a single goal image could only capture one instance of success.
To this end, we study the problem of learninglanguage-conditioned visuomotor manipulation skills from ofﬂine datasets of robotic interaction.
Despite the abundant beneﬁts of being able to command robots with natural language, such agentshave remained out of reach.
A major challenge in acquiring such agents is that the languageinstructions need to be grounded in the agent’s high-dimensional observation space.
Learning thisgrounding is difﬁcult, and requires diverse interaction data paired with language annotations.
Recentworks have made progress towards learning such grounding by annotating data collected by humans[5, 6, 7]; however, collecting many human teleoperated trajectories on real robots can be costly andtime consuming, and is thus difﬁcult to scale to a broad set of language conditioned behaviors.
Our key insight is that a practical and scalable way to ground language is to combine autonomouslycollected ofﬂine datasets of robotic interaction with post-hoc crowd-sourced natural language labels.
Unlike prior work, we do not assume this data comes from a human expert or contains optimalactions, allowing us to leverage a wide range of data sources such as autonomous exploration data(e.g. random, scripted, intrinsically motivated), replay buffers of trained RL agents, human expertdata (e.g. demonstrations, human play), and data without action labels.
Given such pre-collected data,we can then use crowd-sourcing to scalably label trajectories with natural language labels describingthe behaviors in the data.
To learn from this sub-optimal data with noisy annotations, we learn aclassiﬁer which takes as input a natural language instruction and an initial and ﬁnal image, andpredicts whether or not the transition completes the instruction.
This learned classiﬁer can then beused as a language-conditioned reward for ofﬂine RL to learn language-conditioned behaviors.
Concretely, in this work we propose to learn language conditioned skills from vision using suboptimal, autonomously-collected ofﬂine data and crowd-sourced annotations (See Figure 1).
Wepresent a simple technique to learn language-conditioned rewards from this data, which we call
L anguage-conditioned O fﬂine R eward L earning (LORL), and combine it with visual modelpredictive control to complete language conditioned tasks (See Figure 2).
In our experiments insimulation, we observe that even with data collected by a random policy our proposed method solveslanguage-conditioned object manipulation tasks 25% more effectively than language conditionedimitation learning techniques, as well as ⇠ 30% more effectively than goal-image conditioned comparisons which over-specify the task.
Additionally, we observe that by virtue of leveraging pretrainedlanguage models our learned reward is capable of generalizing from scripted language instructions tounseen natural language zero-shot, suggesting that knowledge in pretrained language models canenable more efﬁcient learning of grounded language as observed in prior work [6, 8].
Finally, weleverage an existing real robot dataset of sub-optimal data, label the dataset using crowd-sourcing,and use it to complete ﬁve visuomotor tasks speciﬁed by natural language, such as “open the rightdrawer” or “move the stapler” on a real Franka Emika Panda robot.
Related Work
There is a rich literature of work which studies interactive agents, and grounding their behaviors inlanguage [9, 10, 11, 12].
Many prior works have studied this problem in the context of instructionfollowing , where an agent aims to complete a task speciﬁed by formal language/programs [13, 14, 15,16, 17, 18] or natural language [10, 11, 19, 20].
While these approaches have been largely studied insimulated spatial games [19, 21, 22, 23] or in object-directed visual navigation in simulated robots[24, 25, 26, 27, 28, 29, 23] some of which include high-level object interaction [30], in this work wefocus on the domain of learning control for vision-based robotic manipulation .
Early works have approached instruction following with strategies like semantic parsing mapped tomotion primitives or pre-deﬁned actions to execute tasks in virtual domains [31, 32, 33, 34] and onmobile robots [35, 36].
Like our approach these methods don’t require expert demos; however unlikethese approaches, we directly learn robotic control from images and natural language instructions,and don’t assume any predeﬁned motion primitives.
More recently, end-to-end deep learning hasbeen used to condition agents on natural language instructions [37, 24, 38, 27, 39, 6, 7, 40], whichare then trained under an imitation and/or reinforcement learning objective.
In the reinforcementlearning setting, works have adopted a range of strategies, from language-conditioned reinforcementlearning while leveraging environment rewards [21, 41, 42, 43, 44, 45] to using language as areward bonus to densify the environment reward and aid in exploration [46, 47, 48, 49, 50].
Incontrast, we do not assume any environment provided reward signal, and rather aim to learn effectivelanguage-conditioned rewards from annotated data of interaction.
[Figure]
Numerous prior works have also studied learning language conditioned rewards online from demon- [Figure] strations or examples of successful completion of tasks and language annotations [5, 22, 39, 49].
[Figure] These works then use the learned reward to optimize policies through online RL, often using the [Figure] agents own online experience to train the reward [22, 51].
While these works also learn language- [Figure] conditioned rewards, and in some cases also use discriminative techniques to learn the reward [22, 49], [Figure]
running language-conditioned online RL on a physical robot can be prohibitively time consuming. [Figure] Our work aims to learn language-conditioned behaviors from entirely ofﬂine datasets (which may behighly sub-optimal), making it feasible to learn language-conditioned behaviors on real robots.
Other works have studied using ofﬂine data in the form of demonstrations [7] or human teleoperatedtrajectories (i.e. “play data”) [6], to learn language-conditioned robotic agents in simulation.
Mostrelated is Lynch and Sermanet [6] who also use crowd-sourcing to annotate play data with naturallanguage instructions.
Critically, these works treat the ofﬂine data as near optimal, to the extentthat behavior cloning techniques can be used to learn language-conditioned policies.
Unlike theseworks, we don’t make any assumptions about the optimality of the actions in the collected data,allowing us to learn from arbitrary ofﬂine datasets, including autonomously collected data, whichcan be considerably easier to collect at scale on a real robot than human tele-operation data [52, 53].
Moreover, we observe in Section 5.1 that our proposed approach outperforms imitation learningtechniques on such data, and in Section 5.3 that our method is effective on a real robot.
Many prior works have studied how robots can learn to complete a wide range of tasks from vision.
While many approaches have been taken to task-speciﬁcation, including task IDs [54, 55], robot andhuman demonstrations [56, 57, 58], and meta-learning from rewards [59], a common approach is goalconditioned learning [60, 61, 2, 1], where an agent learns to reach particular goal states or distributions[62].
Many approaches have been applied to this domain, ranging from goal-conditioned modelfree learning [2, 63, 55, 64] with goal relabeling [61], model-based planning with a learned visualdynamics model [65, 66], to methods which combine the both [67].
Unlike these works, the focus ofthis work is multi-task visuomotor learning from natural language speciﬁcations.
Furthermore, weﬁnd in Section 5.1 that using our language-conditioned reward we can more effectively completetasks than leveraging a goal image speciﬁcation, while requiring less human effort to specify the task.
Preliminaries
In this work we consider an interactive agent which aims to complete K tasks {T k } K 1 ⇢ T where T 128 denotes the space of all tasks.
For each task T i 2 T the agent operates in a Markov decision process 129 MDP
M = ( S , A , p, R i , T ) where S is the state space (in our case RGB images), A is the robot’s 130 action space, p ( s t +1 | s t , a t ) is the robot environment’s stochastic dynamics,
R i : S ⇥ S ! { 0 , 1 } 131 indicates the binary reward at state s for completing task
T i from initial state s 0 , and T is the episodehorizon.
Lastly, let L denote the set of all natural language, and let L i ⇢ L denote the set of language instructions which describe task i .
Note that there can exist many instructions l 2 L i which describea task T
i (e.g. “pick up the blue marker” and “grasp and lift the blue marker”), and any particularinstruction l 2 L can describe multiple tasks (e.g. “pick up the marker” can describe the task ofpicking up the blue marker or the green marker).
In this work we assume that the true reward function R i for each task T i is unobserved, and mustbe inferred from natural language.
Concretely, we assume access to an ofﬂine dataset D of Ntrajectories, where each trajectory ⌧ n is a tuple containing a sequence of states and actions, and asingle natural language instruction ⌧ n =
([( s 0 , a 0 ) , ( s 1 , a 1 ) , ..., ( s T )] , l n ) .
We assume that l n 2 L ifor at least one task T i 2 T for which R i ( s 0 , s T ) = 1 .
Note T i does not need to be in {T k } K 1 , 142 meaning the ofﬂine data/annotations can consist of tasks unrelated to the robot’s target tasks (e.g.“doing nothing”).
Our goal then is to learn a parametrized reward model R ✓ : S ⇥ S ⇥ L ! [0 , 1]which conditioned on a language instruction l , initial state s 0 , and state s infers the true rewardfunction R i ( s 0 , s ) for some task T i for which l 2 L i .
Given R ✓ , we aim to instantiate a stochastic 146 language-conditioned policy ⇡ : S ⇥ S ⇥ L ! A , which conditioned on a natural language instructionl produces actions to maximize the expected sum of rewards P
Tt =
R i ( s 0 , s t ) for the inferred task T i .
148 Note that this formulation captures tasks that are reﬂected in a change of state, but not tasks whichare path dependent (e.g. “close the drawer slowly ”).
Language-conditioned Ofﬂine Reward Learning (LORL)
Now we describe how we go about learning our parametrized language-conditioned reward R ✓from D , as well as how we instantiate our language-conditioned policy ⇡ to maximize the learnedreward, also shown in Figure 2.
Our key idea is that while we cannot make assumptions about theoptimality of the behavior in D , we can use the the initial and ﬁnal states of each trajectory andprovided language annotations to ground what changes in state correspond to successful completionof language instructions.
Then to learn control we can leverage all of the data in D to learn a globaltask-agnostic model of the dynamics of the robots environment, which can be combined with thelearned reward via model-predictive control (MPC) to complete language conditioned tasks.
4.1 Learning the Reward Function
Given the provided dataset D =
[ ⌧ 1 , ..., ⌧ N ] of N trajectories ⌧ n = ([( s 0 , a 0 ) , ( s 1 , a 1 ) , ..., ( s T )] , l n ) ,how might we go about learning our reward function R ✓ ?
Critically, the behavior policy whichcollected this data could be sub-optimal.
Therefore, we cannot assume the optimality of any particularaction taken.
However, because the human provided annotations describe the task being completed inthe video, the assumption we can make about the data is that going from the start to the end of thetrajectory constitutes completion of l n .
Therefore, we implement our reward function as a binaryclassiﬁer R ✓ ( s 0 , s, l ) which looks at the initial state s 0 , current state s , and language instruction land predicts if going from the initial state to the current state satisﬁes the language instruction.
Training the reward function in this manner has numerous favorable properties.
First, unlike explicitlypredicting a single instance of a successful goal state for a language instruction or vice-versa, aclassiﬁer can easily capture the many-to-many mapping that exists between language instructionsand tasks.
Doing so allows it to capture the full space of successful behavior even in cases wherethere exists many possible language instructions l which can describe completing a task T i andmany possible pairs of initial and ﬁnal states ( s 0 , s T ) which can constitute successfully completingany given instruction l .
Second, by virtue of being context-dependent on the initial state, thereward function can be used in closed loop planning to perform behaviors indeﬁnitely withoutadditional speciﬁcation (i.e. the reward for “move right” is relative to the agent’s current position,so applying it iteratively will encourage the agent to continuously move right).
Lastly, unlikeother works which use classiﬁers for single-task reward learning [68, 69, 53] on robots, a languageconditioned reward classiﬁer can ﬂexibly represent many tasks with an easy to provide form oftask-speciﬁcation.
Concretely, we sample positive examples ( s 0 , s T , l t ) 2 ⌧ n ⇠ D from the 181 annotated dataset which constitute successfully completing an instruction and generate negativeexamples which don’t complete instructions ( s 0 0 , s 0 T , l 0 )
⇠ N also from D (described in detail in the 183 next sections).
We then minimize the binary cross entropy loss:
[Equation]
Positive Selection.
Selecting positive examples for the classiﬁer is straightforward, as we know theinitial and ﬁnal state in an episode labeled with instruction l satisfy that command.
However, it is
LORL
Label =
Label = Label =
[Figure]
LORL
LORL also highly likely that there are other states near the beginning and end of the episode for which theinstruction is satisﬁed.
Therefore we employ a noisy labeling scheme where we label any ( s i , s j , l )for which i  ↵ T and j  (1  ↵ ) T as a positive.
Higher values of ↵ may occasionally include falsepositives, but also signiﬁcantly increase the set of positives which can be used for training. Negative Selection. First, we choose initial and ﬁnal states from episodes with different languageinstructions as negatives. Speciﬁcally, for language command l , we select negatives s 0 0 , s 0 T byselecting any ( s 0 , s T , l 0 )
⇠ D where l 0 6 = l . Note that since there may be instructions l 0 6 = l which 193 describe the same task, this may occasionally yield false negatives, however like prior work [ 68 , 53 ] we ﬁnd that we can learn an effective reward despite noisy negatives.
Second, to encourage thereward to capture temporal progress (as opposed to focusing on spurious visual features), we alsoinclude the example ( s T , s 0 , l ) as a negative in training R ✓ .
Ultimately, we train on balanced batchesof positive examples and both types of negative examples (See Figure 3).
Data Augmentation.
Reward functions trained using classiﬁers have been shown to be prone toover-ﬁtting creating a sparse or incorrect reward signal [53].
This issue is further exacerbated bythe fact that we only have limited positive examples per episode.
To combat this, we use visual dataaugmentation in the form of afﬁne transformations and color jitter as well as uniform noise in theembedding space of language instructions to prevent classiﬁer over-ﬁtting.
Leveraging Pre-Trained Language Models.
Finally, learning the meaning of raw natural languagewhile simultaneously grounding the robot’s actions using only crowd-sourced data of a few thousandrobot episodes with language annotations poses a signiﬁcant challenge.
Therefore we leverage aﬁxed pre-trained distilbert sentence encoder [70], to encode the natural language commands into aﬁxed length vector in R 768 before they go into the classiﬁer.
We ﬁnd in Section 5.2 that by using thepre-trained model we can generalize to unseen natural language commands from synthetic data.
4.2 Learning Language Conditioned Policies with Visual Model Predictive Control
Once trained, the learned reward function R ✓ in principle could be used with any form of ofﬂinereinforcement learning to learn language-conditioned policies.
In this work, we aim to learn visuomotor control on real robots from large datasets of sub-optimal or even random ofﬂine data.
Modelbased RL techniques have been particularly effective in this endeavor [65, 58], and in our case allofﬂine data can be used to train a single task-agnostic visual dynamics model.
We then use this modelwith planning to maximize the learned language-conditioned reward R ✓ .
Speciﬁcally, we learn aforward visual dynamics model s t {++
 ⇠+} p ✓ ( s t , a t ) which is trained on the entire ofﬂine dataset D ,and does not use language annotations.
We leverage off-the-shelf action-conditioned video predictionframeworks for learning this model [71, 72], which we describe in detail in the supplement.
Given the learned dynamics model p ✓ and the learned reward function R ✓ , we then use modelpredictive control to instantiate a policy to complete language-conditioned tasks.
Speciﬁcally, givena language instruction l and initial state s 0 , we sample M different actions sequences of length H ,which we feed through p ✓ to get a predicted future state ˆ s mt + H .
For each prediction we computethe reward as R ✓ ( s 0 , ˆ s mt
+ H , l ) .
Action sequences are optimized to maximize reward using the crossentropy method (CEM) [73], until the best action sequence is applied in the environment (Figure 4). [Figure]
Experiments
[Figure]
In our experiments we aim to study three main questions.
(1) How does our proposed method forlearning language conditioned policies from ofﬂine data compare to both language-conditioned andgoal-image conditioned prior methods?
(2) By virtue of using pre-trained language models, to whatextent can our method generalize to unseen natural language commands?
(3) Can our method be usedto solve visuomotor tasks on a real robot using crowd-sourced annotations?
We study experiments (1)and (2) in simulation, and experiment (3) on a Franka Emika Panda robot positioned in front of a desk.
For qualitative results and videos, please see https://sites.google.com/view/robot-lorl .
[Figure]
We study our ﬁrst two experimental questionsin a simulated domain developed on top of the
Meta-World [54] environment, where a simulated sawyer robot interacts on a tabletop witha drawer, a faucet, and two mugs (see Figure 5(left)).
In this domain, we collect an ofﬂinedataset of 50,000 episodes by running a randompolicy in the environment, and label episodesprocedurally using the true environment stateyielding 2311 unique instructions (see Figure 5(right)).
After training on this data, we evaluateon 6 seen tasks which involve (1) closing thedrawer, (2) opening the drawer, (3) turning thefaucet left and (4) right, and (5) pushing the black mug right, and (6) pushing the white mug down.5.1 Does LORL enable effective language-conditioned behavior compared to prior work?
In this experiment we aim to evaluate how LORL compares to prior techniques for learning languageand goal image conditioned behavior on the 6 target tasks described previously.
Comparisons.
We compare LORL ( Ours ) to language-conditioned behavior-cloning ( LCBC ),which imitates the behavior in the ofﬂine dataset conditioned on the language instruction label, whichis reﬂective of prior works that use imitation learning to learn language-conditioned behavior [6, 7].
We also compare to language-conditioned RL ( LCRL ), which labels the ﬁnal state in each episodeas having reward 1 for the annotated language instruction and 0 elsewhere, and trains a languageconditioned-policy using ofﬂine Q learning, which reﬂects a fully ofﬂine version of the low-levelpolicy used in [41].
Furthermore, we compare to using a goal-image as the task speciﬁcation insteadof language, and provide the agent with a ground truth goal-image of the object in its desired position,with which we use either L2 pixel distance ( Pixel ) or LPIPS [74] simiarlity ( LPIPS ) as a planningcost, reﬂective of prior work in visual MPC [65, 75].
Finally, we include an ( Oracle ) which uses theground truth dynamics model and ground truth reward indicating the upper bound on the performanceof the CEM planner, as well as
[Figure]
Results.
Figure 6 shows the success rates over 3 seeds of 100 trials, ordered by legend.
We observeﬁrst that our proposed approach outperforms the next best method, language-conditioned behaviorcloning, by more than 25%.
By learning a language-conditioned reward and planning over it, therobot executes the tasks more effectively than what it observed in the data.
On the other hand, becausethe data is sub-optimal, language-conditioned imitation learning is only able to learn coarse directionsassociated with each task, and as a result fails on tasks that require more ﬁne-grained motion like“turn faucet left”.
Second, we observe that language-conditioned RL with a binary reward strugglesto learn at all, indicating the difﬁculty in jointly learning language-grounding and control.
Finally,we ﬁnd that using goal-images with a pixel cost also fails, performing comparably with a randompolicy.
We observe that the agent tries to match the arm position in the goal instead of the interactingwith the objects, highlighting the limitation of goal-images in their tendency to over-specify the task.
Using LPIPS similarity improves performance, however is still ⇠ 30% worse than our method.
5.2 Can LORL generalize zero-shot to unseen natural language commands?
[Table]
In our second experiment, we study our methods ability to generalize to unseen instructions by nature of using pre-trainedlanguage models.
Speciﬁcally, for thesix target tasks, we test our method witha rephrased instruction , which was completely unseen during training.
We evaluate task performance on the Original commands, on the commands with an Unseen
Verb (e.g. “turn faucet left” ! “rotatefaucet left”), on the command with an Unseen Noun (e.g. “move black mug right”! “move dark cup right”), and Unseen
Verb+Noun (e.g. “close drawer” ! “shutcabinet”).
Finally, we also test on Unseen Natural Language commands collected from 9 humanvolunteers who were asked to rephrase the command in a creative way, for example “turn faucetleft” ! “Spin nozzle left”.
The full set of unseen instructions for each task is in the supplement.
In Table 1, we see that on average when changing the verb or noun, we only see a drop in successrate of 5%, and when changing both or using human provided natural language, we see at most a 10%drop in success rate.
Furthermore, we compare performance with and without using the pre-trainedlanguage model, and observe that without the pre-trained language model performance is worse onseen instructions and drops signiﬁcantly more on unseen instructions (up to 23% vs 10%), suggestingthat the pre-trained language model is essential to learning and generalization, consistent with resultsin prior work on language-conditioned imitation [6].
This suggests that by using the knowledge inlarge un-grounded language models we can learn language groundings when grounded language datais limited, opening up the potential to use small datasets or entirely programmatic language to learngroundings which generalize to natural language.
5.3 Can LORL be used to learn language-conditioned visuomotor skills on a real robot?
Finally, we study the efﬁcacy of our method in learning language-conditioned behavior on a realrobot using sub-optimal ofﬂine data and crowd-sourced annotation.
We consider a Franka Emika
Panda robot mounted over an IKEA desk with two drawers and a cabinet, which can hold a range ofobjects (Figure 7).
The robot’s observation space consists of 4 camera viewpoints, each providing64 ⇥ 64 RGB images.
The robot’s action space is delta end-effector control.
Data and Annotation.
We use an ofﬂine 3000 episode (150000 frame) dataset withoutany modiﬁcation from concurrent work [76], which trains policies for different behaviorson the IKEA desk using online RL.
As a result, our dataset consists of diverse behaviors, but is also sub-optimal in that it comes from the replay buffer of a learning policywhich will often not complete any task or will complete tasks in highly sub-optimal ways.
[Figure]
To annotate the data, we leverage crowdsourcing, speciﬁcally Amazon Mechanical Turk.
We ask human annotators to describe the behavior, if any, that the robot is doing, and to phraseit as a command without any pre-speciﬁed template.
We collect 6000 annotations, two perepisode, containing a total of 1699 unique instructions, examples of which can be seen in Figure 7.
We ﬁlter out episodes for which annotators wrote the robot did nothing or indicatedthey could not understand what the robot wasdoing.
See supplement for details about the environment, data, tasks, distribution of annotations,and annotator interface.
Results.
We ﬁnd in Table 2 that, by training
LORL on this annotated dataset and using it forvisual MPC with a learned dynamics model in this domain, the robot can complete 5 languageconditioned skills with a 66% success rate on average across skills.
Additionally, we ﬁnd thatremoving negative training examples with the initial state and ﬁnal state ﬂipped ( LORL (-FN) )reduces performance by 30%, suggesting that such negatives are important for the reward to capturetemporal progress and prevent over-ﬁtting to objects.
[Table]
Finally, we test LORL’s robustness to morecomplex vocabulary and instruction length,by replacing the commands for openingthe left drawer and moving the stapler with“Open the small black and white draweron the left fully” and “Push the small graystapler around on top of the black desk”respectively.
We ﬁnd that LORL is stillable to succeed 7/10 and 5/10 times respectively, providing evidence that it is robustto instruction phrasing, consistent with theresults in Section 5.2.
Limitations and Future Work
We have presented LORL, a technique for learning language conditioned behavior from ofﬂine dataand crowd sourced annotation that is effective for visuomotor control on real robots and is capable ofgeneralizing to unseen language instructions.
However, a number of limitations still remain.
First, inits current form LORL can only capture states which are reﬂected in some change of state, but cannotcapture tasks which are path dependent (e.g. “move in a circle slowly”).
One exciting direction forfuture work to address this is to train LORL on full video clips.
Second, while in this work we havefocused on learning short-horizon skills from language, composing these skills to solve long-horizonlanguage-speciﬁed tasks is important for making robots useful in the real world.
Data sources withlong-horizon behaviors, and more powerful planners and visual dynamics models are necessary toenabling these longer horizon tasks.
Third, while we have presented language speciﬁcation as analternative to goal images, goal images maintain the beneﬁt of being self-supervised and in somecases can be effective task speciﬁcation.
Unifying both forms of speciﬁcation for robots would bea valuable future direction.
Finally, further investigation into how best to select negative examples(rather than randomly) could enable learning a more accurate reward.