LILA: Language-Informed Latent Actions
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We introduce Language-Informed Latent Actions (LILA), a frameworkfor learning natural language interfaces in the context of human-robot collaboration.
LILA falls under the shared autonomy paradigm: in addition to providingdiscrete language inputs, humans are given a low-dimensional controller – e.g., a2 degree-of-freedom (DoF) joystick that moves left/right or up/down – for operating the robot.
LILA learns to use language to modulate this controller, providingusers with a control space that adapts based on the provided language: given aninstruction like “place the cereal bowl on the tray,” LILA might learn a 2-DoF control space where one dimension controls the distance from the robot’s end-effectorto the bowl, and the other dimension controls the robot’s end-effector pose relativeto the grasp point on the bowl.
We evaluate LILA with real-world user studiespairing users with a 7-DoF Franka Emika Panda Arm, with the goal of completinga series of complex manipulation tasks with a lower dimensional control spaceconditioned on language.
LILA models are not only more sample efﬁcient andperformant than imitation learning and end-effector control baselines, but also arequalitatively preferred by users.
Introduction
Effective human-robot collaboration is rooted in the ability to strike the right balance between thecapabilities of human users and the power of robotics.
Finding this balance is important; nearly amillion American adults live with physical disabilities, requiring assistance for everyday tasks liketaking a bite of food or pouring a glass of milk [1].
One path forward is with shared autonomy[2, 3, 4], a framework that couples user input with automated robot assistance.
Shared autonomyapproaches tap into the complementary strengths of both humans and robots; humans continuouslyprovide inputs, while the robot deftly ingests a stream of rich and diverse observations, assistingthe human in fulﬁlling their desired objective.
Such approaches have boundless promise for interpretability, safety, and sample efﬁciency, amongst others.
Despite their promise, shared autonomyapproaches are missing rich, intuitive ways of specifying and disambiguating tasks – a bridge thatcan be built by tapping into the potential of natural language.
Consider a kitchen or dining setting where a high-dimensional (high-DoF) robot (such as awheelchair-mounted manipulator) aids a human user who may not be physically able to performtasks requiring ﬁne-grained manipulation.
While one option for the human is to manually teleoperate the arm by controlling individual “modes” (separate degrees-of-freedom) of the robot’s endeffector, past work has shown this to be unintuitive, slow, and frustrating [3, 5].
However, learnedlatent actions [6, 7] is a promising new paradigm that seeks to address the problem of buildingintuitive low-dimensional (low-DoF) controllers for high-DoF robot manipulators by learning taskspeciﬁc control spaces via dimensionality reduction.
While these approaches are reliable and sampleefﬁcient, they are limited by their single task nature.
Existing latent action controllers fail to disambiguate user intent in multi-task settings, proving unable to assist in realizing the human’s goal ahuman wishes to achieve.
u u
Learning Latent Actions
Controlling with Latent Actions
[Figure]
” u u
[Figure]
a s a z s a’
Encoder
Decoder
[Figure] “Grab [Figure]
u z s a’
Decoder
Language is an expressive way for humans to communicate intent.
A user can easily express theirdesire to “grab the cereal bowl” or their preference to “slowly pour the milk into the glass till it’shalf full” .
Prior work integrates language in robotics settings via full autonomy [8, 9, 10, 11, 12].
Unfortunately, these approaches are not silver bullets: they are often plagued with poor sampleefﬁciency, inability to recover from failure, and poor generalization to novel states – many issuesthat shared autonomy methods (including learned latent actions) seek to address.
We propose Language-Informed Latent Actions (LILA), a framework for incorporating languagewithin the broader paradigm of learned latent actions.
Key to LILA is the principle that languagemodulates a user’s low-level controller based on their provided utterance; given the utterance “grabthe cereal bowl” , our assistive robot exposes a semantically meaningful, low-dimensional (2-DoF)control space where one dimension (one joystick axis) may control the distance from the robot’send-effector to the cereal bowl, whereas the other might control the angle of the end-effector relativeto the bowl such that the robot’s gripper can obtain a solid grasp of the object.
Future utterancesmodulate the controller in similar ways – “pour the milk into the cup” might result in one joystickdimension controlling the angle at which the milk jug is tilted, whilst the other controls the height ofthe jug relative to the cup.
Language not only is a natural means for a human to communicate theirintent to the robot, but it helps disambiguate across a wide variety of tasks as well.
In addition to providing a formalism for integrating natural language within the learned latent actionsframework, and a robust, sample-efﬁcient model architecture that allows for reliable task execution,we perform a comprehensive user study across 10 users who use natural language and our learned LILA controllers to complete a variety of diverse manipulation tasks in a simpliﬁed assistive “buffet”setting.
Our results show that not only are LILA models more reliable, performant, and sampleefﬁcient than fully autonomous (imitation learning) and end-effector control baselines, but they arequalitatively preferred by users as well.
Related Work
We build LILA within a shared autonomy framework [2, 13], applied to assistive teleoperation[14, 15].
We additionally build off work at the intersection of language and robotics [16, 17, 18].
Shared Autonomy & Assistive Teleoperation.
Shared autonomy casts robot control as a collaborative process between humans and robots [2, 4, 13, 19].
While other work focuses on methodsthat actively assist users by “blending” human inputs with possibly task-agnostic policies [20, 21], inthis work, we focus on assistive teleoperation, where a user is provided a low-dimensional controller(e.g., a joystick, sip-and-puff device) to directly control a high-dimensional robot manipulator.
Using these controllers to control the end-effector via inverse kinematics or operational space control[22] is incredibly difﬁcult, requiring frequent mode-switching to control speciﬁc robot DoFs, failing to provide users with ﬁne-grained control [3, 23]. Instead, we adopt learned latent actions [Figure] “Put
[Figure]
banana away” [Figure] ↦ a ′ [6, 7, 24, 25] a recent framework that uses conditional auto-encoders [26] to learn task-speciﬁclatent action spaces from demonstrations.
These latent spaces match the dimensionality of the lowdimensional control interface, and provide intuitive, semantically meaningful control.
However,existing models fail to differentiate between tasks when presented with similar states, hindering theability to perform diverse behaviors in a workspace (e.g., manipulating a jug of milk in differentways – pouring, placing in the fridge, etc.).
In this work, we use language for disambiguation ; usersnaturally specify their intent with language, conditioning latent action models to produce intuitivecontrol spaces that align with user objectives.
Language-Informed Robotics.
A wealth of work looks at different ways to incorporate languagefor robotics, spanning approaches that map language to planning primitives [8, 27, 28, 29, 30], imitation learning from demonstrations paired with language instructions [10, 31, 32, 33], and reinforcement learning from language instructions paired with corresponding reward functions [11, 34, 35].
Other approaches use language in more nuanced ways, for example, to learn language-conditionalreward functions directly [36, 37, 38], or within more adaptive frameworks, where language is usedto correct or deﬁne new behavior [39, 40].
This list is not exhaustive, nor are the explanations complete; we present further discussion – including approaches that combine language with gesturesand other modalities – in the Supplemental.
However, all these approaches fall broadly under thecategory of full autonomy: after providing an instruction, human users cede control over to the robotpolicy, which then takes the actions necessary to perform a task.
While robots trained with these approaches can perform diverse tasks and generalize to new instructions, it is not without cost.
Paramount is sample efﬁciency; imitation learning approaches oftenrequire hundreds to thousands of demonstrations to learn navigation tasks [32, 41], and reinforcement learning approaches can require millions of episodes of experience to learn robust policies[11, 42].
Whereas coarse behaviors are easy to learn, learning to recover from slight deviations fromthe training data, or to perform precise motions in a sequence is incredibly difﬁcult.
By casting ourapproach LILA within a shared autonomy framework instead, we intelligently ofﬂoad these partsthat are harder for robots – but easier and intuitive for humans – onto the user, who provides thenecessary control inputs.
Framing Language for Assistive Teleoperation
Consider our running example (Fig. 1): you – who may not be physically able to perform actions likereaching or lifting – are equipped with a high-dimensional robot manipulator and a low-dimensionalcontroller (e.g., a 2-DoF joystick).
You have simple tastes – breakfast consists of a bowl of cerealwith milk.
How do you achieve this goal?
With LILA, you lead with language – “grab the cereal bowl” – which is fed to a learned modelthat modulates , or restructures, your control interface.
Pressing up on the joystick now moves therobot arm towards the cereal bowl, while pressing right aligns the end-effector angle to make thegrasp. You press a button closing the gripper around the cereal bowl, then hold down on the joystick
[Table]
the
[Table]
and
[Table]
each participant only attempted 2 tasks, can ﬂuctuate signiﬁcantly, as is to bring it back to your tray (e.g. blue trajectory in Fig. 1). You speak – “pour some milk into mycereal” and your controls adapt; holding down has new meaning, bringing the robot closer to the cupof milk and angles the end-effector to grasp it. Tilting right begins to pour the milk, and so on.
Formalism.
We formulate a user’s task (on a per-user basis ) as a fully-observable, languageaugmented, Markov Decision Process (MDP) M deﬁned by the tuple ( S , U , A , T, R, γ ) similarto prior work in language-conditional robotics [9, 31, 39].
Let u ∈ U denote a user’s language utterance provided at the start of each task, where U is the full set of language utterances a user couldprovide to a robot.
Let s ∈ S ⊂ R n be the robot’s state, and a ∈
A ⊂ R m be the robot’s action:taking action a in state s results in a next state s (cid:48)
according to the transition function T ( s, a ) .
Giventhe language utterance u , the user implicitly deﬁnes a reward function R ( s, a, u ) ∈ R ; the humanand robot collaboratively maximize this reward subject to discount factor γ ∈ [0 , 1] .
In this work, we experiment exclusively with the Franka Emika Panda robot arm, a 7-DoF manipulator.
States are taken to be the robot’s joint states ( n = 7 ) while actions are provided as continuousjoint velocities ( m = 7 ).
Language utterances u are typed into a text console, but future work couldeasily extend this to use off-the-shelf speech recognition systems.
Problem Statement.
This MDP forms the basis of a shared autonomy task wherein a human isequipped with a low-dimensional control interface for the robot.
Let z ∈ Z ⊂ R d where d << mbe the human’s control input to the robot, such as the d = 1 DoF controller in Fig.
Previous workon learning latent actions for assistive teleoperation [7, 25] learn a decoder φ ( z, s ) :
Z × S → Athat maps user low-dimensional inputs z ∈ Z and current state s ∈ S to a high-dimensional actiona ∈ A .
However, in situations such as the disambiguation point that may arise when task trajectoriesoverlap (Fig. 2), too low a control input dimension d may lead to collapse in the latent action space.
We aim to learn a decoder φ ( z, s, u ) :
Z × S × U →
A that incorporates the user’s control inputand initial language utterance u , and predicts the robot action a u that maximizes reward.
Thus, wepropose using language as an expressive conditioning variable to enable latent actions to be usefulin multi-task environments.
LILA – Language-Informed Latent Actions
LILA builds off the learned latent actions paradigm for assistive teleoperation.
Speciﬁcally, weextend the original conditional auto-encoder models (CAE) from [6] to incorporate language andaddress the disambiguation problem discussed above.
4.1 Extending Learned Latent Actions with Language
We assume a dataset of demonstrations associated with a corresponding language utterance.
We spliteach demonstration into triples ( s, u, a ) and use these to learn a conditional auto-encoder, consistingof a language-conditional encoder ψ : S ×
A × U →
Z that maps language and states to a latent zand a decoder φ : Z × S × U →
A that attempts to reconstruct the original action a .
We minimizethe mean-squared error between the predicted action and the original action:
[Equation]
Both the encoder ψ and decoder φ are neural networks optimized with stochastic gradient descent;we discuss how we integrate language (including the use of pretrained language models) in §4.2.
4.2 Integrating Language
Large, pretrained language models such as BERT, T5, and GPT-3 [43, 44, 45] have revolutionized NLP, providing language representations that capture sentence-level semantics.
Inspired by thesuccess of using these models for encoding language in the context of robotics and reinforcementlearning tasks [35, 46, 47], we take a similar approach.
For encoding language, we use a distilledvariant of the RoBERTa-Base model [48], from Sentence-Transformers [49, 50].
This modelis ﬁne-tuned on a large corpus of paraphrased sentences, allowing it to pick up on longer-term,sentence-level (utterance length) semantics.
We generate sentence embeddings by performingmean-pooling over token embeddings for an utterance, as is done in prior work [35, 46].
We incorporate these embeddings into our latent actions models using feature-wise linear modulation (FiLM)layers [51], which incorporates linguistic information into learned features h by mapping languageto parameters ( γ, β ) of an afﬁne transformation h (cid:48)
=
γ ∗ h + β that is fed to later layers of the network
Nearest-Neighbor Retrieval at Inference.
A major concern for work in language-conditionedrobotics is the ability to generalize to novel language inputs not seen at training.
While it may beunreasonable to expect generalization to completely new tasks, for user-facing systems with cleartask distribution (as in our work), there is an implicit expectation that any language-informed systemis capable of handling moderate variations or paraphrases of utterances from the training set.
To do this, adding linguistically diverse data has been the gold standard [41, 47]; however, a newclass of approaches have emerged that sidestep additional data requirements by tapping into thepotential of pretrained language models [40, 46].
These approaches frame language interpretationat inference (when interfacing with real users) as a retrieval problem: each new user utterance u (cid:48)is embedded (with the same pretrained model as above) and then used to query a nearest neighborsstore containing all training exemplars; once the nearest neighbor u i has been identiﬁed, it replacesu (cid:48) as an input to the LILA decoder.
The key beneﬁt of such an approach is that there is no mismatch between train and test languageinputs; all “test inputs” are drawn solely from the training set.
This does mean, however, that userutterances that describe new tasks, or are otherwise unachievable also get mapped to language seenat training.
While this limits generalization, it again highlights the beneﬁts of operating within theshared autonomy paradigm (doubly so, considering the cost of a mistake in an assistive domain likethe one we consider in this work): if, while providing control inputs to the robot, a user feels therobot is not acting in alignment with the user’s desired objective, they can always stop execution.
User Study
We evaluate LILA with a real-world user study on a 7-DoF Franka Emika Panda Arm, on a seriesof complex manipulation tasks.
Each user is provided a 2-DoF joystick for control.
We compareagainst a non-learning, End-Effector (EE) control baseline where users “mode switch,” controllingthe velocity of 2 axes of the End-Effector pose at a given time – [(X, Y), (Z, Roll), (Pitch, Yaw)].
Wealso compare against a fully autonomous Imitation Learning (IL) strategy where users solely providelanguage inputs, and the robot attempts to perform a task without additional input. A baseline that isostensibly missing is a non-language variant of the latent actions model, to demonstrate the necessity [Figure] [Figure]
S u cc e ss R a t e
X Y 0. J e r k ( C o n t r o l I n p u t ) 1e
X
Y
Z Yaw Pitch Roll 0.
J e r k (
E n d E ff e c t o r
P o s i t i o n ) 1e of language.
However, upon evaluating this baseline, we found it to be unintuitive, and unable tomake progress on any task – showing another input such as language is necessary.
{+We explore thisfurther with experiments and formal arguments in the Supplemental. 
+}
Environment.
Fig.
1 shows our “breakfast buffet” setting, a scaled down version of an assistivefeeding domain.
We perform a within-subjects study where each user is assigned 2 of the following5 tasks to perform with each control method: 1) Place Banana : placing the banana in the purplefruit basket, 2) Place Basket : grasping the purple basket by the handles, and dropping it on thetray, 3) Place Bowl : grasping the green cereal bowl by its edge, and moving it to the tray, 4) Pour
Bowl : pouring the blue cup of marbles (a proxy for actual milk) into the cereal bowl (positioned onthe tray), and 5) Pour Cup : pouring the blue cup of marbles into the yellow coffee cup.
Fig.
1 showsidealized example trajectories for the Place Bowl (blue) and Place Banana (orange) tasks.
These tasks vary in difﬁculty, with many requiring precise grasping and dexterous manipulation(e.g., pouring).
Quantitatively, we evaluate partial success based on how many of the following4 subtasks users are able to complete: 1) Reaching : touching the desired object, 2)
Grasping :executing a successful grasp, 3) Bring to Target : successfully transporting the manipulated objectto the target location, and 4) Task Completion : whether users are able to ﬁnish the given task(placing an object without excessive force, in an acceptable orientation, or pouring all marbles intothe desired receptacle).
Demonstration Collection.
Both LILA and IL models require learning from (language, demonstration) pairs for each of the 5 tasks.
We collect task demonstrations kinesthetically, by manuallybackdriving the robot arm, as in prior work [6, 25].
We trace out the trajectory necessary and recordjoint states at a ﬁxed frequency.
We initially collected 15 demonstrations per task for each method.
However, on manually testing the IL model, we found it incapable of performing even rudimentaryreaching behaviors.
To give IL the best chance, we collected twice the number of demonstrations(30 per task; 150 total), requiring an extra 2 hours of labor.
Further details, including videos of ourdemonstration collection process for each method, are included in the Supplemental.
Crowdsourcing Language Annotation.
To realize our goal of building a natural language interface for human-robot collaboration, we collect language annotations for each task by crowdsourcingutterances.
In order to capture the diverse ways users may refer to the objects and actions our tasksentail without any additional information (simulating a real user walking into our environment forthe ﬁrst time), we recruited 30 workers on Amazon Mechanical Turk and showed them a video ofa recorded task demonstration, asking them to provide “a short instruction that you would wantto provide the robot to complete this task independently in the box below.” .
However, this procedure alone resulted in some annotations containing “spam”, noise, or extremely out-of-domain text.
To address this issue systematically, without introducing our own bias, we ﬁlter data to identifyworkers who provided “atypical” annotations, deﬁned by measuring the cosine distance betweenthe sentence embedding (per the pretrained language model from §4.2) of an annotator’s providedsentence, and the average sentence embedding aggregated over all annotators for the given task. We
[Figure] used annotations only from the 15 “best” annotators under this metric as our ground-truth languageutterances for training our models.
Further details on our data collection and ﬁltering procedure arein the Supplemental, as well as example “spam” and noisy annotations that were ﬁltered out.
Table 1provides examples of crowdworker utterances
Participants & Procedure.
We conducted our study with a participant pool of 10 university students (5 female and 5 male, age range 23 .
2 ± 1 . 87 ) with a Franka Emika Panda Arm (7-DoF).
Foursubjects had prior experience teleoperation a robot arm.
3 We conduct a within-subjects study, whereeach participant completed 2 tasks, chosen randomly, with each of the three methods.
Users weregiven 2 trials to complete each tasks, and an allotted 3 minutes per control strategy to practice.
Userswere also given a sheet describing controller inputs and details for each control method, which weinclude in the Supplemental.
For Imitation Learning and LILA controllers, which require languageinputs, participants provided a natural language utterance which was entered into the model – participants were allowed to verify the proctor entered their input accurately.
This user-provided languageutterance is used as the query in the nearest-neighbor retrieval described in §4.2; the retrieval setconsists of all training utterances collected via the crowdsourcing procedure above.
In addition totracking quantitative success rates (normalized, based on progress relative to each of the 4 deﬁnedsubtasks), time taken per task, and detailed logs of the robot behavior, at the end of each study,users were asked to ﬁll out a qualitative survey evaluating each control method.
We present bothquantitative and qualitative results in the following paragraphs.
Quantitative Results.
Fig.
3 summarizes our objective results.
We evaluate both full- and partialtask success rates for each task across all control methods, in addition to computing smoothnessmetrics directly on the logged user inputs and robot actions.
Intuitively, smoothness is a measurefor intuitiveness (when measured on user 2-DoF joystick inputs), and ease of use (when measuredon the robot’s end-effector pose).
We measure smoothness by calculating jerk, the time-derivativeof acceleration.
Our results show that LILA signiﬁcantly ( p > 0 . 05 ) outperforms both methodsacross all sub-tasks, and is also smoother to use both in control input space (2-DoF input) andend-effector space (6-DoF).
However, the relative drop in performance of LILA for the ﬁnal subtask “Complete Task” shows the room for improvement in ﬁne-grained control, such as pouringmotions.
Additionally of note is the poor performance of Imitation Learning.
To explore this fully,we perform an ablation, and show that sample inefﬁciency is a likely cause – especially since we arein a low-data regime.
{+These results and arguments can be found in the Supplemental. 
+}
Qualitative Results.
Fig.
4 summarizes our subjective results.
We administered a 7-point Likertscale survey after users ﬁnished performing tasks with each of the three different methods; thissurvey included questions such as the the perceived helpfulness of the controller in completingthe tasks ( helpful ) and whether the participant would use the control method again ( use again ).
The survey results show that LILA outperforms both Imitation Learning and End Effector controlsacross most qualitative metrics, with statistically signiﬁcant results ( p > 0 . 05 ) marked with an ∗ . We
Additional experiment videos, and a summary of why imitation learning results are poor
can [Figure] additionally visualize samples of the observed end-effector trajectories by individual users (collectedduring our study) for 3 of our 5 tasks in Fig.
We observe that across all tasks, LILA results insmoother end-effector trajectories than end-effector control, while Imitation Learning comes closeto the target object but is unable to complete the entire trajectory for the task.
Discussion
Summary.
We present Language-Informed Latent Actions (LILA) a framework that marks the ﬁrststep in combining the expressiveness and naturalness of language for specifying and disambiguatingtasks within the context of assistive teleoperation.
Our user study results show that when comparedto fully autonomous, imitation learning approaches, LILA is more sample-efﬁcient and performant,training on half the number of task demonstrations, but obtaining signiﬁcantly higher success rates.
Compared with no-learning end-effector control methods, we again show LILA’s effectiveness atobtaining high success rates, but also demonstrate its ability to produce intuitive low-dimensionalcontrol spaces from language input.
Qualitatively, we ﬁnd that users prefer LILA to alternativemethods across the board, opening the door for additional work on language & latent actions.
Limitations and Future Work.
Currently, LILA is geared to use language as a mechanism fortask disambiguation – in its current form, it has no mechanism for generalizing to completely unseentasks or language speciﬁcations.
We believe that the ability to disambiguate with language, and theintegration of language within the latent actions framework is a strong research contribution, andhope that future work looks to dynamic states – perhaps by leveraging visual latent actions [25] –and to adapting to new utterances and tasks dynamically [52, 40].
Furthermore, while users found LILA intuitive and natural , they found themselves wanting to further modulate the robot’s behaviorwith language instructions during the course of execution .
Many users, upon seeing the robot makeslight deviations from a desired path would instinctively provide spoken corrections – “a little tothe right” , “no, grasp it by the handle!”
– indicating a desire for multi-resolution language control.
Future work will look into this as well.
Shared Autonomy and LILA.
LILA ﬁts within the shared autonomy paradigm, where the roleof language has been underexplored.
With LILA (and shared autonomy approaches in general),humans retain agency – they are responsible for robot motion, and if the robot moves in a way that isnot safe, or does not align with their objectives, they stop providing control (and possibly reset, givea new instruction, or drop into a more complex control mode).
Behavior is interpretable – the latentactions model, critically informed by language , produces intuitive control spaces that humans canquickly grasp.
Finally, language is natural – users specify their objectives as they would if speakingto another person, and the robot uses that language to shape their control space.
These properties– preserving agency, maintaining interpretability, and leveraging the expressive and natural featuresof language for specifying objectives – are critical for widespread human-robot collaboration, andwe hope this work is a ﬁrst step towards achieving that goal.