Skill Induction and Planning with Latent Language (2022)

Abstract: We present a framework for learning hierarchical policies from demonstrations, using sparse natural language annotations to guide the discovery of reusable skills for autonomous decision-making. We formulate a generative model of action sequences in which goals generate sequences of high-level subtask descriptions, and these descriptions generate sequences of low-level actions. We describe how to train this model using primarily unannotated demonstrations by parsing demonstrations into sequences of named high-level sub-tasks, using only a small number of seed annotations to ground language in action. In trained models, natural language commands index a combinatorial library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals. We evaluate this approach in the ALFRED household simulation environment, providing natural language annotations for only 10% of demonstrations. It achieves performance comparable state-of-the-art models on ALFRED success rate, outperforming several recent methods with access to ground-truth plans during training and evaluation.

## introduction building autonomous agents that integrate highlevel reasoning with low-level perception and control is a long-standing challenge in artificial intelligence @xcite @xcite . fig. 1 shows an example: to accomplish a task such as cooking an egg, an agent must first find the egg, then grasp it, then locate a stove or microwave, at each step reasoning about both these subtasks and complex, unstructured sensor data. hierarchical planning models (e.g. @xcite -which first reason about abstract 1 code and visualizations: https://sites.google.com/ view/skill-induction-latent-lang/ . ## preliminaries we consider learning problems in which agents must perform multi-step tasks (like cooking an egg; fig. 1 ) in interactive environments. we formalize these problems as undiscounted, episodic, partially observed markov decision processes (pomdps) defined by a tuple (s, a, t, ω, o), where s is a set of states, a is a set of actions, t : s × @xmath0, ω is a set of observations, and o : @xmath1. 2 we assume that observations include a distinguished goal specification g that remains constant throughout an episode; given a dataset d of consisting of goals g and demonstrations d (i.e. @xmath2, g 1 ), (d 2 , g 2 ) . . .}; @xmath3, a 1 ), (o 2 , a 2 ), . . .]; @xmath4, @xmath5, we aim to learn a goal-conditional policy π(a t | a :t-1 , o :t , g) = π(a t | a 1 , . . . , a t-1 , o 1 , . . . , o t , g) that generalizes demonstrated behaviors to novel goals and states. for tasks like the ones depicted in fig. 1 , this learning problem requires agents to accomplish multiple subgoals (like finding an egg or operating an appliance) in a feasible sequence. as in past work, we address this challenge by focusing on hierarchical policy representations that plan over temporal abstractions of low-level action sequences. we consider a generic class of hierarchical policies that first predict a sequence of subtask specifications τ from a distribution π c (τ i | τ :i-1 , g) (the controller), then from each τ generate a sequence of actions a 1 . . . a n from a distribution π e (a i | a :i-1 , o :i , τ ) (the executor). figure 2: (a) when a hierarchical policy is deployed, π c generates a sequence of subtask specifications, and π e translates each of these to a low-level action sequence ending in stop. at training time, this hierarchical structure is not available, and must be inferred to train our model. to do so, we assign each action a i an auxiliary alignment variable α i identifying the subtask that produced it. alignments divide an action sequence into a sequence of segments s containing actions aligned to the same subtask. automatically segmenting training demonstrations makes it possible to learn modular, reusable policies for individual subtasks without direct supervision. (b) overview of the proposed learning algorithm (sl) 3 , which alternates between segmenting (by aligning) actions to fixed subtask specifications; labeling segments given fixed alignments, and updating model parameters. in fig. 2(a ). trajectories generated by hierarchical policies themselves have hierarchical structure: each subtask specification τ generates a segment of a trajectory (delimited by a stop action) that accomplishes a specific subgoal. training a hierarchical policy requires first defining a space of subtask specifications τ , then parameterizing controller and executor policies that can generate these specifications appropriately. most past research has either pre-defined an inventory of target skills and independently supervised π c and π e @xcite @xcite ; or performed unsupervised discovery of a finite skill inventory using clustering techniques @xcite . both methods have limitations, and recent work has explored methods for using richer supervision to guide discovery of skills that are more robust than human-specified ones and more generalizable than automatically discovered ones. one frequently proposed source of supervision is language: in latent language policies, π c is trained to generate goal-relevant instructions in natural language, π e is trained to follow instructions, and the space of abstract actions available for planning is in principle as structured and expressive as language itself. but current approaches to llp training remain impractical, requiring large datasets of independent, finegrained supervision for π c and π e . below, we describe how to overcome this limitation, and instead learn from large collections of unlabeled demonstrations augmented with only a small amount of natural language supervision. ## approach overview we train hierarchical policies on unannotated action sequences by inferring latent natural language descriptions of the subtasks they accomplish (fig. 2(b) ). we present a learning algorithm that jointly partitions these action sequences into smaller segments exhibiting reusable, task-general skills, labels each segment with a description, trains π c to generate subtask descriptions from goals, and π e to generate actions from subtask descriptions. formally, we assume access to two kinds of training data: a large collection of unannotated demonstrations @xmath6, g 1 ), (d 2 , g 2 ), . . .} and a smaller collection of annotated demonstrations d @xmath7, g 1 , τ 1 ), (d 2 , g 2 , τ 2 ), . . .} where each τ consists of a sequence of natural language instructions [τ 1 , τ 2 , . . .] corresponding to the subtask sequence that should be generated by π c . we assume that even annotated trajectories leave much of the structure depicted in fig. 2(a ) unspecified, containing no explicit segmentations or stop markers. (the number of instructions |τ | will in general be smaller than the number of actions |d|.) training π e ## experimental setup our experiments aim to answer two questions. first, does the latent-language policy representation described in section 3 improve downstream performance on complex tasks? second, how many natural language annotations are needed to train an effective latent language policy given an initial dataset of unannotated demonstrations? environment we investigate these questions in the alfred environment of @xcite . alfred consists of a set of interactive simulated households containing a total of 120 rooms, accompanied by a dataset of 8,055 expert task demonstrations for an embodied agent annotated with 25,743 english-language instructions. observations o are bitmap images from a forward-facing camera, and actions a are drawn from a set of 12 low-level navigation and manipulation primitives. manipulation actions (7 of the 12) additionally require predicting a mask over the visual input to select an object for interaction. see @xcite for details. while the alfred environment is typically used to evaluate instruction following models, which map from detailed, step-by-step natural language descriptions to action sequences @xcite @xcite , our experiments focus on an goal-only evaluation in which agents are given goals (but not fine-grained instructions) at test time. several previous studies have also considered goal-only evaluation for al-fred, but most use extremely fine-grained supervision at training time, including full supervision of symbolic plan representations and their alignments to demonstrations @xcite , or derived sub-task segmentations using alfred-specific rules @xcite . in contrast, our approach supports learning from partial, language-based annotations without segmentations or alignments, and this data condition is the main focus of our evaluation. modeling details π c and π e are implemented as sequence-to-sequence transformer networks @xcite . π c , which maps from text-based goal specifications to text-based instruction sequences, is initialized with a pre-trained t5-small language model @xcite . π e , which maps from (textual) instructions and (imagebased) observations to (textual) actions and (imagebased) object selection masks is also initialized with t5-small; to incorporate visual input, this model first embeds observations using a pretrained resnet18 model @xcite and transforms these linearly to the same dimensionality as the word embedding layer. details about the architecture of π c and π e may be found in appendix c. ## related work our approach draws on a large body of research at the intersection of natural language processing, representation learning, and autonomous control. language-based supervision and representation the use of natural language annotations to scaffold learning, especially in computer vision and program synthesis applications, has been the subject of a number of previous studies @xcite @xcite . here, we use language to support policy learning, specifically by using natural language instructions to discover compositional subtask abstractions that can support autonomous control. our approach is closely related to previous work on learning skill libraries from policy sketches @xcite ; instead of the fixed skill inventory used by policy sketches, (sl) 3 learns an open-ended, compositional library of behaviors indexed by natural language strings. hierarchical policies hierarchical policy learning and temporal abstraction have been major areas of focus since the earliest research on reinforcement learning and imitation learning @xcite @xcite . past work typically relies on direct supervision or manual specification of the space of high-level skills @xcite or fully unsupervised skill discovery @xcite . our approach uses policy architectures from this literature, but aims to provide a mechanism for supervision that allows fine-grained control over the space of learned skills (as in fully supervised approaches) while requiring only small amounts of easy-to-gather human supervision. language and interaction outside of language-based supervision, problems at the intersection of language and control include instruction following @xcite @xcite @xcite , embodied question answering @xcite and dialog tasks @xcite . as in our work, representations of language learned from large text corpora facilitate grounded language learning @xcite , and interaction with the environment can in turn improve the accuracy of language generation @xcite ; future work might extend our framework for semi-supervised inference of plan descriptions to these settings as well. ## conclusion we have presented (sl) 3 , a framework for learning hierarchical policies from demonstrations sparsely annotated with natural language descriptions. using these annotations, (sl) 3 infers the latent structure of unannotated demonstrations, automatically segmenting them into subtasks and labeling each subtask with a compositional description. learning yields a hierarchical policy in which natural language serves as an abstract representation of subgoals and plans: a controller sub-policy maps from goals to natural language plan specifications, and a modular executor that maps each component of the plan to a sequence of low-level actions. in simulated household environments, this model can complete abstract goals (like slice a tomato) with accuracy comparable to state-of-the-art models trained and evaluated with fine-grained plans (find a knife, carry the knife to the tomato, . . . ). while our evaluation has focused on household robotics tasks, the hierarchical structure inferred by (sl) 3 is present in a variety of learning problems, including image understanding, program synthesis, and language generation. in all those domains, generalized versions of (sl) 3 might offer a framework for building high-quality models using only a small amount of rich natural language supervision.