Hierarchical Control of Situated Agents through Natural Language (2022)

Abstract: When humans perform a particular task, they do so hierarchically: splitting higher-level tasks into smaller sub-tasks. However, most works on natural language (NL) command of situated agents have treated the procedures to be executed as flat sequences of simple actions, or any hierarchies of procedures have been shallow at best. In this paper, we propose a formalism of procedures as programs, a method for representing hierarchical procedural knowledge for agent command and control aimed at enabling easy application to various scenarios. We further propose a modeling paradigm of hierarchical modular networks, which consist of a planner and reactors that convert NL intents to predictions of executable programs and probe the environment for information necessary to complete the program execution. We instantiate this framework on the IQA and ALFRED datasets for NL instruction following. Our model outperforms reactive baselines by a large margin on both datasets. We also demonstrate that our framework is more data-efficient, and that it allows for fast iterative development.

## introduction procedural knowledge, or "how-to" knowledge, refers to knowledge of how to execute particular tasks. it is inherently hierarchical; high-level procedures consist of many lower-level procedures. for example, "cooking a pizza" comprises many lowerlevel procedures, including "buying ingredients", "knead dough", etc. there are also multiple levels of hierarchy; "buying ingredients" can be further decomposed to "go to a grocery", "paying" etc. there has been significant prior work on benchmarks and methods for complex task completion using situated agents given natural language (nl) instructions, such as agents trained to navigate the web and mobile uis @xcite 1 all code will be released upon acceptance. or solve household tasks @xcite . however, most methods applied to these tasks use a reactive strategy that makes decisions on the lowlevel atomic actions available to the agent while making steps through the environment @xcite , or define procedures in a shallow way where there only exists one level of hierarchy @xcite @xcite .these approaches are often data-inefficient due to the semantic gap between abstract natural language instructions and concrete executions. in contrast, several works have demonstrated that using specially designed intermediate representations tailored to individual tasks @xcite @xcite can help reduce this expense and improve performance, albeit at the cost of significant effort on the part of the researchers devising these methods. in this paper, we propose a framework to improve the execution of complex natural language commands (example in fig. 1 ) by expressing procedures as programs (pap) written in a high-level programming language like python ( §4). this makes it easy for human engineers to express and leverage their hierarchical procedural knowledge, and the execution of each program yields actions to accomplish a task described in nl. there are several merits to this approach. first, programs are inherently hierarchical; they apply nested function calls to realize higher-level functionality with multiple calls to lower-level functionality. second, programs have built-in control-flow operators, making it possible to deal with multiple divergent situations without the loss of higher-level abstraction. third, programs provide a flexible way to define, share and call different machine-learned components to perceive the environment through an embodied agent's executions. finally, programs in a familiar high-level programming language are comprehensible and curatable, allowing for fast development on various tasks. these four features remain largely unexplored in the existing representations @xcite @xcite , as discussed further in §2. coupled with this representation, we propose a modeling paradigm of hierarchical modular networks (hmn; §5) that has (1) a learnable plan-ner that maps nl to the corresponding executable programs and (2) a collection of reactors that perceive the environment and provide contextsensitive feedback to decide the further execution of the program. such modular design can facilitate training efficiency and improve the performance of each individual component @xcite . we instantiate our framework on two task settings: the iqa dataset @xcite where an agent explores the environment to answer questions regarding objects; and the alfred dataset @xcite , in which an agent must map natural language instructions to actions to complete household tasks ( §6). in experiments ( §7), we find that our framework outperforms the reactive baseline by a significant margin on both datasets, and is significantly more data-efficient. we also demonstrate the flexibility of our framework for fast iterative development of program libraries. we end with a discussion of the limitations of the framework and the potential solutions, paving the way for future works that scale our framework to more open-domain tasks ( §7). ## contrast to previous formalisms while designing intermediate representations that stand between nl and low-level actions for individual tasks has been studied in the literature, our goal is to design a framework that makes it simple to design such representations for new tasks, with a particular focus on capturing the hierarchical nature of procedures. in contrast to most previous works in this area, which employ relatively esoteric representation methods such as lambda calculus @xcite , pap uses widely-adopted general-purpose programming languages (e.g. python) to specify and represent hierarchical procedures. these are comprehensible to most engineers and do not require system designers to learn a new task-specific language. pap also enable easy creation of more hierarchical procedures with reusable sub-routines. existing works either do not model such sub-procedures as reusable components @xcite , or define procedures as a flat sequence of actions without any hierarchy @xcite . the hierarchical procedures with reusable sub-routines is also reminiscent of works in semantic parsing, which compose programs from idiomatic program structures @xcite . more discussions are in §e. additionally, pap uses control flow with divergent branches to handle environment-specific variations of a high-level procedure. a single procedure could therefore dynamically adapt to a variety of environments following the branches triggered by the environments. this makes our representations more compact. this feature also allows developers to easily inject human priors of executions traces under different conditions, which might be challenging to learn in a data-efficient manner. to our best knowledge, this feature is largely unexplored in the literature on designing intermediate representations for agent control. finally, pap provides a convenient interface for procedures to query and interact with task-specific situated components (e.g. a visual component). un-der pap, situated components are exposed as predefined apis, and can be easily called by highlevel procedures. in contrast, existing works either require separate mechanisms to call such components @xcite , or the environment where they are expected to work is less complex, and thus the flexible use of a collection of situated components is not a necessity @xcite . we can also view the pap formalism as a way to construct behavior trees @xcite , which have been used in robotic planning and game design literature. we can use the offthe-shelf tools to convert the programs to abstract syntax trees (ast) which resemble these trees. previous works on robotics also leverage planning domain definition language (pddl) and answer set planners (asp) for task planning @xcite , which is conceptually different from our formalism. pddl+asp searches for an action sequences based on the initial and the final states, while our formalism focuses on describing the actual procedure used to accomplish a task. ## task: controlling situated agents first, we define the task of controlling an agent in some situated environment e through natural language. the environment e provides a set of atomic actions a @xmath0, a a 2 , ...} to interact with the environment. each atomic action can take zero or more arguments that specify which parts of the environment to which it is to be applied. we denote action a a i 's jth argument as r i,j . the specific type of each argument will depend on the action and environment; it could be discrete symbols, scalar values, tensors describing regions of the visual space, etc. given a user intent x, the control system aims at creating an atomic action sequence consisting of a sequence of actions @xmath1, a 2 , ...] (a @xmath2. this action sequence is executed against the environment to achieve a result ŷ = e(a, r), which is compared against a gold-standard result y using a score function s(y, ŷ). action sequences realizing the intent will receive a high score, and those that do not will receive a low score. ## representing procedures as programs next, we introduce the main components of our formalism. a few examples are listed in tab. 1. foot_0 # c1: an atomic action to toggle on an appliance def atomic_toggle_on(obj): env.call("toggle_on", obj) # c2: a procedural action to pick and then put an object def udp_pick_and_put_object(obj, dst): udp_pickup_object(obj) udp_put_object(obj, dst) # c3: an emptying receptacle procedure with for-loop def udp_empty_recep(recep, dst): @xmath3: udp_pick_and_put_object(obj, dst) # c4: a pickup object procedure with control flow def udp_pickup_object(obj interface to atomic actions a a (c1) atomic actions provide a medium for direct interaction with the environment. the call of an atomic action with proper argument types will invoke the corresponding execution in the environment. procedural actions a p (c2-c4) procedural actions describe abstractions of higher-level procedures composed of either lower-level procedures or atomic actions. notably, lower-level procedures can be re-used across many higher-level procedures without re-definition. formalizing the hierarchies in this compact way can not only facilitate the procedure library curation process but also potentially benefit automatic library induction (e.g. through minimal description length @xcite ). ## hierarchical modular networks this section introduces how to use the procedure library a to generate executable programs to complete tasks described in natural language x. we propose a modeling method of hierarchical modular networks (hmn) that consists of two main components. first, there is a hmn-planner that convert x to an executable procedural action a @xmath4, a 2 , ..., a n } where a i either belongs to atomic functions a a or procedural functions a p . we model the hmn-planner as a sequence-tosequence model where the encoder takes x as input, and the decoder generates one function a i at a time from a constrained vocabulary a p a a , conditioned on x and the action history {a 1 , ..., a i-1 }. next, we define the collection of situated components, "reactors," as hmn-reactors. each reactor is a classifier that predicts one or many labels given the observed information (e.g. the nl input, the visual observation. for example, reactor2 in c4 in tab. 1 probes the status of a receptacle based on receptacle name and the visual input. hmn-reactors allows us to flexibly share the same reactor among different functions and design separated reactors to serve different purposes. for example in c4, we use two reactors to find the possible receptacle of an object (reactor1) and to perceive the open/closed status of a receptacle (reactor2) since these two tasks presumably require more mutually exclusive information. at the same time, we share reactor2 to also probe the related openable property of a receptacle for more efficient parameter sharing. this sort of modular design leads to efficient training and improved performance @xcite . ## instantiations in this section, we introduce two concrete realizations of the proposed framework over the iqa dataset @xcite and the alfred dataset @xcite . both are based on egocentric vision in a high-fidelity simulated environment thor @xcite . ## experiments we compare our proposed framework with the baseline reactive agents that predicts a single atomic action at each time step. notably, we apply the same pretrained vision models, pre-searched map and the navigate atomic action used in pap-hmn to the baseline to ensure a fair comparison. more details in §c.2. we attempt to answer the following research questions: (1) does our framework performs better in complex tasks with inherent hierarchical structures, comparing to a purely reactive system? if so, in what way? (2) can our framework leverage the procedural knowledge encoded in the procedure library and the modularity of its hmn to learn more efficiently? and (3) can our framework accelerate the development of the task of interest? ## limitations and future work overall, our experiments demonstrate the benefit of our framework for encoding hierarchical procedural knowledge, especially under low-data or few-shot generalization regimes. one limitation of the experiments here is that they covered domains where it is relatively easy to enumerate the tasks that must be solved in the domain. one intuitive solution in situations where this is not possible is to manually create libraries that cover major procedures but fall back to atomic/reactive control when necessary. for example, as in tab. 7, the program can call a reactor implemented as a the neural network (atomic_reactor) to predict atomic actions when using different appliance to heat an object, instead of enumerating different conditional branches. another possibility is to automate procedure library creation through mining structured procedural knowledge from web @xcite , or through induction of high-level procedures from corpora of atomic action sequences @xcite . another interesting note is that though hierarchical procedural knowledge is ubiquitous in human daily life, most existing nl instruction following benchmarks do not feature such complex, hierarchical procedures. although there can be hierarchies embedded in vision-language navigation tasks @xcite , game playing through reading documentation @xcite or through nl communication @xcite and mobile phone navigation @xcite , the hierarchies are shallow at best, or the occasional complex ones are limited in their breadth. therefore, creating nl instruction following benchmarks that feature more realistic and diverse procedures is one final important direction for future work.