Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models (2023)

Abstract: Pre-trained and frozen LLMs can effectively map simple scene re-arrangement instructions to programs over a robot’s visuomotor functions through appropriate few-shot example prompting. To parse open-domain natural language and adapt to a user’s idiosyncratic procedures, not known during prompt engineering time, fixed prompts fall short. In this paper, we introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting: relevant memories are retrieved based on the current dialogue, instruction, correction or VLM description, and used as in-context prompt examples for LLM querying. The memory is expanded during deployment to include pairs of user’s language and action plans, to assist future inferences and personalize them to the user’s language and routines. HELPER sets a new state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TfD), with 1.7x improvement over the previous SOTA for TfD. Our models, code and video results can be found in our project’s website: https://helper-agent-llm.github.io.

## introduction parsing free-form human instructions and humanrobot dialogue into task plans that a robot can execute is challenging due to the open-endedness of environments and procedures to accomplish, and to the diversity and complexity of language humans use to communicate their desires. human language often contains long-term references, questions, errors, omissions, or references and descriptions of routines specific to a particular user @xcite @xcite . instructions need to be interpreted in the environmental context in which they are issued, and plans need to adapt in a closed-loop to execution failures. large language models (llms) trained on internet-scale text can parse language instructions to task plans with appropriate plan-like or code-like prompts, without any finetuning of the language model, as shown in recent works @xcite @xcite @xcite . the state of the environment is provided as a list of objects and their spatial coordinates, or as a free-form text description from a vision-language model @xcite @xcite . using llms for task planning requires engineering a prompt that includes a description of the task for the llm to perform, a robot api with function documentation and expressive function names, environment and task instruction inputs, and a set of in-context examples for inputs and outputs for the task @xcite . these methods are not trained in the domain of interest; rather they are prompt-engineered having in mind the domain at hand. how can we extend llm-prompting for semantic parsing and task planning to open-domain, freeform instructions, corrections, human-robot dialogue, and users' idiosyncratic routines, not known at prompt engineering time? the prompts used for the domain of tabletop rearrangement are already approaching the maximum context window of widely used llms @xcite . even as context window size grows, more prompt examples result in larger attention operations and cause an increase in both inference time and resource usage. to this end, we introduce helper (humaninstructable embodied language parsing via evolving routines), a model that uses retrievalaugmented situated prompting of llms to parse free-form dialogue, instructions and corrections from humans and vision-language models to programs over a set of parameterized visuomotor rou-figure 1 : open-ended instructable agents with retrieval-augmented llms. we equip llms with an external memory of language and program pairs to retrieve in-context examples for prompts during llm querying for task plans. our model takes as input instructions, dialogue segments, corrections and vlm environment descriptions, retrieves relevant memories to use as in-context examples, and prompts llms to predict task plans and plan adjustments. our agent executes the predicted plans from visual input using occupancy and semantic map building, 3d object detection and state tracking, and active exploration using guidance from llms' common sense to locate objects not present in the maps. successful programs are added to the memory paired with their language context, allowing for personalized subsequent interactions. we test helper on the teach benchmark @xcite , which evaluates agents in their ability to complete a variety of longhorizon household tasks from rgb input given natural language dialogue between a commander (the instruction-giving user) and a follower (the instruction-seeking user). we achieve a new stateof-the-art in the teach execution from dialog history and trajectory-from-dialogue settings, improving task success by 1.7x and goal-condition success by 2.1x compared to prior work in tfd. by further soliciting and incorporating user feedback, helper attains an additional 1.3x boost in task success. our work is inspired by works in the language domain @xcite @xcite that retrieve in-context prompt examples based on the input language query for nlp tasks. helper extends this capability to the domain of instructable embodied agents, and demonstrates the potential of memory-augmented llms for semantic parsing of open-ended free-form instructive language into an expandable library of programs. the work closest to helper is llm planner @xcite which uses memoryaugmented prompting of pretrained llms for instruction following. however, it differs from helper in several areas such as plan memory expansion, vlm-guided correction, and usage of llms for object search. furthermore, while @xcite frequently seeks human feedback, helper requests feedback only post full task execution and employs visual-language models (vlms) for error feedback, reducing user interruptions. evaluating home assistant frameworks, including habitat @xcite , gibsonworld @xcite , threedworld @xcite , and ai2thor @xcite . alfred @xcite and teach @xcite are benchmarks in the ai2thor environment @xcite , measuring agents' competence in household tasks through natural language. our research focuses on the 'trajectory from dialogue' (tfd) evaluation in teach, mirroring alfred but with greater task and input complexity. ## method helper is an embodied agent designed to map human-robot dialogue, corrections and vlm descriptions to actions programs over a fixed api of parameterized navigation and manipulation primitives. its architecture is outlined in figure 2 . at its heart, it generates plans and plan adjustments by querying llms using retrieval of relevant language-program pairs to include as in-context examples in the llm prompt. the generated programs are then sent to the executor module, which translates each program step into specific navigation and manipulation action. before executing each step in the program, the executor verifies if the necessary preconditions for an action, such as the robot already holding an object, are met. if not, the plan is adjusted according to the current environmental and agent state. should a step involve an undetected object, the executor calls on the locator module to efficiently search for the required object by utilizing previous user instructions and llms' common sense knowledge. if any action fails during execution, a vlm predicts the reason for this failure from pixel input and feeds this into the planner for generating plan adjustments. ## experiments we test helper in the teach benchmark @xcite . our experiments aim to answer the following questions: ## limitations our model in its current form has the following limitations: ## conclusion we presented helper, an instructable embodied agent that uses memory-augmented prompting of pre-trained llms to parse dialogue segments, instructions and corrections to programs over action primitives, that it executes in household environments from visual input. helper updates its memory with user-instructed action programs after successful execution, allowing personalized interactions by recalling and adapting them. it sets a new state-of-the-art in the teach benchmark. future research directions include extending the model to include a visual modality by encoding visual context during memory retrieval or as direct input to the llm. we believe our work contributes towards exciting new capabilities for instructable and conversable systems, for assisting users and personalizing human-robot communication.