Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on M inecraft (2024)

Abstract: In the Minecraft Collaborative Building Task, two players collaborate: an Architect (A) provides instructions to a Builder (B) to assemble a specified structure using 3D blocks. In this work, we investigate the use of large language models (LLMs) to predict the sequence of actions taken by the Builder. Leveraging LLMs’ in-context learning abilities, we use few-shot prompting techniques, that significantly improve performance over baseline methods. Additionally, we present a detailed analysis of the gaps in performance for future work.

## introduction accurate action prediction is fundamental in developing interactive agents. these agents need to anticipate and respond effectively by interpreting the environment, including its contextual cues @xcite . a key part of this process is the interpretation of instructions and intentions in dialogues to predict appropriate subsequent actions @xcite @xcite . as agents continuously predict actions based on the evolving context, precise action prediction becomes essential for ensuring seamless interaction and coordination @xcite @xcite . building on these principles, our research focuses on modeling the builder's sequence of actions (see figure 1 ) in the minecraft collaborative building task @xcite . this task requires close coordination between an architect (a) and a builder (b). the architect provides instructions to the builder, who assembles a specified structure using 3d blocks. this setup provides an ideal testbed for investigating how advanced computational models can interpret and predict actions based on natural language instructions. prior works @xcite @xcite proposed end-to-end neural models for modeling builder actions prediction. however, achieving high accuracy for this task remains challenging. a key challenge lies in effectively utilizing richer dialogue history when modeling action sequence prediction. meanwhile, llms have demonstrated impressive performance in various natural language processing tasks, primarily due to their in-context learning @xcite capabilities. few-shot prompting techniques, in particular, allow these models to generalize from a limited number of examples @xcite @xcite . this makes llms well-suited for tasks that require a nuanced understanding of longer contexts and action prediction @xcite @xcite @xcite . in this work, we explore the application of llms to predict the builder's actions. taking advantage of their in-context learning capabilities, we prompted llms to predict these actions. to capture the nuances of this task, we adopted a retrievalaugmented approach @xcite . this enhances the llm's ability to predict action se-quences that are not only syntactically correct but also contextually appropriate. consequently, these predictions can be expressed either as textual descriptions or code-based commands. leveraging the code-generation capabilities of llms @xcite , we frame the action prediction task as a code-generation problem. specifically, we model the task as predicting a series of pick() and place() commands for the builder. while the proposed work focuses on a specific task within minecraft, the underlying principles can be extended to more complex domains. modeling action prediction across multiple dialogue turns as a code-generation task is well-suited for structured, repetitive workflows, such as those in industrial automation and robotics, where automation through code can streamline tasks that require precise coordination and context understanding. building on this dataset @xcite ) modeled builder action sequences using an encoder-decoder architecture with dialogue history and world state representation. further advancing this task @xcite proposed a joint learning task for collaborative building and asking clarification questions. however, their approach focuses on generating action sequence of length one, rather than a series of actions. despite these efforts, the task is far from being fully solved. ## dataset minecraft is used for studying collaborative tasks in a 3d voxel grid, with agents following architect's natural language instructions to build structures. the collaborative dialogue dataset @xcite has conversations for 150 target structures with varying levels of abstractions and complexity, grouped into 547 dialogue games, split into train (309 games, 3,792 turn-code pairs), development (101 games, 1,335 turn-code pairs) and test sets (137 games, 1,615 turn-code pairs). we used the test set for our evaluation. ## conclusion we investigate prompting llms to build complex structures for the minecraft building task. predicting suitable builder action sequences for a given natural language instruction is challenging as these llms need to accurately interpret language abstractions, decode spatial co-references and reason about repetitions solely based on the in-context samples in the prompt. we compared multiple instruction-tuned llms, both closed and opensource. in addition to showcasing improved performance over baseline results, we also conduct an in-depth analysis of the generated responses across multiple dimensions. although these models demonstrated a strong ability to accurately capture the action structure (pick, place), they struggled in dealing with spatial references, geometric shapes and anaphora. in the future, we plan to investigate model architectures that can address the shortcomings identified in the evaluated models.