Can LLM ’s Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis (2024)

Abstract: We present a novel approach to automatically synthesize “wayfinding instructions” for an embodied robot agent. In contrast to prior approaches that are heavily reliant on human-annotated datasets designed exclusively for specific simulation platforms, our algorithm uses in-context learning to condition an LLM to generate instructions using just a few references. Using an LLM-based Visual Question Answering strategy, we gather detailed information about the environment which is used by the LLM for instruction synthesis. We implement our approach on multiple simulation platforms including Matterport3D, AI Habitat and ThreeDWorld, thereby demonstrating its platform-agnostic nature. We subjectively evaluate our approach via a user study and observe that 83.3% of users find the synthesized instructions accurately capture the details of the environment and show characteristics similar to those of human-generated instructions. Further, we conduct zero-shot navigation with multiple approaches on the REVERIE dataset using the generated instructions, and observe very close correlation with the baseline on standard success metrics (< 1% change in SR), quantifying the viability of generated instructions in replacing human-annotated data. We finally discuss the applicability of our approach in enabling a generalizable evaluation of embodied navigation policies. To the best of our knowledge, ours is the first LLM-driven approach capable of generating “human-like” instructions in a platform-agnostic manner, without training.

## introduction in embodied navigation tasks, language is primarily used to convey wayfinding instructions to an agent operating in a simulation platform. these instructions convey the path that the agent should take to reach a target location. generating these instructions usually takes place in the form of creating datasets that require several human annotation hours @xcite @xcite . in addition, the current datasets are exclusive to the embodied simulation platform in which the agent operates, preventing the transfer of instruction-following approaches across platforms. for instance, an embodied agent trained to follow instructions present in the r2r @xcite or reverie @xcite datasets is limited to scenarios (object arrangements and scene layouts) in the matterport3d @xcite environment, the most commonly used platform for indoor datasets @xcite . the scenarios themselves are also limited (around 90 real-world scans). if its performance needs to be evaluated on another simulation environment such as tdw @xcite or procthor @xcite , the corresponding reverie or r2r-style instructions simply do not exist, posing a major hurdle for researchers conducting generalizability experiments to assess the adaptability of their navigation models. as such, to alleviate these issues, it is important to design an approach for synthesizing wayfinding instructions that are platform-agnostic, and is not cumbersome to generate. some recent works have looked at synthesizing instructions from input visual landmarks @xcite @xcite . these approaches however are not easily generalizable and require training a separate model for each instruction dataset to infer synthetic instructions. moreover, they only focus on the matterport3d environment, as indoor instruction datasets are scarce on other platforms. main results: we present a novel approach to synthesize wayfinding instructions for an embodied robot agent. figure 1 presents an overview of our approach. given a set of egocentric images captured from a simulator, we perform visual question answering to gather information about the scene, and use this to condition an llm with reference texts to generate different styles of instructions. the novel components of our work include: in contrast to prior work which is limited to a single simulation platform and instruction style, we use in-context learning in llms to achieve instruction synthesis of multiple styles on different embodied simulation platforms, including matterport3d, ai habitat and threedworld. our evaluation both via a user study and navigation performance indicates that the synthesized instructions are sufficiently representative of human-like texts for them to be used as a scalable alternative for generating instructions for embodied navigation tasks. ## approach our approach consists of two components. first, we perform visual question answering (vqa) on egocentric images taken along an agent's path in a simulation environment. this gives us spatial knowledge about the scene. next, we combine this spatial knowledge with a few reference wayfinding instructions in an in-context learning @xcite prompt to condition an llm for synthesizing instructions that would lead the agent to the target location. ## evaluation & results in this section, we discuss our evaluation strategy and present results. ## discussion: evaluating generalizability of embodied navigation policies the overarching motive of our work is to construct a generalist navigation agent that performs consistently irrespective of the environment that it is present in. current approaches to solve this task are limited to evaluation on human-annotated datasets created specifically for a particular simulator, be it mp3d @xcite , ai habitat (ramakrishnan et al., 2021), robothor @xcite etc.. while some methods claim generalizability @xcite , they back their claims by showing improved performance on unseen subsets of a dataset on the same simulator, rather than measuring performance across simulators. for a true measure of generalizability, we believe it is necessary to measure the navigation performance of agents that aren't bounded to a particular dataset. in this direction, our approach solves a crucial data procurement problem in providing a simple method to generate human-like instructions across simulation platforms. in doing so, we empower resource-constrained researchers to create their own datasets for generalizable experiments on their navigation models; therein presenting the true novelty of our work. current datasets cover a wide range of languageguided navigation scenarios, ranging from initialinstruction based guidance (fine and coarsegrained) to oracle and dialogue based navigation that provide verbal human assistance @xcite . there also exist several outdoor datasets including touchdown @xcite , talk2nav @xcite and streetnav @xcite , where the beyond the instruction, the structure and semantics of the scene are drastically different from indoors. to account for the diversity and measure true generalizability, we propose integrating our scheme for synthesis to measure the robustness of navigation policies in two ways as follows:-• cross-platform generalizability: in the first experiment, we gather a set of instructionpath pairs across simulators to train a crossplatform model for a generalist navigation agent. consistent performance on each simulator present in the dataset during inference would indicate that the navigation policy is globally robust with low bias towards a specific simulator. a generalist navigation agent would have a policy that is both globally and locally robust. our approach paves the way to measure this robustness for a fair evaluation of state-of-the-art embodied navigation policies. ## conclusion we present a simple, cross-platform approach to synthesize multiple styles of wayfinding instructions for embodied navigation. our approach requires no training and instead utilizes an llm with in-context learning to produce instructions across multiple simulation platforms. we verify the quality of the instructions generated both via a user study and by evaluating zero-shot vln performance. from these evaluations, we positively infer that our llm-generated instructions are a good replacement to human-annotated ones, and further, that our approach provides for a scalable and accessible solution for creating wayfinding instructions. we finally touch upon how our approach can be used for measuring the key quality of robustness while evaluating language-guided navigation policies; a defining metric to evaluate a generalist navigation agent.