C or N av: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation (2024)

Abstract: Understanding and following natural language instructions while navigating through complex, real-world environments poses a significant challenge for general-purpose robots. These environments often include obstacles and pedestrians, making it essential for autonomous agents to possess the capability of self-corrected planning to adjust their actions based on feedback from the surroundings. However, the majority of existing vision-and-language navigation (VLN) methods primarily operate in less realistic simulator settings and do not incorporate environmental feedback into their decision-making processes. To address this gap, we introduce a novel zero-shot framework called CorNav, utilizing a large language model for decision-making and comprising two key components: 1) incorporating environmental feedback for refining future plans and adjusting its actions, and 2) multiple domain experts for parsing instructions, scene understanding, and refining predicted actions. In addition to the framework, we develop a 3D simulator that renders realistic scenarios using Unreal Engine 5. To evaluate the effectiveness and generalization of navigation agents in a zero-shot multi-task setting, we create a benchmark called NavBench. Our empirical study involves deploying 7 baselines across four tasks, i.e., goal-conditioned navigation given a specific object category, goal-conditioned navigation given simple instructions, finding abstract objects based on high-level instructions, and step-by-step instruction following. Extensive experiments demonstrate that CorNav consistently outperforms all baselines by a significant margin across all tasks. On average, CorNav achieves a success rate of 28.1%, surpassing the best baseline’s performance of 20.5%.

## introduction language-driven navigation is a critical skill for robot assistants when it comes to performing a * equal contribution. † corresponding author. planner environment plan observation (a) single agent planning planner expert environment communication p la n o b s e r v a ti o n (b) multi-agent planning planner expert environment communication plan refiner observation plan out-of-plan/in-plan feedback (c) self-corrected planning wide range of real-world tasks. most autonomous agents are trained using predefined datasets and tasks, and perform well in familiar environments. however, the real world is filled with a multitude of objects and scenes, making it challenging to train an agent that can generalize effectively. recently, large language models (llms) @xcite @xcite @xcite have demonstrated remarkable effectiveness across various tasks and have emerged as versatile autonomous agents capable of informed decision-making @xcite @xcite . these llms are pre-trained on massive textual data, endowing them with extensive commonsense knowledge that proves invaluable for navigation tasks. for instance, they can infer that a stove is likely to be found in the kitchen and that a bed is typically located in a bedroom. the success of gpt has highlighted the efficacy of utilizing human instructions for zero-shot navi-gation tasks. recently, zero-shot agents based on gpt @xcite , have harnessed the power of @xcite to make decisions in r2r dataset @xcite . however, r2r dataset is based on a static, discrete, and unrealistic environment that lacks the complexity of real-world scenarios, including obstacle avoidance. these gpt-based methods may struggle when applied to real-world settings due to their limited consideration of environmental feedback. additionally, other agents @xcite only focus on object navigation but are unable to comprehend complex instructions. to address the aforementioned challenges, we present cornav, an autonomous agent with self-corrected planning for zero-shot vision-andlanguage navigation in continuous environment. cornav possesses several key capabilities, including the ability to understand complex instructions, engage in self-corrected planning based on both environmental and historical feedback, and consult domain experts for crucial information. here is a breakdown of cornav's functionalities: 1) self-corrected planning: during exploration, cornav actively adapts its plan based on feedback. if the agent receives in-plan feedback, indicating that the environmental observation aligns with the plan, it adheres to the generated plan and proceeds with the next action. however, when faced with outof-plan feedback, it modifies the plan accordingly. we also develop a near-realistic simulator using unreal engine 5, which offers enhanced lighting and intricate details compared to previous simuhttps://www.unrealengine.com/ lators @xcite . our simulator encompasses four scenes carefully modeled from realworld scenarios. building upon this novel simulator, we have established a multi-task benchmark named navbench for zero-shot vision-andlanguage navigation. unlike traditional data collection methods, we have harnessed the capabilities of powerful gpt-4 to generate high-quality instructions for various tasks. navbench has been designed to reflect realistic scenarios, covering four distinct tasks: 1) object navigation, namely, goalconditioned navigation given a specific object category, which is a well-explored aspect of zero-shot navigation; 2) goal-conditioned navigation given a simple instruction, e.g., "i want to go upstairs. please help me find the elevator"; 3) completing abstract instruction, e.g., "the floor is dirty and i want to sweep it", implying that the robot should locate a broom; 4) step-by-step instruction following, which simulates common realistic navigation scenarios where the agent must follow a series of stepby-step instructions. we have conducted an extensive study involving various large language models and open-vocabulary models, implementing 7 zeroshot baselines within the navbench framework. our experiments not only showcase the zero-shot capabilities of these foundational models but also highlight the challenging nature of navbench as a benchmark. in summary, our work presents three-fold contributions: the performance of existing vln methods often falls short when applied to the challenges of continuous 3d simulated environments, as exemplified by the more demanding task of vln-ce @xcite . recent advancements in the field, driven by the availability of large-scale datasets and the development of continuous environment simulators like habitat @xcite , gibso-nenv @xcite , and ai2thor @xcite ), have enabled a new set of tasks and benchmarks, which include pointgoal navigation @xcite , object-goal navigation @xcite @xcite , and instructions following navigation @xcite @xcite . @xcite propose a large-scale indoor dataset designed for multimodal and multitask navigation in continuous and audiovisual complex environments. ## cornav our cornav mainly comprises two pivotal components, i.e., multiple domain experts and plan refinement with environmental feedback. an overview of our architecture is shown in figure 2 . in this section, we will delve into the intricacies of these components, followed by a discussion of our navigation discussion mechanism and local policy. ## navbench our simulator is featured for indoor navigation. it stands out by seamlessly integrating a state-of-theart real-time physics engine, which significantly elevates the quality of visual rendering. notably, our simulator boasts dynamic global illumination and diffuse global illumination, allowing for more highly detailed geometry rendering than ever before. our benchmark, navbench, is purposefully designed to address the challenges of zero-shot multitask vision-and-language navigation. it encompasses a wide range of tasks, including navigation to specific goal objects, abstract objects, and specific locations, all guided by natural language instructions. unlike previous benchmarks relying on manual data labeling, we collect data using large language models. ## conclusion in this paper, we introduce cornav, an innovative autonomous agent designed for zero-shot vln. cornav excels in leveraging environmental feedback to refine its plans in realistic scenarios, ensuring adaptability to dynamic surroundings. it also incorporates multiple domain experts for instruction parsing, scene comprehension, and action refinement. our experimental results demonstrate cornav's significant performance advantages over baseline methods across various navigation tasks. furthermore, we contribute to the field by developing a more realistic simulator powered by unreal engine 5. to evaluate our agent's capabilities, limitations while cornav has demonstrated remarkable performance across tasks, it relies on the outcomes of the image tagging and object detection models. these models may introduce noise or miss certain objects in the environment. future research could explore fine-tuning existing visionlanguage models specifically for navigation, potentially yielding even better results.