M ap GPT : Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation (2024)

Abstract: Embodied agents equipped with GPT as their brain have exhibited extraordinary decision-making and generalization abilities across various tasks. However, existing zero-shot agents for vision-and-language navigation (VLN) only prompt the GPT-4 to select potential locations within localized environments, without constructing an effective “global-view” for the agent to understand the overall environment. In this work, we present a novel **map**-guided **GPT**-based agent, dubbed **MapGPT**, which introduces an online linguistic-formed map to encourage the global exploration. Specifically, we build an online map and incorporate it into the prompts that include node information and topological relationships, to help GPT understand the spatial environment. Benefiting from this design, we further propose an adaptive planning mechanism to assist the agent in performing multi-step path planning based on a map, systematically exploring multiple candidate nodes or sub-goals step by step. Extensive experiments demonstrate that our MapGPT is applicable to both GPT-4 and GPT-4V, achieving state-of-the-art zero-shot performance on the R2R and REVERIE simultaneously (~10% and ~12% improvements in SR), and showcasing the newly emergent global thinking and path planning abilities of the GPT.

## introduction large language models (llms) @xcite @xcite have demonstrated strong performance in various domains. as the most powerful llms, the gpt series models @xcite can even serve as the brains of embodied agents @xcite @xcite , enabling them to engage in explicit thinking and decision-making process. moreover, these gpt-based agents are typically zeroshot or few-shot, eliminating the burdensome tasks of data annotation and model training, and they also demonstrate remarkable sim-to-real abilities. recently, llms have also been adopted in vision-and-language navigation (vln), where agents are given human instructions that require them to visually perceive and navigate an indoor environment. previous learning-based vln methods @xcite @xcite @xcite relied on the training on largescale domain-specific data with expert instruction annotations to execute navigation tasks. to address their reliance on training data and potential sim-to-real gap, some gpt-based zero-shot agents @xcite with explainable decision-making abilities have been proposed. these methods translated visual observations into textual prompts and required gpt-4 to act as an agent to select the correct position or direction. besides, they also employed multiple additional experts to handle various subtasks in text form, such as summarizing the history and fusing repeated expert predictions. however, such zero-shot vln agents face some challenges. firstly, these methods are developed for a multi-expert pure language system based on gpt-4, necessitating the conversion of visual observations into text and multiple rounds of text summarizations, which inevitably lead to information loss and hinder the application to multimodal llms (e.g., gpt-4v). more importantly, these agents make decisions based solely on the observations of the local environment. as shown in figure 1 (left), given only the local action space, when an agent realizes that it has engaged in an erroneous exploration, it can only continue to explore surrounding environment aimlessly. in this paper, we propose mapgpt which contains a topological map in the linguistic form to assist in global exploration and adaptive path planning. we first develop a simple yet efficient prompt system with only one navigation expert that can be applied to both gpt-4 and multimodal gpt-4v flexibly. to encourage global exploration, we propose a map-guided prompting method for the gpt model, to build a "global-view" for the agent. specifically, for an online constructed map, we have discarded the precise gps coordinates that are difficult for gpt to understand, while preserving the topological relationships between nodes and incorporating them into prompts to assist in understanding the navigation environment, as shown in figure 1 (right). given this tailored map, we further propose an adaptive planning mechanism to activate gpt's multi-step path planning ability. instead of documenting the thinking process of each step as in previous works, mapgpt generates a multi-step planning similar to a human work plan, and updates it iteratively to achieve strategic exploration of potential objectives. benefiting from this, the agent can adaptively perform path planning based on the map, systematically explore multiple candidate nodes or sub-goals step by step, and backtrack to a specific node for re-exploration when necessary. we conduct experiments on two popular vln benchmarks, namely r2r @xcite and reverie @xcite , which contain step-by-step and high-level instructions respectively. experimental results show the superiority of mapgpt over existing zero-shot vln agents. especially in reverie, mapgpt exhibits enhanced competitiveness (31.6% sr), surpassing even some learning-based methods trained on reverie. extensive ablation studies reveal the advantage of our introduced map and adaptive path planning mechanism in encouraging systematic exploration to improve navigation. our contributions can be summarized as follows. vision-and-language navigation (vln) as a representative multi-modal embodied ai task, vln requires an agent to combine human instructions and visual observations to navigate and locate targets in real-world scenes. previous learning-based approaches @xcite @xcite proposed various model architectures and trained their models on domain-specific datasets. besides, pretrained models @xcite @xcite @xcite @xcite @xcite have been widely applied to produce better multi-modal representations. recently, to address the reliance on domainspecific data and the possible sim-to-real gap, some zero-shot agents based on gpt @xcite have been proposed. however, they suffer from several limitations. for example, navgpt @xcite has limited performance and relies on a two-stage language-only system. discussnav @xcite introduces a sequential multi-experts system to discuss and summarize various information and fuses five repeated predictions to improve performance. some of their designs limit the agent's capability to only address step-by-step instructions in the r2r dataset @xcite , and have not been validated on other styles of instructions (e.g., reverie @xcite ). besides, these agents are limited to local exploration as they can only reason and make decisions within adjacent navigable points. in this paper, we propose a map-guided prompting method with adaptive path planning for global exploration, achieving impressive performance on both r2r and reverie. large language models (llms) llms @xcite @xcite have demonstrated remarkable capabilities in multiple domains. recently, llm-based agents @xcite @xcite @xcite @xcite have also attracted significant interest of the community. for example, voxposer @xcite utilizes llm and vision-language models to extract affordances and constraints, which enables motion planners to generate trajectories for manipulation. langnav @xcite employs llms for navigation, but it merely utilizes @xcite to synthesize some data and performs fine-tuning using llama2 @xcite as the backbone, rather than directly employing llm as a zero-shot agent. in fact, the application of llmbased zero-shot agents in navigation tasks is still limited, and how to prompt llms (including multimodal gpt-4v) to activate global thinking and planning abilities required by navigation task have yet to be explored @xcite . maps for navigation maps used for navigation tasks can be primarily categorized into two types, i.e., metric maps and topological maps. employing slam @xcite for constructing metric maps @xcite is widely used in navigation. however, this type of approach requires a trade-off between map size and computational efficiency, which affects navigation performance. to address this limitation, graph-based topological maps @xcite @xcite have been proposed for preexploring environment or enabling global exploration, such as backtracking to previously visited nodes. however, these methods are all designed for model learning. it remains unexplored how to build a map with prompts and leverage the powerful capabilities of llms for zero-shot reasoning and planning based on the map. ## method in this section, we introduce our newly designed prompt system (sec. 3.1), the details of our map-guided prompting method to help the agent understand global environment (sec. 3.2), and the novel adaptive mechanism that encourages the agent to make multi-step path planning (sec. 3.3). ## conclusion in this paper, we propose a novel zero-shot agent, named mapgpt, for the vln task. mapgpt utilizes map-guided prompting, which builds online constructed maps using prompts that provide gpt with node information and topological relationships to activate global exploration. additionally, we propose an adaptive planning mechanism that enables multi-step path planning based on the map, allowing the agent to systematically explore potential objectives. through extensive experiments, we demonstrate that mapgpt achieves state-of-the-art zero-shot performance with global thinking and path planning capabilities.