Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation (2024)

Abstract: This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems. The project is now available at https://zehao-wang.github.io/navnuances.

## introduction in the vision-language navigation (vln; @xcite task, an agent is instructed to navigate through virtual environments by following detailed natural language instructions. this task requires an understanding of the interplay between natural language instructions, visual cues, and the sequence of actions undertaken by the agent. this capability is crucial for a wide range of robotic applications, extending from healthcare support to everyday household assistance. despite significant advancements in the latest research, we argue that the performance of vln models may be overestimated. the current standard for evaluating vision-language navigation, as exemplified by the room-to-room (r2r; @xcite and room-across-room (rxr; @xcite 2020) datasets, predominantly hinges on endpoint success rates and broad path alignment metrics. the recent work @xcite suggests the performance of the state of the art is high and even quite close to human performance on these standards. does this mean that the major challenges of the vln task are almost solved? this perspective might be overly optimistic. for instance, a simple intervention shown in figure . 1 on a common vln dataset does not trigger a consistent strong response in the model. in addition, the high success rate of a randomly navigating agent @xcite is non-negligible. this indicates that current evaluation metrics may be insufficiently detailed. furthermore, agents enhanced by large multimodal models (lmms; @xcite ) perform unexpectedly low on standard vln datasets. this contrasts with the strong multimodal understanding demonstrated by lmms in other domains @xcite . this discrepancy motivates us to revisit the evaluation of vln models. in this work, we introduce a new evaluation framework that focuses on atomic instructions, i.e., the singular actions fundamental to vln instructions. diagnosing vln models at the atomicinstruction level allows us to gauge performance through various nuanced perspectives. to achieve this, we first iteratively construct a context-free grammar (cfg; @xcite with the help of llms to systematically articulate the structure of vln task instructions. cfg, treated as a comprehensive representation of vln instructions, allows us to induct and define atomic instruction categories. we group the components in our cfg into five main categories (i.e., direction change, vertical movement, landmark recognition, region recognition, and numerical comprehension) and generate data accordingly to form our novel evaluation dataset navnuances. for each entry in navnuances, a candidate path is determined by the specific path proposing strategy according to its instruction category. the instruction is then generated using cfg and further enriched by llms. to ensure the data correctness, we incorporate human refinement into this automated generation process in the end. the rigorous evaluation protocols in our dataset pose significant challenges, as they require models to demonstrate a thorough understanding of individual concepts. we benchmark various types of models based on our proposed evaluation framework. experiments with navnuances expose model discrepancies and common issues. we observe that recent advancements in the standard r2r dataset primarily stem from enhanced capabilities in vertical movement and region recognition. despite this progress, numerical comprehension shows stagnation across various models. in terms of specific models, zeroshot agents enhanced by llms demonstrated even significant superiority over traditional supervised ones in handling changes in direction and recognizing landmarks. traditional supervised approaches suffer from selective bias, often leading to deficiencies in adapting to shifts in atomic concepts, as demonstrated in figure 1 . our contributions are threefold: firstly, we devise a comprehensive evaluation framework that addresses diverse facets of vision-and-language navigation (vln) at a granular level. secondly, our work includes a thorough benchmarking of prevalent methodologies on ninety diverse scenes, coupled with an in-depth analysis. the experiments demonstrate the deficiencies and differences in the capabilities of previous models, providing valuable insights for advancing the development of vln methods. thirdly, we present a zeroshot baseline as a minor contribution, which enhances navgpt @xcite with gpt-4-vision @xcite integrating direct vision-instruction alignment. ## navnuances dataset the challenge of curating a nuanced dataset is to comprehensively cover the atomic categories in vln instructions. to achieve this, our approach begins by iteratively constructing a context-free grammar (cfg) with the help of llm to articulate and cover all components of vln instructions in a unified representation (section 3.1). then, we induct and categorize the atomic components of the cfg into five principal categories (section 3.2). building on these categorizations, we develop a semi-automatic process for data annotation of each atomic instruction category, adhering to the cfgdefined natural instruction standards (section 3.3). ## experiment we conduct a comprehensive evaluation of various existing vln models across the five main categories in our navnuances dataset. ## conclusion in this study, we establish a systematic framework to diagnose deficiencies in the capabilities of vision-language navigation (vln) models at the atomic instruction level. our experiment results on navnuances across diverse models clearly uncover the limitations of specific models and reveal common issues, which highlight ongoing challenges in the vln task. in addition, our investigation into a modified zero-shot agent enhanced by gpt-4-vision provides empirical evidence that a direct alignment between vision and instructions significantly enhances landmark recognition performance. this insight underscores the potential for leveraging advanced large multimodal models in improving vln systems.