ULN : Towards Underspecified Vision-and-Language Navigation (2022)

Abstract: Vision-and-Language Navigation (VLN) is a task to guide an embodied agent moving to a target position using language instructions. Despite the significant performance improvement, the wide use of fine-grained instructions fails to characterize more practical linguistic variations in reality. To fill in this gap, we introduce a new setting, namely Underspecified vision-and-Language Navigation (ULN), and associated evaluation datasets. ULN evaluates agents using multi-level underspecified instructions instead of purely fine-grained or coarse-grained, which is a more realistic and general setting. As a primary step toward ULN, we propose a VLN framework that consists of a classification module, a navigation agent, and an Exploitation-to-Exploration (E2E) module. Specifically, we propose to learn Granularity Specific Sub-networks (GSS) for the agent to ground multi-level instructions with minimal additional parameters. Then, our E2E module estimates grounding uncertainty and conducts multi-step lookahead exploration to improve the success rate further. Experimental results show that existing VLN models are still brittle to multi-level language underspecification. Our framework is more robust and outperforms the baselines on ULN by ~10% relative success rate across all levels.

## introduction vision-and-language navigation (vln) allows a human user to command or instruct an embodied agent to reach target locations using verbal instructions. for this application to step out of curated datasets in real-world settings, the agents must generalize to many linguistic variations of human instructions. despite significant progress in vln datasets @xcite @xcite and agent design @xcite @xcite , 2021), it remains a question whether existing models are generalized and robust enough to deal with all kinds of language variations. for the language input in an indoor environment, some datasets focus on long and detailed instructions with the route description at every step to achieve fine-grained language grounding @xcite or long-horizon navigation @xcite . for instance, from room-to-room (r2r) @xcite , to room-across-room (rxr) @xcite , the average instruction length increases from 29 to 129 words. other datasets have coarse-grained instructions like reverie @xcite or soon @xcite . agents are trained and evaluated on a single granu-larity or one type of expression. in contrast, we propose to evaluate vln agents on multi-level granularity to better understand the behavior of embodied agents with respect to language variations. our motivation is that users are inclined to give shorter instructions instead of detailed route descriptions because 1) users are not omniscient observers who follow the route and describe it step by step for the agent; 2) shorter instructions are more practical, reproducible, and efficient from a user's perspective. 3) users tend to underspecify commands in familiar environments like personal households. therefore, we propose a new setting, namely underspecified vision-and-language navigation (uln) and associated evaluation datasets on top of r2r, namely r2r-uln to address these issues. r2r-uln contains underspecified instructions where route descriptions are successively removed from the original instructions. each long r2r instruction corresponds to three shortened and rephrased instructions belonging to different levels, which preserves partial alignment but also introduces variances. as shown in fig. 1 , the goal of uln is to facilitate the development of a generalized vln design that achieves balanced performance across all granularity levels. as a primary step toward uln, we propose a modular vln framework that consists of an instruction classification module, a navigational agent, and an exploitation-to-exploration (e2e) module. the classification module first classifies the input instruction as high-level or low-level in granularity so that our agent can encode these two types accordingly. as for the agent, we propose to learn granularity specific sub-networks (gss) to handle both levels with minimally additional parameters. a sub-network, e.g., the text encoder, is trained for each level while other parameters are shared. finally, the e2e module estimates the stepwise language grounding uncertainty and conducts multi-step lookahead exploration to rectify wrong decisions that originated from underspecified language. our vln framework is model-agnostic and can be applied to many previous agents that follow a "encode-then-fuse" mechanism for language and visual inputs. we establish our framework based on two state-of-the-art (sota) vln agents to demonstrate its effectiveness. we conduct extensive experiments to analyze the generalization of existing agents and our framework in uln and the orig-inal datasets with fine-grained instructions. our contribution is three-fold: vln datasets vln has gained much attention @xcite with emergence of various simulation environments and datasets @xcite @xcite @xcite . r2r @xcite and rxr @xcite provide fine-grained instructions which guide the agent in a step-wise manner. fg-r2r @xcite and landmark-rxr @xcite segments the instructions into action units and explicitly ground sub-instructions on visual observation. in contrast, reverie @xcite , and soon @xcite proposes to use referring expression with no guidance on intermediate steps that lead to the final destination. compared to these datasets, uln aims to build an agent that can generalize to multilevel granularity after training once, which is more practical for real-world applications. embodied navigation agents learning to ground instructions on visual observations is one major problem for an agent to generalize to an unseen environment @xcite @xcite . previous studies demonstrate significant improvement by data augmentations @xcite @xcite @xcite , designing pre-training tasks @xcite @xcite and decoding algorithms @xcite @xcite . for exploration-based methods, fast @xcite proposes a searching algorithm that allows the agent to backtrack to the most promising trajectory. ssm @xcite memorizes local and global action spaces and estimates multiple scores for candidate nodes in the frontier of trajectories. compared to e2e, active vln @xcite is the most relevant work where they learn an additional policy for multi-step exploration. however, they define the reward function based on distances to target locations, while our uncertainty estimation is based on step-wise grounding mismatch. our e2e module is also more efficient that has fewer parameters and low training complexity. ## underspecification in vln our dataset construction is three-fold: we first obtain underspecified instructions by asking workers to simplify and rephrase the r2r instructions. then, we validate that the goals are still reachable with underspecified instructions. finally, we verify that instructions from r2r-uln are preferred to r2r ones from a user's perspective, which proves the necessity of the uln setting. we briefly de- level instructions l 0 turn around and go down the stairs. at the bottom turn slightly right and enter the room with the tv on the wall and a green table. walk to the right past the tv. stop at the door to the right facing into the bathroom. (from r2r) l 1 take the stairs to the bottom and enter the room with the tv on the wall and a green table. walk past the tv. stop at the door to facing into the bathroom. (redundancy removed) l 2 take the stairs to the bottom and enter the room a green table. walk past the tv. stop at the bathroom door. (partial route description) l 3 go to the door of the bathroom next to the room with a green table. (no route description) ## conclusion in this work, we consider a new setting, underspecified vision-and-language navigation (uln). we collected a large-scale evaluation set, r2r-uln, with multi-level underspecified instructions. we show that uln is a reasonable and practical setting. uln presents a novel direction where explorations are necessary and justifiable. as a first step towards uln, we propose two novel components to build a vln framework, granularity specific sub-network (gss) and exploitation-to-exploration (e2e). experimental results show that gss and e2e effectively mitigate the performance gap across all levels of instruction. finally, we believe that uln is a more challenging setting for future vln development, and our framework can be further improved with more sophisticated policy design or language grounding models.