L ang N av: Language as a Perceptual Representation for Navigation (2024)

Abstract: We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent’s egocentric panoramic view at each time step into natural language descriptions. We then finetune a pretrained language model to select an action, based on the current view and the trajectory history, that would best fulfill the navigation instructions. In contrast to the standard setup which adapts a pretrained language model to work directly with continuous visual features from pretrained vision models, our approach instead uses (discrete) language as the perceptual representation. We explore several use cases of our language-based navigation (LangNav) approach on the R2R VLN benchmark: generating synthetic trajectories from a prompted language model (GPT-4) with which to finetune a smaller language model; domain transfer where we transfer a policy learned on one simulated environment (ALFRED) to another (more realistic) environment (R2R); and combining both vision- and language-based representations for VLN. Our approach is found to improve upon baselines that rely on visual features in settings where only a few expert trajectories (10-100) are available, demonstrating the potential of language as a perceptual representation for navigation.

## introduction applications of large language models (lms) to non-linguistic embodied tasks have generally focused on using the implicit world knowledge within lms to predict sub-tasks and actions for planning @xcite @xcite . for instance, recent work has shown that lms can be prompted to create a list of actions (e.g., gotobathroom, locatetoothbrush) given a highlevel goal given in natural language (e.g., "brush teeth") @xcite . these approaches rely on the lm's priors on action sequences and inter-object correlations acquired through largescale pretraining @xcite @xcite , and it has not been clear whether text-only models can be finetuned for tasks such as vision-and-language navigation which requires an egocentric agent follow instructions to navigate a 3d environment using visual input. to be clear, there is a substantial body of work on using pretrained lms for vision-and-language navigation tasks @xcite @xcite . the standard approach is to use a pretrained lm over the natural language instructions to extract text features that are combined with the agent's perceptual representations, which are given by continuous image features extracted from pretrained vision models @xcite . while effective in data-rich regimes, the direct use of vision features makes the approach difficult to apply in cases where only a few labeled trajectories exist (e.g., 10 trajectories), as these approaches need to learn a full joint vision-language module that combines a pretrained vision model with a pretrained text model. a popular strategy in such low data regimes is to generate synthetic data or transfer knowledge from other domains. however, generating realistic perception data is itself a difficult task, and domain transfer with models that rely purely on visual features can overfit to the non-transferable features @xcite . this paper explores an alternative approach for vision-and-language navigation by exploiting language itself as the perceptual representation space. our approach uses off-the-shelf vision models to obtain textual descriptions of the agent's egocentric panoramic view. the text descriptions are then fed to an lm which must select the next action given the instruction and (text descriptions of) the previous actions or observations. see figure 1 for an overview. the use of language to represent an you are a navigation agent who must navigate according to instructions given only descriptions of your current position via natural language. the natural language description is sometimes incorrect. instruction: "go down the stairs and straight into the living room. in the living room walkout onto the patio. on the patio stop outside the doorway." ## background: room-to-room vision-language navigation a popular testbed for vision-and-language navigation (vln) is the room-to-room dataset (r2r; @xcite , in which an agent must perceive and navigate a real-world 3d environment based on a language instruction u and an initial state s 0 . at each time step t, the agent uses the current observation o t , the original language instructions u , and the trajectory history h t , to predict the panoramic action a t . the current observation is given by a set of panoramic images that describe the agent's egocentric view, i.e., o @xmath0,0 , ..., i t,v } where v corresponds to the number of discretized view angles. 1 the panoramic action a t corresponds to which navigable view in o t to go towards, i.e., a @xmath1. after selecting an action, the state transitions from s t to s t+1 . the aim is to output the command stop after reaching the goal g specified by u in state s 0 . the standard approach in r2r is to process the panoramic images {i t,0 , ..., i t,v } with a pretrained visual encoder e v to extract continuous visual features f t,@xmath2,0 ), ..., e(i t,v )}. the language instruction is typically processed by a pretrained language encoder e l (e.g., bert @xcite ) to extract the language features f @xmath3. these features, along with a hidden state representation of the trajectory history h t-1 , are fed to a joint vision-language module (e.g., another transformer) that attends over {i t,0 , ..., i t,v } to select the action a t . ## language as a perceptual representation for navigation we begin by describing the perception-to-text models employed for converting visual observations into text ( § 3.1). we then discuss the prompt templates for converting the text into natural language ( § 3.2), followed by a description of the offline imitation learning algorithm for learning ( § 3.3). ## empirical study our primary experiments with langnav target the low-data setting, motivated by the observation that obtaining annotated data for embodied tasks such as vision-language navigation can be very costly (often more so than is the case for text-only or vision-only tasks). specifically, we are interested in learning the most performant system based on a small number (10 or 100) of in-domain seed navigation trajectories. we sample our seed trajectories from the room-to-room (r2r) dataset @xcite , a popular vision-and-language navigation dataset consisting of 21,567 navigation instructions in the matterport3d environment. the dataset includes 90 scenes, with 61 scenes in the train and validation "seen" sets, and 11 scenes in the validation "unseen" set. our 10-shot dataset is randomly sampled the train set within 1 scene, while our 100-shot dataset spans 2 scenes. evaluation. to contextualize our approach against prior work, we evaluate langnav on both "seen" and "unseen" sets from r2r. the "seen" set contains scenes identical to the training set (but the instructions and trajectories differ). however, this distinction is less important for our low-data regime, since we only make use of 1 scene (for the 10-shot case) or 2 scenes (for the 100-shot case). i.e., the majority of scenes in the "seen" validation subset are actually never seen by the agent. we use the standard r2r task performance metrics @xcite : navigation error (ne), the average distance between the agent's final position and the goal in meters; success rate (sr), the ratio of trajectories in which the agent stopped within 3 meters of the goal; oracle success rate (osr), the ratio of trajectories in which the agent stopped within 3 meters to the goal with a view of the goal; and success weighted by the normalized inverse of the path length (spl). ## discussion interpretability and editability through language. our use of language as a "bottleneck" per-ceptual representation makes it possible to (more easily) interpret and edit a model's predictions. as a qualitative case study, we inspect trajectories where the model made a mistake and manually inspect the captions. we find that model mistakes are generally due to incorrect or ambiguous captions. we manually edit the captions to be correct, and find that in many cases, this is able to change the model's predictions to be correct. see figure 4 for a concrete example. we applied this procedure to 10 randomly selected trajectories which contained an error, and found that we were able to edit the model's decision to the correct one in 7 out of 10 trajectories. (for the other 3 trajectories, the failure was not due to incorrect captions). ## conclusion we show that we can learn to navigate in a realworld environments by using language as a perceptual representation. language naturally abstracts away low-level perceptual details, which we find to be beneficial for efficient data generation and sim-to-real transfer. however, this is also a serious limitation insofar as a picture really is worth a "thousand words" in some cases; we are certainly not suggesting the abandonment of traditional (continuous) vision features for vision-language navigation. but our case studies nonetheless demonstrate the promise of language as a perceptual representation for vision-language navigation.