Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areas (2022)

Abstract: Vision and language navigation (VLN) is a challenging visually-grounded language understanding task. Given a natural language navigation instruction, a visual agent interacts with a graph-based environment equipped with panorama images and tries to follow the described route. Most prior work has been conducted in indoor scenarios where best results were obtained for navigation on routes that are similar to the training routes, with sharp drops in performance when testing on unseen environments. We focus on VLN in outdoor scenarios and find that in contrast to indoor VLN, most of the gain in outdoor VLN on unseen data is due to features like junction type embedding or heading delta that are specific to the respective environment graph, while image information plays a very minor role in generalizing VLN to unseen outdoor areas. These findings show a bias to specifics of graph representations of urban environments, demanding that VLN tasks grow in scale and diversity of geographical environments.

## introduction vision and language navigation (vln) is a challenging task that requires the agent to process natural language instructions and ground them in a visual environment. the agent is embodied in the environment and receives navigation instructions. based on the instructions, the observed surroundings, and the current trajectory the agent decides its next action. executing this action changes the position and/or heading of the agent within the environment, and eventually the agent follows the described route and stops at the desired goal location. the most common evaluation metric in vln is the proportion of successful agent navigations, called task completion (tc). 1 code: https://github.com/raphael-sch/ map2seq_vln data & demo: https://map2seq.schumann.pub/ vln/ while early work on grounded navigation was confined to grid-world scenarios @xcite , recent work has studied vln in outdoor environment consisting of real-world urban street layouts and corresponding panorama pictures @xcite . recent agent models for outdoor vln treat the task as a sequence-to-sequence problem where the instructions text is the input and the output is a sequence of actions @xcite @xcite . in contrast to indoor vln @xcite , these works only consider a seen scenario, i.e., the agent is tested on routes that are located in the same area as the training routes. however, studies of indoor vln @xcite show a significant performance drop when testing in previously unseen areas. the main goal of our work is to study outdoor vln in unseen areas, pursuing the research question of which representations of an environment and of instructions an agent needs to succeed at this task. we compare existing approaches to a new approach that utilizes features based on the observed environment graph to improve generalization to unseen areas. the first feature, called junction type embedding, encodes the number of outgoing edges at the current agent position; the second feature, called heading delta, encodes the agent's heading change relative to the previous timestep. as our experimental studies show, representations of full images do not contribute very much to successful vln in outdoor scenarios beyond these two features. one reason why restricted features encoding junction type and heading delta are successful in this task is that they seem to be sufficient to encode peculiarities of the graph representation of the environments. another reason is the current restriction of outdoor environments to small urban areas. in our case, one dataset is the widely used touchdown dataset introduced by @xcite , the other dataset is called map2seq and has recently been introduced by @xcite . the map2seq dataset was created for the task of navigation instructions generation but can directly be adopted to vln. we conduct a detailed analysis of the influence of general neural architectures, specific features such as junction type or heading delta, the role of image information and instruction token types, to outdoor vln in seen and unseen environments on these two datasets. our specific findings unravel the contributions of these features on several vln subtasks such as orientation, directions, stopping. our general finding is that current outdoor vln suffers a bias towards urban environments and to artifacts of their graph representation, showing the necessity of more diverse datasets and tasks for outdoor vln. our main contributions are the following: ## vln problem definition the goal of the agent is to follow a route and stop at the desired target location based on natural language navigation instructions. the environment is a directed graph with nodes @xmath0, v) ∈ e. each node is associated with a 360 • panorama image p and each edge is labeled with an angle α @xcite . the agent state @xmath1: ## model architecture in this section we introduce the model that we use to analyze navigation performance in the unseen and seen scenario for outdoor vln. the architecture is inspired by the cross-modal attention model for indoor vln @xcite . first we give a high level overview of the model architecture and rough intuition. afterwards we provide a more formal description. as depicted in figure 1 , the model follows a sequence-to-sequence architecture where the input sequence is the navigation instructions text and the output is a sequence of agent actions. at each decoding timestep, a new visual representation of the current agent state within the environment is computed, where the agent state is dependent on the previously predicted actions. the decoder rnn has two layers where the first encodes metadata and a visual representation. the second rnn layer encodes a contextualized text and visual representation and eventually predicts the next action. the intuition behind the model architecture is to firstly accumulate plain observations available at the current timestep and entangle them with previous observations in the first recurrent layer. based on these observations, the model focuses attention to certain parts of the instructions text and visual features which are again entangled in the second recurrent layer. thus, we use the acronym orar (observation-recurrence attention-recurrence) for the model. in detail, the instructions encoder embeds and encodes the tokens in the navigation instructions sequence @xmath2, ..., x l using a bidirectional lstm @xcite : ## data we use the touchdown @xcite and the map2seq @xcite datasets in our experiments. both datasets contain human written navigation instructions for routes located in the same environment. the environment consists of 29,641 panorama images from manhattan and the corresponding connectivity graph. ## experiments we are interested in the generalization ability to unseen areas and how it is influenced by the two proposed features, types of visual representation, navigation instructions and training set size. alongside of the results in the unseen scenario, we report results in the seen scenario to interpret performance improvements in relation to each other. all experiments foot_2 are repeated ten times with different random seeds. the reported numbers are the average over the ten repetitions. results printed in bold are significantly better than non-bold results in the same column. significance was established by a paired t-test foot_3 on the ten repetition results and a p-@xmath3.05 without multiple hypothesis corrections factor. individual results can be found in the appendix. ## results & analysis the two upper sections of we believe this is a consequence of the different data collection processes. touchdown was specifically collected for vln and annotators navigated the environment graph, while map2seq annotators wrote instructions only seeing the map. removing the junction type embedding leads to a collapse of task completion in the unseen scenario on both datasets. this shows that without this explicit feature, the agent lacks the ability to reliably identify intersections in new areas. ## related work natural language instructed navigation of embodied agents has been studied in generated grid environments that allow a structured representation of the observed environment @xcite . fueled by the advances in image representation learning @xcite , the environments became more realistic by using real-world panorama images of indoor locations @xcite . complementary outdoor environments contain street level panoramas connected by a real-world street layout @xcite @xcite . agents in this outdoor environment are trained to follow human written navigation instructions @xcite , instructions generated by google maps @xcite , or a combination of both @xcite . recent work focuses on analyzing the navigation agents by introducing better trajectory overlap metrics @xcite or diagnosing the performance under certain constraints such as uni-modal inputs @xcite and masking direction or object tokens @xcite . other work used a trained vln agent to evaluate automatically generated navigation instructions @xcite . an open problem in indoor vln is the generalization of navigation performance to previously unseen areas. proposed solutions include back translation with environment dropout @xcite , multi-modal environment representation @xcite or semantic segmented images @xcite . notably the latter work identifies the same problem in the touchdown task. ## conclusion we presented an investigation of outdoor vision and language navigation in seen and unseen environments. we introduced the heading delta feature and junction type embedding to correct an artifact of the environment and explicitly model the number of outgoing edges, respectively. both are helpful to boost and analyze performance in the unseen scenario. we conducted experiments on two datasets and showed that the considered visual features poorly generalize to unseen areas. we conjecture that vln tasks need to grow in scale and diversity of geographical environments and navigation tasks.