VLN -Trans: Translator for the Vision and Language Navigation Agent (2023)

Abstract: Language understanding is essential for the navigation agent to follow instructions. We observe two kinds of issues in the instructions that can make the navigation task challenging: 1. The mentioned landmarks are not recognizable by the navigation agent due to the different vision abilities of the instructor and the modeled agent. 2. The mentioned landmarks are applicable to multiple targets, thus not distinctive for selecting the target among the candidate viewpoints. To deal with these issues, we design a translator module for the navigation agent to convert the original instructions into easy-to-follow sub-instruction representations at each step. The translator needs to focus on the recognizable and distinctive landmarks based on the agent’s visual abilities and the observed visual environment. To achieve this goal, we create a new synthetic sub-instruction dataset and design specific tasks to train the translator and the navigation agent. We evaluate our approach on Room2Room (R2R), Room4room (R4R), and Room2Room Last (R2R-Last) datasets and achieve state-of-the-art results on multiple benchmarks.

## introduction vision-and-language navigation (vln) @xcite task requires an agent to understand and follow complex instructions to arrive at a destination in a photo-realistic simulated environment. this cross-domain task attracts researchers from the communities of computer vision, natural language processing, and robotics @xcite @xcite . to solve the vln task, one streamline of methods is to build the connections between text and vision modalities by grounding the semantic information dynamically @xcite @xcite . however, we observe two types of instructions that make the grounding in the vln task quite enter the door and turn right passing the wall. ## method in our navigation problem setting, the agent is given an instruction, denoted as @xmath0, w 2 , • • • , w l }, where l is the number of tokens. also, the agent observes a panoramic view including 36 viewpoints 1 at each navigation step. there are n candidate viewpoints that the agent can navigate to in a panoramic view, denoted as @xmath1, i 2 , • • • , i n }. the task is to generate a trajectory that takes the agent close to a goal destination. the navigation terminates when the navigation agent selects the current viewpoint, or a pre-defined maximum navigation step is reached. fig. 3 (a) provides an overall picture of our proposed architecture for the navigation agent. we use vln⟳bert @xcite (in sec. 3.1) as the backbone of our navigation agent and equip it with a novel translator module that is trained to convert the full instruction representation into the most relevant sub-instruction representation based 1 12 headings and 3 elevations with 30 degree interval. on the current visual environment. another key point of our method is to create a synthetic subinstruction dataset and design the pre-training tasks to encourage the translator to generate effective sub-instruction representations. we describe the details of our method in the following sections. ## conclusion in the vln task, instructions given to the agent often include landmarks that are not recognizable to the agent or are not distinctive enough to specify the target. our novel idea to solve these issues is to include a translator module in the navigation agent that converts the given instruction representations into effective sub-instruction representations at each navigation step. to train the translator, we construct a synthetic dataset and design pretraining tasks to encourage the translator to generate the sub-instruction with the most recognizable and distinctive landmarks. our method achieves the sota results on multiple navigation datasets. we also provide a comprehensive analysis to show the effectiveness of our method. it is worth noting that while we focus on r2r, the novel components of our technique for generating synthetic data and pre-training the translator are easily applicable to other simulation environments.