LOV i S : Learning Orientation and Visual Signals for Vision and Language Navigation (2022)

Abstract: Understanding spatial and visual information is essential for a navigation agent who follows natural language instructions. The current Transformer-based VLN agents entangle the orientation and vision information, which limits the gain from the learning of each information source. In this paper, we design a neural agent with explicit Orientation and Vision modules. Those modules learn to ground spatial information and landmark mentions in the instructions to the visual environment more effectively. To strengthen the spatial reasoning and visual perception of the agent, we design specific pre-training tasks to feed and better utilize the corresponding modules in our final navigation model. We evaluate our approach on both Room2room (R2R) and Room4room (R4R) datasets and achieve the state of the art results on both benchmarks.

## introduction vision and language navigation (vln) problem @xcite has attracted increasing attention from the communities of computer vision, natural language processing, and robotics because of its broad real-world applications. in this problem setting, the goal of a navigation agent is to move to a target location in a photo-realistic simulated environment by following a detailed natural language instruction, e.g., "walk into the bedroom. walk past the bedroom door. wait at the laundry room door.". two kinds of simulators are used to create the dataset and the corresponding problem formulation: discrete trajectories @xcite and continuous navigation trajectories @xcite . in this paper, we work on the discrete one, that an agent traverses a predefined connectivity graph by selecting the adjacent viewpoint with a higher probability of corresponding to the instruction at each step. the earlier research in the vln area can be divided into two categories. the first category of ## training and inference we train our model with the mixture of imitation learning (il) and reinforcement learning (rl) following @xcite . it minimizes the crossentropy loss of the prediction and ground-truth trajectories by following teacher actions for each navigation step. rl samples an action from the action probability p a t and learns from the rewards. the loss function is the following: text encoder v-o encoder cls transformer cls mlm ssap [cls] turn [mas] to the [mask][sep] … … -180 0 180 -90 -90 orientation encoder vision encoder om vm cls cls cls ## pre-training we follow the model architecture of preva-lent @xcite to obtain the joint cross representations trained on text-image-action triplets, as shown in figure 3 . however, the novelty of our pre-training is that we design new tasks named vision matching (vm) and orientation matching (om) to pretrain for the vision module and orientation module designed in our navigation agent, as shown in figure 2 . moreover, we improve the existing pre-training tasks of the prevalent, ## conclusion the main idea of this paper is to design explicit vision and orientation modules in the neural architecture of a navigating agent. these modules can effectively learn to ground the landmark mentions and spatial information related to the orientation of the agent expressed in the natural language instruction into the visual environment. to make the designed modules more effective, we design new pre-training tasks accordingly to equip the agent with spatial reasoning and visual perception abilities before navigation. we evaluate our model on r2r and r4r datasets and achieve state-of-the-art results. our ablation study shows the effectiveness of our designed modules and pre-training tasks.