Explicit Object Relation Alignment for Vision and Language Navigation (2022)

Abstract: In this paper, we investigate the problem of vision and language navigation. To solve this problem, grounding the landmarks and spatial relations in the textual instructions into visual modality is important. We propose a neural agent named Explicit Object Relation Alignment Agent (EXOR),to explicitly align the spatial information in both instruction and the visual environment, including landmarks and spatial relationships between the agent and landmarks. Empirically, our proposed method surpasses the baseline by a large margin on the R2R dataset. We provide a comprehensive analysis to show our model’s spatial reasoning ability and explainability.

## introduction vision and language navigation (vln) problem @xcite requires the agent to carry out a sequence of actions in an indoor photorealistic simulated environment in response to corresponding natural language instructions. the first vln benchmark to appear was room-to-room navigation (r2r) @xcite , as shown in figure 1 , the agent needs to generate a navigation trajectory in a visual environment rendered from real images following an instruction. this task is challenging because, apart from understanding the language and vision modalities, the agent needs to learn the connection between them without explicit intermediate supervision. to address this challenge, several recent work started to consider the semantic structure from both language and vision sides. @xcite train an implicit entity-relationship graph allowing an agent to learn the latent concepts and relationships between different components (scene, object and direction). they use the object features extracted from faster-rcnn @xcite instead of only using resnet visual features which can easily overfit on the training environment @xcite . although the grounding ability of their agent improves, their experimental results show that the object features do not help the navigation independently unless their relationships to the scene and direction are modeled. this issue indicates their loosely modeled latent space and motivates us to explore the ways the object features can be further exploited. the recent research finds that indoor navigation agents rely on both landmark and direction tokens in the instruction when taking actions @xcite . however, it is difficult to identify which landmarks the agent should pay attention to at each navigation step. previous works @xcite @xcite mainly use the surrounding visual information as a clue to indicate the landmark tokens that the agent should focus on. however, the semantics of instruction should also play an important role. for example, with the understanding of the instruction "go to the table with chair, and then walk towards the door", the agent needs to give the same attention to "table" and "chair", and less attention to "door" at the first navigation step. in terms of direction tokens, the prior works concentrate most on the direction tokens related to motions, such as "turn left", and ignore the spatial description of landmarks, such as "table on the left". we believe distinguishing those different direction tokens can benefit the navigation performance. the last but not least, modeling the landmarks and their spatial relations can improve the explainability of the agent's actions. in this paper, we propose a neural agent, called explicit object relation alignment agent (exor), to explicitly align the spatial semantics between linguistic instructions and the visual environment. specifically, we first split the long instruction into spatial configurations @xcite , and then we select the important landmarks based on such configurations. after that, in the visual environment, we retrieve the most relevant objects according to their similarity with the selected landmarks in the instructions. moreover, we obtain textual spatial relation encoding to model the spatial relations between the agent and landmarks in the textual instructions, and use visual spatial relation encoding to represent the relation between agent and the image in the visual environment. we then establish a mapping between the two encodings to achieve a better alignment. finally, we use the representations of the aligned objects and spatial relations to enrich the image representations. to the best of our knowledge, none of the previous work modeled the explicit spatial relations considering the agent's perspective for this task. our contribution is summarized as follows: 1. our model achieves the explicit alignments between textual and visual spatial information, and such alignments guide the agent to pay more attention to the objects in the visual environment given landmark mentions in the instructions. ## related work vision and language navigation the visionlanguage navigation problem nowadays has gained an increasing popularity, and various navigation datasets and platforms @xcite are proposed to assist the development of this topic in the community, for example, r2r @xcite and touchdown @xcite datasets, which have extended navigation to the photo-realistic simulation environments. more broadly, there are work also related to instruction-guided household task benchmarks such as alfred @xcite . rxr @xcite is a multilingual navigation dataset with spatial-temporal grounding, cvdn and hanna are a dialog-based interactive navigation dataset @xcite , and reveri @xcite navigates to localize a remote object. accompanied with these benchmark works, numerous deep learning methods @xcite @xcite have been proposed. for r2r task, @xcite propose a sequence-to-sequence baseline model to encode the instructions and decode the embeddings to the low-level action sequence with the observed images. speaker-follower agent proposed by @xcite trains a speaker model to generate the augmented samples to improve the generalizability. they also start modeling a panoramic action space for navigation, which further promotes fast iteration of different vln approaches. grounding in vln it has been observed that the connection between linguistic instruction and visual environment can yield a great improvement in vln task, hence many research efforts for modeling such visual-linguistic relation have recently been developed. in general, we categorize these research works into three directions. the first main thread @xcite @xcite @xcite tends to adopt attention mechanisms for establishing language and vision connections in neural navigation agents. for instance, @xcite apply a visual-textual co-grounding module and a progress monitor to guide the execution progress. the second branch of prior works @xcite @xcite @xcite explores the pre-trained vision and language (vl) representation from the transformer-based models. @xcite design a recurrent unit on the vl transformer models, and fine-tune them on the down stream vln task. notably, the increased model size and additional training process help improve navigation performance and surpass the previous performance by a large margin. the third branch works model the semantic structure, based on both language and vision perspectives, to improve the grounding ability, such as @xcite @xcite , and our work also follows such paradigm. from the language side, @xcite segment the long instruction into sub-instructions and annotate their corresponding trajectories to supervise the agent to learn the alignments. from the image side, instead of using only the resnet visual features that easily over-fits on the training environment, some recent work @xcite @xcite use object representations to improve the generalizability. most importantly, one should bridge both linguistic and visual semantics, and ent-rel @xcite obtains the best results in the third branch of work by building an implicit languagevisual entity relation graph to learn the connections between the two modalities. our work serves as a new method in the third method category. we explicitly model the alignments between landmarks and visual objects and model the spatial relations to improve the spatial reasoning ability of the agent. ## experimental setups dataset we use room-room(r2r) dataset @xcite that is built upon the matterport3d dataset. r2r dataset contains 7198 paths and 21567 instructions with an average length of 29 words. the whole dataset is partitioned into training, seen validation, unseen validation, and unseen test set. the seen set shares the same visual environments with the training set, while unseen sets contain different environments. ## results and analysis