CLEAR : Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations (2022)

Abstract: Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. In this paper, we aim to solve two key challenges in this task: utilizing multilingual instructions for improved instruction-path grounding and navigating through new environments that are unseen during training. To address these challenges, first, our agent learns a shared and visually-aligned cross-lingual language representation for the three languages (English, Hindi and Telugu) in the Room-Across-Room dataset. Our language representation learning is guided by text pairs that are aligned by visual information. Second, our agent learns an environment-agnostic visual representation by maximizing the similarity between semantically-aligned image pairs (with constraints on object-matching) from different environments. Our environment agnostic visual representation can mitigate the environment bias induced by low-level visual information. Empirically, on the Room-Across-Room dataset, we show that our multi-lingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation. Furthermore, we show that our learned language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task, and present detailed qualitative and quantitative generalization and grounding analysis.

## introduction the vision-and-language navigation task requires an agent to navigate through the environment based on language instructions. this task has two unsolved challenges. first, directly introducing pretrained linguistic and visual representations into these agents suffers from domain shift (i.e., pretrained linguistic and visual representation might not generalize to vln task) @xcite . learning the instruction representation while also learning how to navigate based on the instruction is even more challenging for a multi-lingual agent, since more language variance is injected via multi-lingual instructions. at the same time, it also poses the important question that whether we can utilize multi-lingual instructions to learn a better cross-lingual representation and improve instruction-path grounding and referencing. second, previous works @xcite @xcite @xcite @xcite on vision-language navigation have seen that agents tend to perform substantially worse in environments that are unseen during training, indicating the lack of generalizability of the navigation agent. in this paper, we propose to address these two challenges via crosslingual and environment agnostic representations. although some initial progress @xcite @xcite has been made towards introducing pre-trained linguistic representations into vision-language navigation agents, how to understand and utilize paired multilingual instructions to transfer the pre-trained linguistic representation to multilingual navigation agent still remains unexplored. we argue that for a multilingual agent, the linguistic representation can capture more visual concepts from learning the similarity between paired multilingual instructions. as shown in figure 1 , though the three instructions shown here are in different languages and vary in length and level of detail foot_1 , all of them correspond to the sample path -path a. hence, by learning the similarity between these paired instructions, the cross-lingual language representation of the same visual concept mentioned in these paired instructions (e.g., the red words correspond to the same visual object "wash basin") will be close to each other, making it easier for the agent to comprehend. furthermore, the cross-lingual language representation will benefit from the complementary information from instructions in different languages since they elicit more references to visible entities. for example, in figure 1 , the target room environment "washroom" is only mentioned in english instructions. hindi and telugu instructions could benefit from learning the connection between "washroom" and "wash basin" through learning from the english instruction. moreover, many methods have been proposed to encourage agent generalization to unseen environments during training @xcite @xcite . @xcite has shown that it is the low-level appearance information that causes the environment bias. to mitigate this bias, previous works only consider one single environment when learning the visual representation for a given path. we instead learn an environment-agnostic visual representation by exploring the connections between multiple environments. for the example shown in figure 1 , path a and path b are two semantically aligned paths in different environments. in both cases, the agent needs to head into the washroom and stop beside the wash basin. learning the relationship between these paired paths helps the agent comprehend concepts like "bath tub", and not be distracted by the low-level appearance of the objects in unseen environments. overall, in this paper, we propose 'clear: cross-lingual and environment-agnostic representations' to address the two challenges above. first, we define a visually-aligned instruction pair as two instructions that correspond to the same navigation path. given the instruction pairs, we transfer the pre-trained multilingual @xcite to the vision-language navigation task by encouraging these paired instructions to be embedded close to each other. second, we identify semantically-aligned path pairs based on the similarity between instructions. intuitively, if the similarity between the two instructions is high, then their corresponding navigation path will be semantically similar (i.e., mentioning the same objects like "wash basin"). we further filter out image pairs (a pair of paths will contain multiple image pairs) that do not contain the same objects, for higher path pair similarity. then, we train an environment agnostic visual representation that learns the connection between these semantically-aligned path pairs. we conduct experiments on the room-across-room (rxr) dataset @xcite , which contains instruction in three languages (english, hindi, and telugu). empirical results show that our proposed representations significantly improves the performance over the mono-lingual model @xcite ) by 2.59% in ndtw score on rxr test leaderboard. we further show that our clear approach outperforms our baseline that utilizes resnet @xcite to extract image features by 5.3% in success rate and 4.3% in ndtw score (and it also outperforms a stronger baseline that utilizes the recent clip @xcite ) method to extract image features). moreover, our clear approach shows better generalizability when transferred to room-to-room (r2r) dataset @xcite and cooperative vision-and-dialogue navigation dataset @xcite , and adapted to other sota vln agent @xcite . we also demonstrate the advantage of optimizing similarity between all the three languages in rxr dataset for language representation learning and the effectiveness of the way we generate positive path pairs for visual representation learning. lastly, we demonstrate that our cross-lingual language representation captures visual semantics underlying the instructions, and our environment-agnostic visual representation generalizes better to the unseen environment with both qualitative and quantitative analysis. multiple methods have been proposed to encourage generalization to unseen environments during training @xcite @xcite @xcite . @xcite demonstrates that it is the low-level appearance information that causes the large performance gap between seen and unseen environments. @xcite proposes to use environ-ment dropout on visual features to create new environments and @xcite utilizes adversarial path sampling to encourage generalization. however, both of these methods rely on a speaker module to generate synthetic training data and can be considered as data augmentation methods, which are complementary to our proposed environmentagnostic visual representation. the closest work to ours is @xcite , where they proposes to pair an environment classifier with gradient reversal layer to learn an environment-agnostic representation. however, they only consider one single environment when learning the visual representation for a given path (i.e., given one path and predict its environment). in our environmentagnostic representation learning, we explore the connections between multiple environments (i.e., maximize the similarity between paths from different environments). vision-and-language with multilinguality. there has been growing interest in combining vision and language for tasks such as visual-guided machine translation @xcite @xcite , multi-lingual visual question answering @xcite @xcite , multi-lingual image captioning @xcite , multilingual video captioning @xcite , and multi-lingual image-sentence retrieval @xcite . in this paper, we work on multi-lingual vision-and-language navigation. we use vision (i.e., navigation path) as a bridge between multi-lingual instructions and learn a crosslingual representation that captures visual concepts. moreover, our method also use language as a bridge between different visual environments to learn an environment-agnostic visual representation. ## method in this section, we present our clear method that learns cross-lingual language representations and environment-agnostic visual representations. given these learned language and visual representations, we then train the agent on the vision-andlanguage navigation task with imitation learning and reinforcement learning. the overall representation learning and navigation agent training processes are illustrated in figure 2 . we next describe our representation learning methods in sec. 3.1 and sec. 3.2. the navigation model @xcite and training process are detailed in appendix. you will start by standing in front of a glass door and on your right is a doorway…. మీరు బల్లు వె ౖ పు తిరి గి ఉనా్నరు. ఎడమవె ౖ పుకి తిరగండి …. multilingual encoder visual encoder negative pairs positive pairs (with object-matching constraints) stage1: representation learning navigation decoder right now you are facing towards a corner of a door, slightly turn left and move forward... cross-modal attention visual encoder multilingual encoder stage 2:training navigation agent figure 2 : left: the agent learns a cross-lingual language representation and an environment-agnostic visual representation via maximizing the similarity between positive pairs (connected with blue line) and minimizing the similarity between negative pairs (connected with red dashed line). for simplicity, we use 3 as batch size when illustrating the positive pairs and negative pairs. right: then the agent is trained on the vision-and-language navigation task based on these learned representations. ## conclusion in this paper, we presented the clear method that learns a cross-lingual and environment-agnostic representation. we demonstrated that our crosslingual language representation captures more visual semantics and our environment-agnostic representation generalizes better to unseen environments. our experiments on room-across-room dataset suggest that our clear method improved the performance in all evaluation metrics over a strong baseline. furthermore, we qualitatively and quantitatively analyze the effectiveness of every component of our clear approach and its generalizability to other tasks and base vln agents.