Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021)

Abstract: Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations.

## introduction vision-language navigation defines the task of requiring an agent to navigate through a visual environment based on natural language instructions @xcite @xcite @xcite . this task poses several challenges. to complete this task, an embodied agent needs to perceive the surrounding environment, understand the given natural language instructions, and most importantly, ground navigation steps: parse tree: instruction: walk forward then turn right at the stairs then go down the stairs. walk forward then turn at stairs go stairs then right the down . the figure 1 : an example in the room-to-room task. we generate the dependency parse tree for the instruction. the words are grouped by the head node (highlighted in the tree). each sub-instruction (i.e., grouped words) corresponds to one step in the navigation with the same color. modifiers in red boxes can be easily identified from the tree structure. although these models achieve significant improvement in performance, they do not explicitly consider syntactic linguistic information in their alignment and decision-making. we argue that the syntactic information (e.g., phrases, word functions, modifiers) captured by dependency parse trees is crucial for accurate alignment between the instructions and the environment. as shown in figure 1 , for the instruction "walk forward then turn right at the stairs then go down the stairs.", the dependency parse tree effectively aggregates syntactically-close words together for the agent (e.g., groups phrase information "walk forward" at node "walk"), and each phrase here corresponds to one navigation action. besides, the dependency tree structure also helps identify modifiers like "forward" and "right". this syntactic information helps the agent identify important words, locate phrases (e.g., sub-instructions), and learn a better alignment between the instruction and the visual environment. therefore, in this paper, we propose an encoder module that can incorporate simple but important syntactic information from parse trees for the vision-language-navigation task. our proposed encoder utilizes the childsum tree-lstm @xcite over a dependency tree to achieve a syntaxaware representation of the instruction, enabling the agent to focus on more syntactically important words and align not only words but also phrases with the visual scenes. we conduct experiments on both the room-to-room (r2r) dataset @xcite and the room-across-room (rxr) dataset @xcite . empirical results show that our proposed approach significantly improves the performance over the baseline model on success rate and achieves the new state-of-the-art (at the time of submission) on the rxr dataset, which contains instruction in three languages (english, hindi, and telugu). moreover, by using structured information from syntax, we are also able to avoid word-level shallow overfitting of the model and hence achieve better generalization in the unseen environment. our analysis further shows that our syntax-aware agent has better interpretability and learns better cross-modality matching. the closest work to ours is from hong et al. (2020), where they use the dependency tree to generate pre-divided sub-instructions, and then propose a shifting module to select and attend to a single sub-instruction at each time-step, which implicitly captures some syntax information. compared with them, we directly use the dependency tree to explicitly incorporate syntactic information and get syntax-aware instruction representations, hence achieving substantial improvement in generalizing to the unseen environment. tree-based language representations. dependency tree provides essential syntactic information for understanding a sentence. tree-lstm @xcite has been widely used to encode parsed tree information and shown improvement over multiple tasks, such as relation extraction @xcite , machine translation @xcite @xcite , dialogue @xcite , and language inference @xcite . we are novel in incorporating a dependency tree into the visionlanguage navigation task via a tree-lstm for better phrase-level alignment between the visual environment and language instructions. ## method as illustrated in figure 2 , our base model follows the sequence-to-sequence architecture of previous vln agents. our tree-based encoder module is built on top of the strong environment drop agent @xcite . the main difference is that we employ a tree-based language encoder to encode dependency tree information to allow better language grounding. at each time step, we ground all the encoded nodes (i.e., syntax-aware representations) in the dependency tree with the visual information to get the attended textual representation. generating dependency tree representation with tree-lstm. we first generate the dependency parse tree with stanford corenlp @xcite for english instructions and stanza toolkit @xcite for hindi and telugu instructions. since the tree-structure is invariant to the children's order (i.e., switching the order of the children of a node doesn't change the tree structure), directly using a tree-lstm over an embedding layer may lose important word order information in the instruction. thus, here we use a bidirectional lstm with an embedding layer to generate word representations that preserve sequential information of the instruction. specifically, given an instruction {w i } l @xmath0, we generate syntaxaware representation {û dep i } l @xmath1: visual encoder and navigation decoder. given panoramic features {f t,p } 36 @xmath2,k } k @xmath3, we picks the next viewpoint from k navigable locations as: ## conclusion in this paper, we presented a tree-based encoder module that incorporates phrase-level information from parse trees. we demonstrated that syntax information can help the agent learn a better alignment between instruction and visual scenes, and generalize better to unseen environments. our experiments on room-to-room dataset and room-across-room dataset both suggest that incorporating syntax information encoded by our tree-based encoder module can significantly improve the performance over baseline vln models.