Towards Navigation by Reasoning over Spatial Configurations (2021)

Abstract: We deal with the navigation problem where the agent follows natural language instructions while observing the environment. Focusing on language understanding, we show the importance of spatial semantics in grounding navigation instructions into visual perceptions. We propose a neural agent that uses the elements of spatial configurations and investigate their influence on the navigation agent’s reasoning ability. Moreover, we model the sequential execution order and align visual objects with spatial configurations in the instruction. Our neural agent improves strong baselines on the seen environments and shows competitive performance on the unseen environments. Additionally, the experimental results demonstrate that explicit modeling of spatial semantic elements in the instructions can improve the grounding and spatial reasoning of the model.

## introduction the ability to understand and follow natural language instructions is critical for intelligent agents to interact with humans and the physical world. one of the recently designed tasks in this direction is vision-and-language navigation (vln) @xcite , which requires an agent to carry out a sequence of actions in a photo-realistic simulated environment in response to a sequence of natural language instructions. to accomplish this task, the agent should have three abilities: understanding linguistic semantics, perceiving the visual environment, and reasoning over both modalities @xcite . while understanding vision and language are difficult problems by themselves, learning the connection between them without direct supervision makes this task even more challenging @xcite . to address this challenge, some neural agents establish the connection using attention mechanism to relate the tokens from a given instruction to the images in a panoramic photo @xcite @xcite . surprisingly, although those models can improve the performance, @xcite found they ignore the visual information. there is no clear evidence that the agent can correspond the components of the visual environment to the instructions @xcite . based on these results, recent research started to improve the agent's reasoning ability by explicitly considering the structure of language and image. from the language side, @xcite annotated fine-grained subinstructions and their corresponding trajectories and used the co-grounded features of a part of instruction and the image to predict the next action. from the image side, @xcite induced a high-level object-based visual representation to ground the language into the visual context. in the same direction, we propose a neural agent, namely spatial-configuration-based-navigation (spc-nav), and consider the structure of both modalities, that is, spatial semantics of the instructions and the objects in the images. we use the notion of spatial configuration @xcite to model the instructions and design a state attention to ensure the execution order of spatial configurations. then, we utilize the spatial semantics elements, namely motion indicator, spatial indicator and landmark in spatial configuration to establish the connection with the visual environment. specifically, we use the similarity score between the landmark representation in the spatial configurations and the object representation in the panoramic images to control the transitions between configurations. also, we align object representations with the configuration representations enriched with motion indicator, spatial indicator and landmark representations to finally select the navigable image. a spatial configuration is the smallest linguistic the instruction "move to the table with chair, and stop." can be split into two spatial configurations: "move to the table with chair" and "stop". in configuration1, "move" is motion indicator; "to" is spatial indicator; "table" is landmark. "table with chair" is a nested spatial configuration of configuration1. the role of "table" is trajector; "with" is spatial indicator; and "chair" is landmark. in configuration2, "stop" is motion indicator. older studies on navigation before the deep learning era are mostly symbolic grounding methods, which are based on parsing the semantics of the instruction and learning probabilistic models. macmahon et al. (2006) used the parser to associate the linguistic elements in free-form instruction to their corresponding action, location and object in the environment. tellex et al. (2011) represented the spatial language as a hierarchy of spatial description clauses (sdc) and proposed a discriminative probabilistic graphical model to find the most probable path with the extracted sdc and the detected visual landmark. mei et al. (2016) provided a good overview of the past classical work on navigation. however, one of the biggest limitations of those methods is that they required prior linguistic structure and manual annotations. in recent years, given the new capabilities created by deep learning architectures, the navigation task is extended to the photo-realistic simulated environments @xcite @xcite . based on this, a sequence-to-sequence (seq2seq) baseline model was proposed by @xcite to encode the instructions and decode the embeddings to identify the corresponding output action sequence with the observed images. @xcite proposed to train a speaker model to augment the instructions for the follower model. @xcite introduced a visual and textual co-attention mechanism and a progress monitor loss to track the execution progress. although those agents achieved better performance, the semantic structures on both language and vision sides were ignored. we aim to exploit both symbolic grounding and neural models in the spatial domain. @xcite designed the neurons to learn the meaning of spatial prepositions. @xcite explored the effects of spatial knowledge in a generative neural language model for the image description. we mainly work on incorporating the spatial semantics in navigation neural agent. hong et al. (2020) recently provided a method to segment the long instruction into sub-instructions. they used a shifting attention module to infer whether the current sub-instruction has been completed. subinstructions differ from us as they manually aligned the instructions and viewpoints to learn the alignments, while we modeled spatial semantics to guide the alignment automatically. moreover, their proposed shifting attention module is hard attention, and a threshold is set to decide whether the agent should execute the next sub-instruction. however, we utilize the grounding between the landmarks and the objects to control the transitions between sub-instructions. ## experimental setup dataset we evaluate our model with room-to-room (r2r) dataset @xcite , which is built upon the matterport3d dataset @xcite . spl is recommended as the primary metric because it considers both the effectiveness and efficiency of navigation performance. ## results and analysis training strategies are orthogonal to our work, and our model is friendly to the strategies widely used in the literature (t1/t2) (see section 3.8). we evaluate spc-nav with both t1 and t2 and compare the results with their baseline models as well as sub-instruction. we do not apply data augmentation in this setting. as shown in ## conclusion we propose a neural agent that incorporates the semantic elements of spatial language for visionand-language navigation. we use the notion of spatial configurations as the main linguistic unit of the instructions and enhance the spatial configuration representation with the representations of motion indicator and landmark. we design a state attention to guarantee the sequential execution order of configurations and use the similarity score between the representations of landmarks and objects to control the transitions between configurations. based on our results, incorporating the spatial semantics improves reasoning ability over navigation. future work could investigate more fine-grained spatial semantics and the geometry of spatial relations. also, we will deal with novel objects in a zero-shot setting to improve the unseen environments results.