Diagnosing Vision-and-Language Navigation: What Really Matters (2022)

Abstract: Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Multiple setups have been proposed, and researchers apply new model architectures or training techniques to boost navigation performance. However, there still exist non-negligible gaps between machines’ performance and human benchmarks. Moreover, the agents’ inner mechanisms for navigation decisions remain unclear. To the best of our knowledge, how the agents perceive the multimodal input is under-studied and needs investigation. In this work, we conduct a series of diagnostic experiments to unveil agents’ focus during navigation. Results show that indoor navigation agents refer to both object and direction tokens when making decisions. In contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. Transformer-based agents acquire a better cross-modal understanding of objects and display strong numerical reasoning ability than non-Transformer-based agents. When it comes to vision-and-language alignments, many models claim that they can align object tokens with specific visual targets. We find unbalanced attention on the vision and text input and doubt the reliability of such cross-modal alignments.

## introduction a key challenge for artificial intelligence (ai) research is to move beyond independent and identically distributed (i.i.d.) data analysis: we need to teach ai agents to understand multimodal input data, and jointly learn to reason and perform incremental and dynamic decision-making with the help from humans. vision-and-language navigation (vln) has received much attention due to its active perception and multimodal grounding setting, dynamic decision-making nature, rich applications, ## 53 17 various setups have been proposed to address to the vln task. researchers generate visual trajectories and collect human-annotated instructions for indoor @xcite @xcite and outdoor environment @xcite @xcite . there are also interactive vln settings based on dialogues @xcite @xcite , and task that navigates agents to localize a remote object @xcite . however, few studies ask the why and how questions: why do these agents work (or do not work)? how do agents make decisions in different setups? through the years, agents with different model architectures and training mechanisms have been proposed for indoor vln @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite and outdoor vln @xcite @xcite @xcite . back-translation eases the urgent problem of data scarcity @xcite . imitation learning and reinforcement learning enhance agents' generalization ability @xcite . with the rise of bert-based models, researchers also apply transformer and pre-training to further improve navigation performance @xcite @xcite . while applying new techniques to the navigation agents might boost their performance, we still know little about how agents make each turning decision. treatment of the agents' processing of instructions and perception of the visual environment as a black box might hinder the design of a generic model that fully understands visual and textual input regardless of vln setups. therefore, we focus on analyzing how the navigation agents understand the multimodal input data in this work. we conduct our investigation from the perspectives of natural language instruction, visual environment, and the interpretation of vision-language alignment. we create counterfactual interventions to alter the instructions and the visual environment in the validation dataset, focusing on variables related to objects, directions and numerics. more specifically, we modify the instruction by removing or replacing the object/direction/numeric tokens, and we adjust the environment by masking out visual instances or horizontally flipping the viewpoint images. subsequently, we examine the interventions' treatment effects on agents' evaluation performance while keeping other variables unchanged. we set up experiments on the r2r @xcite and the rxr dataset @xcite for indoor vln and the touchdown dataset @xcite for outdoor vln. we examine nine vln agents on the three datasets with quantitative ablation diagnostics on the text and visual inputs. in summary, our key findings include: ## related work instruction following is a long-standing topic in ai studies that ask an agent to follow natural language instructions and accomplish target tasks, which can be dated back to the shrldu @xcite . efforts made to tackle this classic problem spans from defining templates @xcite , designing hardencoded concepts to ground visual attributes and spatial relations @xcite @xcite @xcite , to constructing varies datasets and learning environments @xcite @xcite . many methods have been proposed to map the instructions into sequence of actions, such as reinforcement learning @xcite @xcite , semantic parsing @xcite , alignment-based model @xcite , and neural networks @xcite @xcite . vision-and-language navigation is a task where an agent comprehends the natural language instructions and reasons through the visual environment. many studies aim at improving vln agents' performance in one way or another. to enrich training data, a line of work @xcite use back-translation to generate augmented instructions. to enforce cross-modal grounding, rpa and rcm @xcite use reinforcement learning, smna @xcite uses a visual-textual co-grounding module to improve cross-modal alignment, rel-graph @xcite uses graphs for task formulation. to address the generalizability problem to unseen environment, press @xcite introduces a stochastic sampling scheme, envdrop @xcite proposes environment dropout. to utilize visual information from the environment, auxrn @xcite uses auxiliary tasks to assist semantic information extraction, vln-hamt @xcite incorporates panorama history with a hierarchical vision transformer. fast @xcite makes use of asynchronous search and allows the agent to backtrack if it discerns a mistake after attending to global and local knowledge. with the success of bertrelated models in nlp, researchers also start to build transformer-based navigation agents and add a pre-training process before fine-tuning on the downstream vln task @xcite @xcite . ## background and research questions we first bring in the task of vision-and-language navigation and introduce the datasets and agents used for comparison. then we list out the research questions to study in this work. ## analysis on instruction understanding this section examines whether and to what extent the agent understands vln instructions. we focus on how the agent perceives object-related tokens, direction-related tokens, and numeric tokens, and their effects on final navigation performance. we record the validation score in the "vanilla" setting, and report the relative performance change for each ablation setting. for object-token ablations, the "mask" setting masks out all the object-tokens, while the controlled trial masks out the same amount of random tokens. the same applies to direction-token ablations. ## analysis on visual environment this section investigates what the agent perceives in the visual environment. we set an eye on inspecting the agent's understanding of the surrounding objects and direction-related information. ## analysis on vision-language alignment this section examines the agents' ability to learn vision-language alignment when executing the navigation. we focus on whether the agents can understand the objects mentioned in the instruction and align them to the correct visual instance in the environment, which is crucial to completing this multimodal task. to verify the existence of visionlanguage alignment, we add perturbations to the visual and textual input, and check how they affect agents' performance. ## conclusion in this paper, we inspect how the navigation agents understand the multimodal information by conducting ablation diagnostics input data. we find out that indoor navigation agents refer to both object tokens and direction tokens in the instruction when making decisions. in contrast, outdoor navigation agents heavily rely on direction tokens and poorly understand the object tokens. when it comes to vision-and-language alignments, we witness unbalanced attention on text and vision, and doubt the reliability of cross-modal alignments. we hope this work encourages more investigation and research into understanding neural vln agents' black-box and improves the task setups and navigation agents' capacity for future studies.