Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions (2022)

Abstract: A long-term goal of AI research is to build intelligent agents that can communicate with humans in natural language, perceive the environment, and perform real-world tasks. Vision-and-Language Navigation (VLN) is a fundamental and interdisciplinary research topic towards this goal, and receives increasing attention from natural language processing, computer vision, robotics, and machine learning communities. In this paper, we review contemporary studies in the emerging field of VLN, covering tasks, evaluation metrics, methods, etc. Through structured analysis of current progress and challenges, we also highlight the limitations of current VLN and opportunities for future work. This paper serves as a thorough reference for the VLN research community.

## introduction humans communicate with each other using natural language to issue tasks and request help. an agent that can understand human language and navigate intelligently would significantly benefit human society, both personally and professionally. such an agent can be spoken to in natural language, and would autonomously execute tasks such as household chores indoors, repetitive delivery work outdoors, or work in hazardous conditions following human commands (bridge inspection; fire-fighting). scientifically, developing such an agent explores how an artificial agent interprets natural language from humans, perceives its visual environment, and utilizes that information to navigate to complete a task successfully. vision-and-language navigation (vln) @xcite @xcite is an emerging research field that aims to build such an embodied agent that can communicate with humans in natural language and navigate in real 3d environments. vln extends visual navigation in both simulated @xcite and real environments @xcite with natural language communication. as illustrated in figure 1 , vln is a task that involves the oracle (frequently a human), the agent, and the environment. the agent and the oracle communicate in natural language. the agent may ask for guidance and the oracle could respond. the agent navigates and interacts with the environment to complete the task according to the instructions received and the environment observed. meanwhile, the oracle observes the environment and agent status, and may interact with the environment to help the agent. since the development and release of works such as room-to-room (r2r) @xcite , many vln datasets have been introduced. regarding the degree of communication, researchers create benchmarks where the agent is required to passively understand one instruction before navigation, to benchmarks where agents converse with the oracle in free-form dialog. regarding the task objective, the requirements for the agent range from strictly following the route described in the ini-tial instruction to actively exploring the environment and interacting with objects. in a slight abuse of terminology, we refer to benchmarks that involve object interaction together with substantial sub-problems of navigation and localization, such as alfred @xcite , as vln benchmarks. many challenges exist in vln tasks. first, vln faces a complex environment and requires effective understanding and alignment of information from different modalities. second, vln agents require a reasoning strategy for the navigation process. data scarcity is also an obstacle. lastly, the generalization of a model trained in seen environments to unseen environments is also essential. we categorize the solutions according to the respective challenges. (1) representation learning methods help understand information from different modalities. (2) action strategy learning aims to make reasonable decisions based on gathered information. (3) data-centric learning methods effectively utilize the data and address data challenges such as data scarcity. (4) prior exploration helps the model familiarize itself with the test environment, improving its ability to generalize. we make three primary contributions. (1) we systematically categorize current vln benchmarks from communication complexity and task objective perspectives, with each category focusing on a different type of vln task. (2) we hierarchically classify current solutions and the papers within the scope. (3) we discuss potential opportunities and identify future directions. ## tasks and datasets the ability for an agent to interpret natural language instructions (and in some instances, request feedback during navigation) is what makes vln unique from visual navigation @xcite . in communication complexity defines the level at which the agent may converse with the oracle, and we differentiate three levels: in the first level, the agent is only required to understand an initial instruction before navigation starts. in the second level, the agent sends a signal for help whenever it is unsure, utilizing the guidance from the oracle. in the third level, the agent with dialogue ability asks questions in the form of natural language during the navigation and understands further oracle guidance. task objective defines how the agent attains its goal based on the initial instructions from the oracle. in the first objective type, fine-grained navigation, the agent can find the target according to a detailed step-by-step route description. in the second type, coarse-grained navigation, the agent is required to find a distant target goal with a coarse navigation description, requiring the agent to reason a path in a navigable environment and possibly elicit additional oracle help. tasks in the previous two types only require the agent to navigate to complete the mission. in the third type, navigation and object interaction, besides reasoning a path, the agent also needs to interact with objects in the environment to achieve the goal since the object might be hidden or need to change physical states. 2 as with coarse-grained navigation, some object interaction tasks can require additional supervision via dialogue with the oracle. ## evaluation goal-oriented metrics mainly consider the agent's proximity to the goal. the most intuitive is success rate (sr), which measures how frequently an agent completes the task within a certain distance of the goal. goal progress @xcite ## vln methods as shown in figure 2 , we categorize existing methods into representation learning, action strategy learning, data-centric learning, and prior exploration. representation learning methods help agent understand relations between these modalities since vln involves multiple modalities, including vision, language, and action. moreover, vln is a complex reasoning task where mission results depend on the accumulating steps, and better action strategies help the decision-making process. additionally, vln tasks face challenges within their training data. one severe problem is scarcity. collecting training data for vln is expensive and time-consuming, and the existing vln datasets are relatively small with respect to the complexity of vln tasks. therefore, data-centric methods help to utilize the existing data and create more training data. prior exploration helps adapt agents to previously unseen environments, improving their ability to generalize, decreasing the performance gap between seen versus unseen environments. ## related visual-and-language tasks this paper focuses on vision-and-language navigation tasks with an emphasis on photo-realistic environments. 2d map may also be a uesful virtual environment for navigation tasks @xcite @xcite . synthetic environments may also be a substitute for realistic environment @xcite . @xcite propose to instantiate a probabilistic graphical model for natural language commands in robotic navigation and mobile manipulation process. in vln, an agent needs to follow the given instruction and even ask for assistants in human language. an agent in visual navigation tasks is usually not required to understand information from textual modality. visual navigation is a problem of navigating an agent from the current location to find the goal target. researchers have achieved success in both simulated environments @xcite and real environments @xcite . ## conclusion and future directions in this paper, we discuss the importance of vln agents as a part of society, how their tasks vary as a function of communication level versus task objective, and how different agents may be evaluated. we broadly review vln methodologies and categorize them. this paper only discusses these issues broadly at an introductory level. in reviewing these papers, we can see the immense progress that has already been made, as well as directions that this research topic can be expanded on. current methods usually do not explicitly utilize external knowledge such as objects and general house descriptions in wikipedia. incorporating knowledge also improves the interpretability and trust of embodied ai. moreover, currently several navigation agents learn which direction to move and with what to interact, but there is a last-mile problem of vln-how to interact with objects. @xcite asked whether a robot could learn to "bring me a spoon"; new research may ask how a robot can learn to "pick up a spoon". the environments also lack diversity: most interior terrestrial vln data consists of american houses, but never warehouses or hospitals: the places where these agents may be of most use. below we detail additional future directions: collaborative vln current vln benchmarks and methods predominantly focus on tasks where only one agent navigates, yet complicated realworld scenarios may require several robots collaborating. multi-agent vln tasks require development in swarm intelligence, information communication, and performance evaluation. meetup! @xcite is a two-player coordination game where players move in a visual environment to find each other. vln studies the relationship between the human and the environment in figure 1 , yet here humans are oracles simply observing (but not acting on) the environment. collaboration between humans and robots is crucial for them to work together as teams (e.g., as personal assistants or helping in construction). future work may target at collaborative vln between multiple agents or between human and agents. simulation to reality there is a performance loss when transferred to real-life robot navigation @xcite . real robots function in continuous space, but most simulators only allow agents to "hop" through a pre-defined navigation graph which is unrealistic for three reasons @xcite . navigation graphs assume: (1) perfect localization-in the real world it is a noisy estimate;