Aerial Vision-and-Dialog Navigation (2023)

Abstract: The ability to converse with humans and follow natural language commands is crucial for intelligent unmanned aerial vehicles (a.k.a. drones). It can relieve people’s burden of holding a controller all the time, allow multitasking, and make drone control more accessible for people with disabilities or with their hands occupied. To this end, we introduce Aerial Vision-and-Dialog Navigation (AVDN), to navigate a drone via natural language conversation. We build a drone simulator with a continuous photorealistic environment and collect a new AVDN dataset of over 3k recorded navigation trajectories with asynchronous human-human dialogs between commanders and followers. The commander provides initial navigation instruction and further guidance by request, while the follower navigates the drone in the simulator and asks questions when needed. During data collection, followers’ attention on the drone’s visual observation is also recorded. Based on the AVDN dataset, we study the tasks of aerial navigation from (full) dialog history and propose an effective Human Attention Aided Transformer model (HAA-Transformer), which learns to predict both navigation waypoints and human attention.

## introduction drones have been widely adopted for many applications in our daily life, from personal entertainment to professional use. it has the advantage of mobility and observing large areas over ground robots. however, compared with ground robots, the control of the aerial robot is more complex because an extra degree of freedom, altitude, is involved. to control a drone, people often need to hold a controller all the time, so it is essential to create a hands-free control experience for drone users and develop an intelligent drone that can complete tasks simply by talking to humans. it can lower the barrier of figure 1 : an example of aerial vision-and-dialog navigation (avdn). the user instructs the agent to fly to a destination. during the navigation, the agent can ask questions while showing the images of past visual observations and relative trajectory. the user will talk back at a convenient time to provide further guidance to the agent without having to monitor the agent all the time. therefore, this work introduces aerial visionand-dialog navigation (avdn), aiming to develop an intelligent drone that can converse with its user to fly to the expected destination. as shown in figure 1 , the user (commander) provides instructions, and the aerial agent (follower) follows the instruction and asks questions when needed. the past visual trajectories are also provided along with the question, which frees the commander from monitoring the drone all the time and minimizes the burden of drone control. in this free-form dialog, potential ambiguities in the instruction can be gradually resolved through the further instructions provided by the commanders upon request. to implement and evaluate the avdn task, we build a photorealistic simulator with continuous state space to simulate a drone flying with its onboard camera pointing straight downward. then we collect an avdn dataset of 3,064 aerial navigation trajectories with human-human dialogs, where crowd-sourcing workers play the commander role and drone experts play the follower role, as illustrated in figure 1 . moreover, we also collect the attention of human followers over the aerial scenes for a better understanding of where humans ground navigation instructions. based on our avdn dataset, we introduce two challenging navigation tasks, aerial navigation from dialog history (andh) and aerial navigation from full dialog history (andh-full). both tasks focus on predicting navigation actions that can lead the agent to the destination area, whereas the difference is that andh-full presents the agent with full dialog and requires it to reach the final destination @xcite , while andh evaluates the agent's completion of the sub-trajectory within a dialog round given the previous dialog information @xcite . the proposed tasks open new challenges of sequential action prediction in a large continuous space and natural language grounding on photorealistic aerial scenes. we propose a sequenceto-sequence human attention aided transformer model (haa-transformer) for both tasks. the haa-transformer model predicts waypoints to reduce the complexity of the search space and learns to stop at the desired location. more importantly, it is jointly trained to predict human attention from the input dialog and visual observations and learns where to look during inference. experiments on our avdn dataset show that multitask learning is beneficial and human attention prediction improves navigation performance. the main contributions are concluded as follows: vision-and-language navigation vision-and-language navigation (vln) is an emerging multimodal task that studies the problem of using both language instructions and visual observation to predict navigation actions. we compare some of the works with our avdn dataset in aerial navigation using both vision and language for aerial navigation is a less studied topic, whereas vision-only aerial navigation for drones is dataset env photorealistic continuous space dialog free form r2r, rxr indoor ✓ ✗ ✗ ✓ vnla, hanna indoor ✓ ✗ ✓ ✗ vln-ce indoor ✗ ✓ ✗ ✓ cvdn indoor ✓ ✗ ✓ ✓ alfred indoor ✗ ✗ ✗ ✓ dialfred indoor ✗ ✗ ✓ ✗ teach indoor ✗ ✓ ✓ ✓ touchdown street-view ✓ ✗ ✗ ✓ modified lani aerial ✗ ✓ ✗ ✓ avdn (ours) aerial ✓ ✓ ✓ ✓ as a result, it allows more diverse and complex navigation and also resolves ambiguities during complicated navigation. ## dataset the avdn dataset includes dialogs, navigation trajectories, and the drone's visual observation with human attention, where an example is shown in figure 2 . with the help of a newly proposed simulator, we record the avdn trajectories created by two groups of humans interacting with each other, playing either the commander role or the follower role. our avdn dataset is the first aerial navigation dataset based on dialogs to the best of our knowledge. ## task following indoor dialog navigation @xcite , we introduce an aerial navigation from dialog history (andh) task and an aerial navigation from full dialog history (andh-full) task based on our avdn dataset and simulator. ## model we proposed a human attention aided (haa-transformer) model for the andh and andh-full tasks as shown in figure 4 , where it takes as input multimodal information and generates multimodal predictions, including human attention prediction and navigation prediction. multimodal encoding the input has three modalities, the drone's direction, images from the drone's visual observation, and history dialogs. at the start of a prediction series, our model uses a bert encoder @xcite to get the language embeddings of the input dialog history, h l 1:l , where special language tokens such as @xcite and @xcite are added in front of each instruction and question in the dialog. then, at every time step, all previous drone directions and images from the drone's visual observation are input to the model. a fully connected direction encoder is used to generate direction embeddings h x 1:t and an xview-pretrained darknet-53 2 @xcite with an attention module is used to extract and flatten the visual features to get visual embeddings h v 1:t . finally, similar to the episodic transformer @xcite , all embeddings from the languages, images and directions, are concatenated and input into a multimodal transformer (f m t ) to produce output multimodal embeddings {z l 1:l , z v 1:t , z x 1:t } as in equation 2 . ## results we conduct experiments to study our avdn dataset and our haa-transformer model on the andh and andh-full tasks. results on the andh task and andh-full task as shown in ## conclusion in this work, we introduce a dataset and a simulator for aerial vision-and-language navigation (avdn). challenging tasks are proposed based on our dataset focusing on navigation. a human attention aided multimodal transformer (haa-transformer) model is designed for both tasks. our work provides the possibilities for further studies to develop stronger models on avdn that not only focus on navigation prediction but also on question generation. furthermore, based on our results, future works may investigate using human attention prediction training to help solve vln problems.