G es N avi: Gesture-guided Outdoor Vision-and-Language Navigation (2024)

Abstract: Vision-and-Language Navigation (VLN) task involves navigating mobility using linguistic commands and has application in developing interfaces for autonomous mobility. In reality, natural human communication also encompasses non-verbal cues like hand gestures and gaze. These gesture-guided instructions have been explored in Human-Robot Interaction systems for effective interaction, particularly in object-referring expressions. However, a notable gap exists in tackling gesture-based demonstrative expressions in outdoor VLN task. To address this, we introduce a novel dataset for gesture-guided outdoor VLN instructions with demonstrative expressions, designed with a focus on complex instructions requiring multi-hop reasoning between the multiple input modalities. In addition, our work also includes a comprehensive analysis of the collected data and a comparative evaluation against the existing datasets.

## introduction with the recent successes of autonomous mobilities, there has been an interest in developing interfaces to interact with such systems, leading to the rise of the vision-and-language navigation (vln) task. however, all the outdoor vln tasks still consider verbal instructions as the only interface for communicating with the mobility @xcite . in reality, humans communicate with each other in their daily lives by using non-verbal cues like gestures as well. to allow the freedom of using this intuitive form of communication through gestures, there have been recent efforts to create datasets incorporating pointing gestures as well as an interface for communication in human-robot interaction (hri) systems @xcite . however, these datasets are designed for indoor referring expression comprehension (rec) tasks and often figure 1 : an example from our gesnavi dataset containing (1) a natural language instruction (text box), and (2) a gestured object (green bounding box) that acts as an intermediate anchor for a multi-hop reasoning instruction to navigate toward (3) the target object (red bounding box) indicated by the instruction. in this work, we tackle a part of the aforementioned challenge by constructing a novel dataset, gesnavi, consisting of instructions with gestureguided demonstrative expressions for an outdoor vln task. we capture images from a simulated environment for a crowded urban neighborhood and crowdsource annotations for free-form linguistic instructions to navigate the mobility. having a simulated environment allowed us to automatically capture images and extract ground truth object labels, resulting in a fast and scalable data collection process. these instructions are also accompanied by a gestured object specified using a bounding box, which is expected to be obtained using a nonverbal gestural modality in practice. the gestured object is referred through demonstratives such as this cone, that tall building, etc., and is used as an intermediate anchor for navigating to the target object. this results in instructions requiring multi-hop reasoning to be performed to locate the target object by considering (1) the relationship between the demonstrative and the scene (detecting an intermediate object) and (2) the relationship between the intermediate object and the target object, as described in figure 1 . furthermore, we also supplement our data with images from another perspective of the same scene, as shown in figure 2 , to facilitate the development of systems that can comprehend multimodal instructions in differing perspectives. this is particularly useful in applications such as service robots where the user is accompanying the mobility robot from outside with a different visual perspective than the mobility's camera. the key contributions of this work are listed below: ## dataset the data collection procedure for our task is divided into two steps. the first step consists of collecting images that capture a wide range of outdoor scenes. then, we collect annotations for the gesture-guided instructions on these images. we will describe each of these steps in detail in the following subsections. ## dataset analyses we collected a total of 3,100 gesture-guided vln instructions on outdoor scenes. the instructions in our dataset comprise a vocabulary of 924 words. since our approach did not rely on templates or impose constraints on linguistic instructions, we were able to capture the commonly used natural language instructions in navigational scenarios, as exemplified in figure 3 . the examples illustrate the wide variety of syntactic and semantic structures present in our instructions. our instructions vary in length from 6 to 34 words, with an average length of 13.1 words. this average length is comparable to the text-only outdoor vln dataset, talk2car (deruyttere et al., 2019), and significantly larger than other gestureguided hri datasets like caesar @xcite and yourefit @xcite , which have average expression lengths of just 5.3 words and 3.7 words, respectively. the longer expressions in our dataset reflect the emphasis on free-form natural instructions requiring complex multi-hop reasoning, in contrast to these earlier works. from the annotated bounding boxes of the gestured and target objects, we determined their labeled class by identifying the ground truth object with the highest intersection over union (iou) overlap. the five most frequent objects used for gesturing and as the target objects are summarized in ## conclusion this work introduces a novel dataset, gesnavi, designed for gesture-guided multimodal interaction with mobility in the context of an outdoor vln task. moreover, in contrast to the prior efforts in the related field of gesture-guided rec tasks, our dataset specifically emphasizes natural free-form instructions that require complex multi-hop reasoning. this is evident from the significantly longer expressions in our dataset compared to the previous works. in the future, this dataset can be expanded to also include the general case of using gestures to refer to multiple objects or a group of objects, rather than a single object in the current setup. another future work includes developing methods to tackle this task, including the current state-of-the-art multimodal architectures for vln @xcite , and evaluating their performances on our gesnavi dataset. it is also worth evaluating how the multimodal llms @xcite perform in our task. it is a particularly challenging task and will require designing models that can effectively combine multimodal information and perform multi-hop reasoning to find the target object.