Action Inference for Destination Prediction in Vision-and-Language Navigation (2024)

Abstract: Vision-and-Language Navigation (VLN) encompasses interacting with autonomous vehicles using language and visual input from the perspective of mobility.Most of the previous work in this field focuses on spatial reasoning and the semantic grounding of visual information.However, reasoning based on the actions of pedestrians in the scene is not much considered.In this study, we provide a VLN dataset for destination prediction with action inference to investigate the extent to which current VLN models perform action inference.We introduce a crowd-sourcing process to construct a dataset for this task in two steps: (1) collecting beliefs about the next action for a pedestrian and (2) annotating the destination considering the pedestrian’s next action.Our benchmarking results of the models on destination prediction lead us to believe that the models can learn to reason about the effect of the action and the next action on the destination to a certain extent.However, there is still much scope for improvement.

## introduction the widespread belief is that autonomous vehicles and mobility services will become commonplace on the roads. among methods being investigated as a means for humans to interact with these devices, one of the most intuitive approaches is to use language. vision-and-language navigation (vln) is a task in which navigation instructions are given in free-form language based on visual information to an autonomous vehicle or mobility. although there are two broad variations of vln, i.e. outdoor and indoor, we focus on outdoor vln to tackle challenges for autonomous vehicles and mobility services. solving outdoor vln tasks requires spatial and semantic grounding of the instructions and considerable research has been done for vln @xcite @xcite . however, none of these works delve into how the actions of pedestrians affect the navigation decisions of the vehicle. we introduce a vln task of destination prediction for picking up a pedestrian i.e. user of the vehicle in the scene that requires action inference. we define action inference as how the actions and beliefs of the next actions performed by the user in the near future affect the destination to pick up the user. for example in figure 1 , we can see how belief in the next action affects where to pick up the user. to create a dataset annotated with action and next action in a scalable way, we propose annotation processes in two steps shown in figure 2 . in the first step of annotation, we collect the next action knowledge about the likely near future, and in the second step, we collect the destination predictions based on the actions and the next actions. we test the models used in destination prediction to check whether they can reason about the near future. our results show that models with insertion of the action and the next action perform better than those without any knowledge insertion. however, since the performance is still quite low, there is room for further improvement. our main contributions in this paper are: ## related work streetnav (hermann et al., 2020) focuses on using navigation system style street-view hence instructions focus on grounding street names and using spatial cues. in touchdown @xcite , the instructions help an agent find an object hidden randomly in a street view map, focusing on 3-dimensional instructions along with more variations of words. talk2nav @xcite improves on these datasets by using landmark-based instructions, but it ends up focusing on identifying the landmarks through the use of different ways of referring to them. all of these datasets end up focusing on immovable items such as landmarks or streets instead of taking advantage of the dynamicity of outdoor scenes. talk2car @xcite has more conversation-style instructions to refer to an object in the scene, but this leads to phrases where the object is mostly present directly in dialogues without any particular consideration given to actions performed in the scene. talk2car-regseg @xcite is a closely associated work to ours and they try to define the destination to navigate to based on the annotation instructions in talk2car. however, in contrast to our work, they do not provide any insights into how destinations are affected by action inference. titan @xcite meanwhile has extensive labels for action performed by pedestrians in real-world scenes, however unlike our dataset they do not provide any linguistic instructions for the mobility to navigate in the scene. ## data collection we discuss the details of our annotation steps for collecting data for destination prediction using action inference. we start by explaining the data preprocessing we use for data annotated with user actions. then we explain the two annotation steps for the next action and destination prediction using action knowledge shown in figure 2 . we additionally collect information regarding the attributes present in our data detailed in the final subsection. ## experiments because of the associated task proximity, we picked up the destination prediction models executed in talk2car-regseg @xcite . the authors of talk2car-regseg propose finding the destination road segment on the image that the user wants the mobility to move to based on a reference expression. they propose two model variations in their study based on the variation used on how to combine multimodal features. one variation is a transformer-based grounding model for combining the multimodal features, and another is a fully convolutional network-based model. we use both these variations to benchmark our data by finetuning and testing. see detailed model settings and parameters used while fine-tuning in appendix c. since the models are based on finding the destination based on reference expression, we use manually created templates to generate reference expressions based on action and next action knowledge. the advantage of templates is that we control which type c i s all action knowledge 19.4 12.4 9.1 next action only 14.7 9.9 7.8 action only 11.5 10.6 11.9 we choose a strategy of fine-tuning and testing the pre-trained versions of the models in talk2car-regseg. we split the 294 images into 236 images for seen fine-tuning data and 58 images for unseen test data. the seen split used for fine-tuning is again divided into 80% for training and 20% for validation. we use three different comparison metrics for our experiments. accuracy refers to the pointing game score as defined by the authors of talk2car-regseg @xcite . according to this definition, accuracy is when the point with the highest likelihood in the output mask is in the ground truth. pointing game scores can be justified based on the general trend of autonomous mobility control algorithms being able to navigate based on single points and not needing an entire segment of navigation points. secondly, recall@100 can be defined as whether at least one of the 100 top likelihood points is in the ground truth. finally, for intersection over union (iou) we use the standard definition of intersection area of predicted and ground truth segments divided by the union of the area of the two segments. we modify all three of the above accuracy scores to give output one when the model correctly assigns that none of the pixels has greater than the threshold accuracy in case of no destination. ## results from ## conclusion in this work, we created a new vln destination prediction dataset for a vehicle to pick up a pedestrian. this dataset focuses on how the actions of the pedestrian and the next actions the pedestrian is likely to do in the near future affect the destination of the vehicle, which we define as action inference. we also provide attributes of the relationship between the action and the next action along with reasoning about how the difference in the next actions affects the overlap of destinations. our experiments on fine-tuning pre-trained destination prediction models resulted in a higher action inference accuracy when action knowledge is present in the instruction phrase. this indicates that models can learn to reason about action knowledge to a certain extent. however, we see a drop of 50% when we compare the test accuracy on our dataset compared to the test accuracy on the pretraining dataset. we also observe underwhelming performance in cases where the next action variation causes a low overlap of destinations. both of these results lead us to believe that there is still scope for improvement. in our future work, we would like to work on creating architectures that could better handle action inference.