Follow the Beaten Path: The Role of Route Patterns on Vision-Language Navigation Agents Generalization Abilities (2025)

Abstract: Vision and language navigation (VLN) is a challenging task towards the creation of embodied agents that requires spatial and temporal reasoning over the instructions provided in natural language and aligning them with the visual perception of an environment. Although a number of methods and approaches have been developed, none achieves human level performance in outdoor settings (by up to 75 percent). The contributions of visual and language modalities to the success of VLN have been studied, however here we focus on an overlooked property of routes and show that navigational instructions can be represented as patterns of actions that also describe trajectory shapes. Through carefully crafted experiments, we show that agents generalization to unseen environments depends not only on visual and linguistic features, but also on the shape of trajectories presented to the model during the fine-tuning. Our experiments show that the diversity of patterns of actions during training is a key contributor to high success rates for agents. Last, we propose a solution based on data augmentation that fills the gap in missing patterns of training data. Our findings will guide researchers towards improved practices in the development and evaluation of VLN datasets and agents.

## introduction vision-language navigation (vln) is a challenging research area that combines computer vision and natural language processing to enable embodied agents to navigate and understand their environment based on instructions provided in natural language. a typical solution to solving this problem is to train neural network architectures such as lstm @xcite and transformers (schumann and riezler, 2022) from scratch. in contrast, using llms facilitates the development of modular 1 https://github.com/kbaghaei/beaten_path_vln navigation text of route 641 you should be facing the correct direction when you load in. begin by moving forward until you reach an intersection, and then take a right. reorient yourself and take a right at the next intersection. reorient yourself again and move forward though the next intersection. stop three screens after this intersection. if you turn to the left slightly, there should be a traffic barrel near a shopping cart, which is in front of a red car. the pattern of actions: forward, right, forward, right, forward, stop. agents @xcite @xcite by taking advantage of reasoning capabilities learned through pre-training. nonetheless, even with llms, a significant gap remains between human-level and agent-based performance when solving vln tasks in outdoor settings @xcite . eliminating such a performance gap requires a better understanding of the contributing factors to the success and failures of the agents. @xcite studied token-level features of instructions and structural features of routes such as heading difference in turns. @xcite focused on junction types for navigation. in this work, we focus on an overlooked property of navigational routes, which we call pattern of actions (pact), which can be understood as the high-level shape of an agent trajectory. figure 1 shows that each navigational pattern has a corresponding pact. to the best our knowledge, pacts as a contributing factor to vln agent performance have been overlooked. we find that the agents' performance on out of sample test sets highly relies on the pattern of actions seen during the training, a phenomenon we call "pattern leakage". we show that removing some patterns of actions from training results in performance degradation in task completion by up to 8.9 percentage points. based on our findings, we propose a data augmentation method to create synthetic training data that cover the missing patterns in the human generated dataset. our contributions can be summarized as follows: 1. we show that as an intrinsic feature of navigational trajectories, navigational patterns play an important role in model performance. this is reflected in success of the model in navigating routes with similar patterns, even with instructions that are from another routes. we introduce new data splits, where train and test data are separated to minimize pattern leakage, and show that agents largely fail on pattern generalization. 2. successful vln, beyond navigation, requires intermediate sub-tasks such as initial orientation, and stopping at the right point along the navigational path. we perform an in-depth analysis, comparing the fine-tuned agents on different data splits, showing that not observing patterns during training also deteriorates the agents' performance on these sub-tasks by up to 10 percentage points in f1 scores. 3. we explore a data augmentation method to create synthetic training data that makes up for the missing patterns in the human generated data, showing promising initial results. ## vision and language navigation a navigation task can be defined as following instructions provided in natural language in order to ground a destination point within a given environment @xcite . a navigation instruction @xmath0, w 2 , ..., w n ) is a sequence of words in natural language that describes a navigation route @xmath1, n 2 , ..., n m ). each navigational route consists of multiple nodes in the navi-gational graph of the environment. each node n i in the navigational graph also has visual information v i (a 360-degree panorama image). in each round of navigation, a vln agent starts at an initial state s 1 and according to the instruction l and visual observation v 1 predicts a navigational action from the action space of {forward, left, right, turn_around, stop}. after taking the action, it moves to another state, obtains another visual observation, and predicts a navigational action again. this loop continues until the agent decides the action stop, or it runs out of action limit. a navigation is considered successful if the agent stops within one node distance of the destination point. ## patterns of actions patterns of actions (pact) can be considered "principal components" of navigation trajectories. consider the navigation instruction shown in figure 1 , and notice that it consists of largely two components: (a) directional information at key points where the agent should make turns, and (b) description of forward movements. the instructions also contain several references to landmarks. we define pacts as the sequence of actions a 1 , a 2 , ..., a n of an agent, where any consecutive sequence of forward is represented as one single forward action. this way, this pact stands for an abstract representation of the trajectory, capturing the ground truth actions at key decision points. for example, as depicted in figure 1 (bottom), the navigational text can be summarized using the following pact: forward, right, forward, right, forward, stop. although moving forward might mean either one block or several kilometers, such a sequence of actions at key points can represent the structure of a navigational route. for brevity, we will represent each unique pattern with a hash, with the above example represented as frfrfs. figure 1 also shows the actual route 641 on a map. a pact effectively also describes the shape of a trajectory. therefore, throughout the paper we use the phrases shape of trajectory and pattern of actions/pact interchangeably to emphasize the similarity of routes whose patterns of ground truth actions are equal. given our definition of pact, the datasets available in the literature share common pacts. in this work, we base our analysis on those pacts and perform experiments and ablation studies to show the contribution of pattern leakage in agents' performance. while bias of vln models in prediction of actions has been studied before @xcite , to the best of our knowledge, this is the first analysis of vln approaches and datasets using such a pattern based approach. ## experiments and results we are interested in the generalization ability of agents with respect to the patterns presented to the model during training. to this end, we split the datasets into train and test sets based on patterns, fine-tune the models on these splits, and discuss the results. both our datasets, touchdown and map2seq, have only a limited number of unique patterns (pacts) of 85 and 63, respectively. ## related work navigation of environments by visual perception and language instruction is a well studied problem for both indoor @xcite and outdoor settings @xcite @xcite . model-wise, various methods have been proposed to perform this task using lstms @xcite , transformer based models @xcite , and llm-based agents @xcite . from a dataset perspective, navigational routes do not present enough diversity to models as they are sampled using shortest path algorithms @xcite . this presents biases in action space that leads to models failing to generalize to unseen routes @xcite . we study such a bias through the lens of sequence of actions that a llm predicts and how the patterns of sequences of actions affect the models' performance. due to high cost of human annotation, fried et al. (2018), yu et al. (2020a), and wang et al. (2023b) propose methods to create synthetic data to add more diversity in the distribution of samples in the training data. a more in-depth discussion of the related work is available in appendix 6. ## conclusion our evaluation of llm-based vision and language navigation agents shows that navigation instructions contain an abstract representation of the shape of a trajectory, which captures the pattern of actions an agent must take to perform the navigation task. using this patterns as the basis of our evaluation, we show that vln agents' are less likely to generalize to routes whose patterns are not present in training data. we find that using diverse patterns during the training phase improves the agents' performance. we additionally propose a simple data augmentation method that has the potential to improve the agents' task completion rate. a more comprehensive study of data augmentation is required in future work. also, the effect of pacts on indoor environments should be analyzed, as our study focuses solely on outdoor settings. another track of study is whether the pattern of actions in instructions, affect performance of vln models in continuous environments. our suggestion for the development of new datasets for vln is to generate navigational routes with a higher diversity of patterns of actions to improve performance, and to consider this variable when evaluating vln agents and their generalization capabilities.