On the Evaluation of Vision-and-Language Navigation Instructions (2021)

Abstract: Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions. However, existing instruction generators have not been comprehensively evaluated, and the automatic evaluation metrics used to develop them have not been validated. Using human wayfinders, we show that these generators perform on par with or only slightly better than a template-based generator and far worse than human instructors. Furthermore, we discover that BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. To improve instruction evaluation, we propose an instruction-trajectory compatibility model that operates without reference instructions. Our model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. For ranking instruction generation systems, if reference instructions are available we recommend using SPICE.

## introduction generating route instructions is a long studied problem with clear practical applications @xcite . whereas earlier work sought to create instructions for human wayfinders, recent work has focused on using instruction-generation models to improve the performance of agents that follow instructions given by people. in the context of vision-and-language navigation (vln) datasets such as room-to-room (r2r) @xcite , models for generating navigation instructions have improved agents' wayfinding performance in at least two ways: (1) by synthesizing new instructions for data augmentation @xcite , and (2) by fulfilling the role of a probabilistic speaker in a pragmatic reasoning setting @xcite . such data augmentation is so effective that it is nearly ubiquitous in the best performing agents @xcite @xcite . to make further advances in the generation of visually-grounded navigation instructions, accurate evaluation of the generated text is essential. however, the performance of existing instruction generators has not yet been evaluated using human wayfinders, and the efficacy of the automated evaluation metrics used to develop them has not been established. this paper addresses both gaps. to establish benchmarks for navigation instruction generation, we evaluate existing english models @xcite using human wayfinders. these models are effective for data augmentation, but in human trials they perform on par with or only slightly better than a templatebased system, and they are far worse than human instructors. this leaves much headroom for better instruction generation, which may in turn improve agents' wayfinding abilities. next, we consider the evaluation of navigation instructions without human wayfinders, a necessary step for future improvements in both grounded instruction generation (itself a challenging and important language generation problem) and agent wayfinding. we propose a model-based approach (fig. 1 ) to measure the compatibility of an instruction-trajectory pair without needing reference instructions for evaluation. in training this model, we find that adding contrastive losses in addition to pairwise classification losses improves auc by 9-10%, round-trip back-translation improves performance when used to paraphrase positive examples, and that both trajectory and instruction perturbations are useful as hard negatives. finally, we compare our compatibility model to common textual evaluation metrics to assess which metric best correlates with the outcomes of human wayfinding attempts. we discover that bleu @xcite , rouge @xcite , meteor @xcite and cider @xcite are ineffective for evaluating grounded navigation instructions. for system-level evaluations with reference instructions, we recommend spice @xcite . when averaged over many instructions, spice correlates with both human wayfinding performance and subjective human judgments of instruction quality. when scoring individual instructions, our compatibility model most closely reflects human wayfinding performance, outperforming bertscore @xcite and vln agent-based scores. our results are a timely reminder that textual evaluation metrics should always be validated against human judgments when applied to new domains. we plan to release our trained compatibility model and the instructions and human evaluation data we collected. recent work has focused on instruction generation to improve the performance of wayfinding agents. two instruction generators, speaker-follower @xcite and @xcite , have been widely used for r2r data augmentation. they provide ∼170k new instruction-trajectory pairs sampled from training environments. both are seq-to-seq models with attention. they take as input a sequence of panoramas grounded in a 3d trajectory, and output a textual instruction intended to describe it. vision-and-language navigation for vln, embodied agents in 3d environments must follow natural language instructions to reach prescribed goals. most recent efforts (e.g., @xcite @xcite have used the room-to-room (r2r) dataset @xcite , which contains 4675 unique paths in the train split, 340 in the valseen split (same environments, new paths), and an additional 783 paths in the val-unseen split (new environments, new paths). however, our findings are also relevant for similar datasets such as touchdown @xcite , cvdn @xcite , reverie @xcite , and the multilingual room-across-room (rxr) dataset @xcite ). text generation metrics there are many automated metrics that assess textual similarity; we focus on five that are extensively used in the context of image captioning: bleu @xcite , meteor @xcite , rouge @xcite , @xcite and spice @xcite . more recently, model-and semi-model-based metrics have been proposed. bertscore @xcite takes a semi-model-based approach to compute token-wise similarity using contextual embeddings learned with bert @xcite . bleurt @xcite ) is a fully model-based approach combining large-scale synthetic pretraining and domain specific finetuning. however, all of the aforementioned metrics are reference-based, and none is specifically designed for assessing navigation instructions associated with 3d trajectories for an embodied agent, which requires not only languageto-vision grounding but also correct sequencing. ## human wayfinding evaluations to benchmark the current state-of-the-art for navigation instruction generation, we evaluate the outputs of the speaker-follower and envdrop models by asking people to follow them. we use instructions for the 340 and 783 trajectories in the r2r val-seen and val-unseen splits, respectively. both models are trained on the r2r train split and the generated instructions were provided by the respective authors. to contextualize the results, we additionally evaluate instructions from a template-based generator (using ground-truth object annotations), a new set of instructions written by human annotators, and three adversarial perturbations of these human instructions. new navigation instructions and wayfinding evaluations are collected using a lightly modified version of pangea 1 , an open-source annotation toolkit for panoramic graph environments. crafty crafty is a template-based navigation instruction generator. it observes the trajectory's geometry and nearby ground-truth object annotations, identifies salient objects, and creates english instructions using templates describing movement with respect to the trajectory and objects. see the appendix for details. note that crafty has an advantage over the learned models which rely on panoramic images to identify visual references and do not exploit object annotations. ## compatibility model as an alternative to human evaluations, we train an instruction-trajectory compatibility model to assess both the grounding between textual and visual inputs and the alignment of the two sequences. ## experiments we evaluate our compatibility model against alternative model-based evaluations and standard textual similarity metrics. we report instruction classification results in section 5.1, improved data augmentation for vln agents in section 5.2, and correlation with human wayfinder outcomes in 5.3. ## conclusion generating grounded navigation instructions is one of the most promising directions for improving the performance of vln wayfinding agents, and a challenging and important language generation task in its own right. in this paper, we show that efforts to improve navigation instruction generators have been hindered by a lack of suitable automatic evaluation metrics. with the exception of spice, all the standard textual evaluation metrics we evaluated (bleu, cider, meteor and rouge) are ineffective, and -perhaps as a result -existing instruction generators have substantial headroom for improvement. to address this problem, we develop an instruction-trajectory compatibility model that outperforms all existing automatic evaluation metrics on instance-level evaluation without needing any reference instructions -making it suitable for use as a reward function in a reinforcement learning setting, as a discriminator in a generative adversarial network (gan) @xcite , or for filtering instructions in a data augmentation setting. progress in natural language generation (nlg) is increasing the demand for evaluation metrics that can accurately evaluate generated text in a variety of domains. our findings are a timely reminder that textual evaluation metrics should not be trusted in new domains unless they have been comprehensively validated against human judgments. in the case of grounded navigation instructions, for model selection in the presence of reference instructions we recommend using the spice metric. in all other scenarios (e.g., selecting individual instructions, or model selection without reference instructions) we recommend using a learned instruction-trajectory compatibility model.