Hallucination Detection for Grounded Instruction Generation (2023)

Abstract: We investigate the problem of generating instructions to guide humans to navigate in simulated residential environments. A major issue with current models is hallucination: they generate references to actions or objects that are inconsistent with what a human follower would perform or encounter along the described path. We develop a model that detects these hallucinated references by adopting a model pre-trained on a large corpus of image-text pairs, and fine-tuning it with a contrastive loss that separates correct instructions from instructions containing synthesized hallucinations. Our final model outperforms several baselines, including using word probability estimated by the instruction-generation model, and supervised models based on LSTM and Transformer.

## introduction performance of neural-network-based models on generating navigation instructions is substantially inferior to that of humans @xcite . these models often hallucinate, generating references to objects or actions that do not exist or are impossible to execute in the environment. similar behavior has been observed in language models in other domains of text generation @xcite @xcite @xcite . instructions containing hallucinations can confuse or misdirect humans, leading to frustration and sometimes even catastrophic mistakes. detecting hallucinations is therefore essential to improve instruction generation models and inform risk to human users. nevertheless, ground-truth wordlevel hallucination labels are typically not readily available in this domain. meanwhile, hiring crowdworkers to annotate instructions can be very costly @xcite @xcite . we propose a data-efficient weakly supervised approach to hallucination detection. our approach reduces the necessary supervision in two ways. first, we leverage a pre-trained vision-language model @xcite that has learned transferable representations of path-instruction pairs through self-supervised learning. second, we introduce data-augmentation strategies to create synthetic data with "free" hallucination labels. we fine-tune the pre-trained model with the synthesized data using a contrastive learning objective to learn representations that separate positive examples (hallucinations) from negative examples (non-hallucinations). our model outperforms various baselines in terms of f-1 scores on human-annotated evaluation data, beating an lstm-and a transformer-based models by 6.2 and 10.0 points, respectively. ablation studies demonstrate the effectiveness of the proposed self-supervised pre-training and contrastive fine-tuning approach. we release the code, models, and data at https://lingjunzhao. github.io/hallucination_detection.html . ## problem setting grounded instruction generation. our task takes place in an environment, where a speaker model s(u | r) composes an instruction u to communicate an imaginary trajectory r to a follower so that the latter can generate the same trajectory in the environment. an instruction is a sequence of words u i , whereas a trajectory is a sequence of observations o t and actions a t . we employ the matterport3d simulator for experiments @xcite which embeds a follower in a 3d model of a real-world residential building. the observation o t of the follower comprises of an rgb image representing the panoramic view at a location in a building, and orientation features encoding the follower's gaze direction. each action a t moves the follower to a new location close to where it is standing and changes its observation. speaker model. we follow @xcite to train a t5-based @xcite speaker model. this model encodes a trajectory into a sequence of hidden vectors and applies multi-headed attention on those vectors to generate an instruction autoregressively. it is trained on the room-to-room (r2r) dataset provided by the matterport3d simulator. detail about the model is provided in §a.1. hallucination in grounded instruction. instructions generated by our speaker model often contain words that are inconsistent with the input trajectory. we refer to those words as hallucinations. similar to prior work @xcite , we observe two types of hallucinations: we formulate hallucination detection as binary classification: given an input @xmath0, u, i) consisting of a trajectory r, an instruction u, and an index @xmath1, • • • , |u|}, decide whether the word u i is a hallucination, i.e. whether it should be replaced or removed to make u consistent with r. candidate selection. for each instruction, we identify a set of candidate words for classification, which are (a) directional words like left, right, etc. (see §a.2 for a full list) as well as (b) nouns identified by the spacy part-of-speech tagger @xcite . ## conclusion this work is an early attempt to address the hallucination issue in grounded instruction generation. we have shown that techniques like self-supervised type f1 precision recall pre-training on multimodal data and contrastive fine-tuning on synthetic data are promising scalable approaches. we hope that these directions can be further developed in future work.