Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection (2023)

Abstract: Neural sequence generation models are known to “hallucinate”, by producing outputs that are unrelated to the source text. These hallucinations are potentially harmful, yet it remains unclear in what conditions they arise and how to mitigate their impact. In this work, we first identify internal model symptoms of hallucinations by analyzing the relative token contributions to the generation in contrastive hallucinated vs. non-hallucinated outputs generated via source perturbations. We then show that these symptoms are reliable indicators of natural hallucinations, by using them to design a lightweight hallucination detector which outperforms both model-free baselines and strong classifiers based on quality estimation or large pre-trained models on manually annotated English-Chinese and German-English translation test beds.

## introduction while neural language generation models can generate high quality text in many settings, they also fail in counter-intuitive ways, for instance by ''hallucinating'' @xcite @xcite . in the most severe case, known as ''detached hallucinations'' @xcite , the output is completely detached from the source, which not only reveals fundamental limitations of current models, but also risks misleading users and undermining trust @xcite . yet, we lack a systematic understanding of the conditions where hallucinations arise, as hallucinations occur infrequently among translations of naturally occurring text. as a workaround, prior work has largely focused on black-box detection methods which train neural classifiers on synthetic data constructed by heuristics @xcite , and on studying hallucinations given artificially perturbed inputs @xcite . in this paper, we address the problem by first identifying the internal model symptoms that characterize hallucinations given artificial inputs and then testing the discovered symptoms on translations of natural texts. specifically, we study hallucinations in neural machine translation (nmt) using two types of interpretability techniques: saliency analysis and perturbations. we use saliency analysis @xcite to compare the relative contributions of various tokens to the hallucinated vs. nonhallucinated outputs generated by diverse adversarial perturbations in the inputs ( we further show that the symptoms identified generalize to hallucinations on natural inputs by using them to design a lightweight hallucination classifier ( § 4) that we evaluate on manually annotated hallucinations from english-chinese and german-english nmt ( ## hallucinations: definition and hypotheses the term ''hallucinations'' has varying definitions in mt and natural language generation. we adopt the most widely used one, which refers to output text that is unfaithful to the input @xcite @xcite , while others include fluency criteria as part of the definition @xcite . different from previous work that aims to detect partial hallucinations at the token level @xcite , we focus on detached hallucinations where a major part of the output is unfaithful to the input, as these represent severe errors, as illustrated in prior work on understanding the conditions that lead to hallucinations has focused on training conditions and data noise @xcite . for mt, @xcite show that hallucinations under perturbed inputs are caused by training samples in the long tail that tend to be memorized by transformer models, while natural hallucinations given unperturbed inputs can be linked to corpus-level noise. @xcite show that models trained on samples where the source and target side diverge semantically output degenerated text more frequently. wang and sennrich (2020) establish a link between mt hallucinations under domain shift and exposure bias by showing that minimum risk training, a training objective which addresses exposure bias, can reduce the frequency of hallucinations. however, these insights do not yet provide practical strategies for handling mt hallucinations. a complementary approach to diagnosing hallucinations is to identify their symptoms via model introspection at inference time. however, there lacks a systematic study of hallucinations from the model's internal perspective. previous work is either limited to an interpretation method that is tied to an outdated model architecture @xcite or to pseudo-hallucinations @xcite . in this paper, we propose to shed light on the decoding behavior of hallucinations on both artificially perturbed and natural inputs through model introspection based on layerwise relevance propagation (lrp) @xcite , which is applicable to a wide range of neural model architectures. we focus on mt tasks with the widely used transformer model @xcite , and examine existing and new hypotheses for how hallucinations are produced. these hypotheses share the intuition that anomalous patterns of contributions from source tokens are indicative of hallucinations, but operationalize it differently. the low source contribution hypothesis introduced by @xcite states that hallucinations occur when nmt overly relies on the target context over the source. they test the hypothesis by inspecting the relative source and target contributions to nmt predictions on transformer models using lrp. however, their study is limited to pseudo-hallucinations produced by force decoding with random target prefixes. this work will test this hypothesis on actual hallucinations generated by nmt models. the local source contribution hypothesis introduced by @xcite states that hallucinations occur when nmt model overly relies on a small subset of source tokens across all generation steps. they test it by visualizing the dotproduct attention in rnn models, but it is unclear whether these findings generalize to other model architectures. in addition, they only study hallucinations caused by random token insertion. this work will test this hypothesis on hallucinations under various types of source perturbations as well as on natural inputs, and will rely on lrp to quantify token contributions more precisely than with attention. inspired by the previous observation on attention matrices that an nmt model attends repeatedly to the same source tokens throughout inference when it hallucinates @xcite or generates a low-quality translation @xcite , we formalize this observation as the static source contribution hypothesis-the distribution of source contributions remains static along inference steps when an nmt model hallucinates. while prior work @xcite @xcite focuses on the static attention to the eos or full-stop tokens, this hypothesis is agnostic about which source tokens contribute. unlike the low source contribution hypothesis, this hypothesis exclusively relies on the source and does not make any assumption about relative source versus target contributions. unlike the local source contribution hypothesis, this hypothesis is agnostic to the proportion of source tokens contributing to a translation. in this work, we evaluate in a controlled fashion how well each hypothesis explains detached hallucinations, first on artificially perturbed samples that let us contrast hallucinated vs. nonhallucinated outputs in controlled settings ( § 3), and second on natural source inputs that let us test the generalizability of these hypotheses when they are used to automatically detect hallucinations in more realistic settings ( § 5). 1 ## study of hallucinations under perturbations via model introspection hallucinations are typically rare and difficult to identify in natural datasets. to test the aforementioned hypotheses at scale, we first exploit the fact that source perturbations exacerbate nmt hallucinations @xcite . we construct a perturbation-based counterfactual hallucination dataset on @xmath0.1). this dataset lets us directly test the three hypotheses by computing the relative token contributions to the model's predictions using lrp ( § 3.2), and conduct a controlled comparison of patterns on the original and hallucinated samples ( § 3.4). ## a classifier to detect natural hallucinations based on these findings, we design features for a lightweight hallucination detector trained on samples automatically constructed by perturbations. classifier we build a small multi-layer perceptron (mlp) with a single hidden layer and the following input features: this yields small classifiers with input dimension of 9. for each language pair, we train 20 classifiers with different random seeds and select the model with the highest validation f1 score. ## detecting natural hallucinations while the hallucination classifier is trained on hallucinations from perturbations, we collect more realistic data to evaluate it against a wide range of relevant models. ## related work hallucinations occur in all applications of neural models to language generation, including abstractive summarization @xcite , dialogue generation @xcite , data-to-text generation @xcite , and machine translation @xcite . most existing detection approaches view the generation model as a black-box, by 1) training hallucination classifiers on synthetic data constructed by heuristics @xcite , or 2) using external models to measure the faithfulness of the outputs, such as question answering or natural language inference models @xcite . these approaches ignore the signals from the generation model itself and could be highly biased by the heuristics used for synthetic data construction, or the biases in the external semantic models trained for other purposes. concurrent to this work, guerreiro et al. (2022) explore glass-box detection methods based on model confidence scores or attention patterns (e.g., the proportion of attention paid to the eos token and the proportion of source tokens with attention weights higher than a threshold). they evaluate these methods based on hallucination recall, and find that model confidence is a better indicator of hallucinations than attention patterns. in this paper, we investigated varying types of glass-box patterns based on the relative token contributions instead of attention, and find that these patterns yield more accurate hallucination detectors than model confidence. detecting hallucinations in mt has not yet been directly addressed by the mt quality estimation literature. most quality estimation work has focused on predicting a direct assessment of translation quality, which does not distinguish adequacy and fluency errors @xcite . more recent task formulations target critical adequacy errors @xcite , but do not separate hallucinations from other error types, despite arguments that hallucinations should be considered separately from other mt errors @xcite . the critical error detection task at wmt 2022 introduces an additions error category, which refers to hallucinations where the translation content is only partially supported by the source @xcite . additions includes both detached hallucinations (as in this work) and partial hallucinations. methods for addressing all these tasks fall in two categories: 1) black-box methods based on the source and output alone @xcite @xcite , and 2) glass-box methods based on features extracted from the nmt model itself @xcite @xcite . black-box methods typically use resource-heavy deep neural networks trained on large amounts of annotated data. our work is inspired by the glass-box methods that rely on model probabilities, uncertainty quantification, and the entropy of the attention distribution, but shows that relative token contributions computed through lrp provide sharper features to characterize hallucinations. this paper combines interpretability techniques to identify the symptoms of hallucinations. we adopt a saliency method to measure the importance of each input unit through a back-propagation pass @xcite @xcite . while other saliency-based methods measure an abstract quantity reflecting the importance of each input feature by the partial derivative of the prediction with regard to each input unit @xcite , lrp @xcite measures the proportional contribution of each input unit. this makes it wellsuited to compare model behavior across samples. furthermore, lrp does not require neural activations to be differentiable and smooth, and can be applied to a wide range of architectures, including rnn @xcite and transformer @xcite . we apply this technique to analyze counterfactual hallucination samples inspired by perturbation methods @xcite @xcite , but crucially show that the insights generalize to natural hallucinations. ## conclusion we contribute a thorough empirical study of the notorious but poorly understood hallucination phenomenon in nmt, which shows that internal model symptoms exhibited during inference are strong indicators of hallucinations. using counterfactual hallucinations triggered by perturbations, we show that distinctive source contribution patterns alone indicate hallucinations better than the relative contributions of the source and target. we further show that our findings can be used for detecting natural hallucinations much more accurately than model-free baselines and quality estimation models. our detector also outperforms black-box classifiers based on pre-trained models. we release human-annotated test beds of natural english-chinese and german-english hallucinations to enable further research. this work opens a path toward detecting hallucinations in the wild and improving models to minimize hallucinations in mt and other generation tasks.