Mutual Information Alleviates Hallucinations in Abstractive Summarization (2022)

Abstract: Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i.e., output content not supported by the source document. A number of works have tried to fix—or at least uncover the source of—the problem with limited success. In this paper, we identify a simple criterion under which models are significantly more likely to assign more probability to hallucinated content during generation: high model uncertainty. This finding offers a potential explanation for hallucinations: models default to favoring text with high marginal probability, i.e., high-frequency occurrences in the training set, when uncertain about a continuation. It also motivates possible routes for real-time intervention during decoding to prevent such hallucinations. We propose a decoding strategy that switches to optimizing for pointwise mutual information of the source and target token—rather than purely the probability of the target token—when the model exhibits uncertainty. Experiments on the dataset show that our method decreases the probability of hallucinated tokens while maintaining the Rouge and BERT-S scores of top-performing decoding strategies.

## introduction abstractive summarization, the task of condensing long documents into short summaries, has a number of applications, such as providing overviews of news articles or highlighting main points in technical documents. abstractive summarization is usually performed using probabilistic text generators @xcite @xcite , which have shown a strong ability to produce fluent, human-like text @xcite @xcite . however, these models have been observed to hallucinate facts, i.e., add information to the output that was not present in the original text. this behavior is problematic, as presenting users with unsubstantiated content can lead to undesirable effects, such as the spread of misinformation @xcite @xcite . some works have attributed this phenomenon to the specific training corpora for these models, in which ground-truth summaries often contain outside information that may not have been directly deducible from the original text @xcite . others have pointed to model architectures or training strategies @xcite @xcite . while these works have given us an improved understanding of the cause of hallucinations, there still does not exist an efficient and robust set of techniques for identifying and preventing them during the generation process. this work aims to first provide a simple criterion indicating when a model is more likely to assign higher probability to content not necessarily derived from the source document. specifically, we link the start of a hallucination during generation to high model uncertainty about the next token, which we quantify by conditional entropy. we hypothesize that hallucinations may be due to a tendency of models to default to placing probability mass on tokens that appeared frequently in the training corpus, a behavior by language models previously observed in several natural language processing (nlp) tasks @xcite . as a consequence, generations with hallucinations would still be viable candidates, as standard decoding strategies for summarization optimize purely for the probability of the generation. we propose an alternative decoding strategy to combat this behavior: when a model exhibits high uncertainty, we change our decoding objective to pointwise mutual information between the source document and target token (pmi; @xcite , encouraging the model to prioritize tokens relevant to the source document. while changing completely to the pmi objective causes a drop of 3.13% in rouge-l scores, this conditional and temporary change leads to only a 0.977% drop in rouge-l while increasing factuality according to the factscore metric. in experiments, we first observe a strong correlation between conditional entropy and the start of a hallucination on an annotated subset of the xsum dataset @xcite . we next score the targets in the annotated subset under both the standard log-probability objective and cpmi, and observe that the revised log-probability of hallucinated tokens under the cpmi objective is indeed lower . finally, we find that our proposed decoding strategy maintains rouge and berts scores. ## preliminaries in this work, we consider probabilistic models for abstractive summarization. explicitly, we consider models with distribution p(y | x), where x is the source document that we wish to summarize and @xmath0, . . . , y t ⟩ is a string, represented as a sequence of tokens from the model's vocabulary v. the set of valid sequences y is then defined as all sequences y such that y 0 @xmath1, the beginning-and end-of-sequence tokens, respectively, and y @xmath2. note that standard models are locally normalized, i.e., they provide a probability distribution over v @xmath3, x). the probability of an entire string y can then be computed as p(y | x) = t @xmath4, x), where for shorthand we define @xmath5, . . . , y t-1 ⟩. generation from p is performed token-by-token due to the autoregressive natures of most language generators. we typically seek to generate a string that maximizes some score function in the case of probabilistic models, this function is often simply score(y | x) = log p(y | x), i.e., we want to generate a high probability string y. note that searching over the entire space y is usually infeasible (or at least impractical) due to the non-markovian nature of most neural models. thus we often use an approximate search algorithm such as beam search, as given in alg 1, that optimizes for our score function somewhat greedily. this procedure meshes well with the use of log p as algorithm 1 standard beam search. foot_0 • used to represent string concatenation. input: x: source document k: maximum beam size n max : maximum hypothesis length score(• | •): scoring function 1: b 0 ← {⟨0, bos⟩} → beam set 2: for @xmath6, . . . , n max } do 3: b ← ∅ 4: for ⟨s, y⟩ ∈ b t-1 do 5: for @xmath7: s ← score(y • y | x) 7: b.add(⟨s, y • y⟩) 8: b t ← b.top(k) 9: return b nmax .max() the score function since it can be decomposed as the sum of individual token log-probabilities, i.e., we can instead consider a token-wise score function score(y | @xmath8, x) = log p(y | @xmath9, x) using the fact that score(y | x) = t @xmath10, x ). we only consider decoding strategies for score functions that can be decomposed in this manner. evaluation. abstractive summarization systems are usually evaluated using automatic metrics, such as rouge @xcite . while rouge generally correlates poorly with human judgments @xcite and is only weakly correlated with factuality, foot_1 it is quick to compute, making it useful for quickly testing modeling choices. recently, entailment metrics (factcc; @xcite and contextual embedding methods (bertscore; @xcite have surfaced as reasonable indicators of factuality. ## finding and combating hallucinations it is not well understood when summarization models start to hallucinate, i.e., when they start to place high probability on continuations that are unfaithful (not entailed by the information presented in the source document). in this work, we hypothesize that such moments correlate with high model uncertainty. in other problem settings, it has been observed that nlp models default to placing an inappropriately large portion of probability mass on high-frequency (with respect to the training cor-pus) tokens; this is especially the case when making predictions for data points of a type that the model has not had much exposure to during training @xcite . in this same setting, models often have high (epistemic) uncertainty about their predictions @xcite . we extrapolate on these findings and posit that summarization models may score highly more marginally likely-but perhaps unrelated-tokens in settings for which they are not well-calibrated. fortunately, both model certainty and marginal likelihood have quantifications that can be easily computed at any given point in the decoding process, making it possible to test for relationships between these quantities and the start of hallucinations. specifically, we can use the standard equation for shannon entropy with our conditional distribution to quantify model uncertainty at time step t: entropy is not a holistic measure of uncertainty, 3 but our use of is motivated by previous research that has likewise employed it to quantify the uncertainty of model predictions in classification @xcite and summarization @xcite tasks. further, we can directly compute marginal probabilities p(y | @xmath11. 4 ## related work understanding hallucinations. several prior works have tried to identify the cause of hallucinations in various natural language generation tasks, along with methods for alleviating them. for example, both wang and sennrich (2020) and @xcite suggest that exposure bias, i.e., the failure of a model to predict accurate continuations following its own generations rather than the ground-truth context as a result of the discrepancy between procedures at training and inference, leads to hallucinations, as it causes the model to over-rely on target contributions when decoding. they propose using minimum risk training (mrt), which can alleviate exposure bias, to make models more robust. however these results show only a tentative connection to exposure bias, and are based on models for neural machine translation (nmt) rather than summarization. other works have shown that pre-training and training on more data generate summaries more faithful to the source @xcite . in contrast to these works, our method does not require any changes to model training. rather, it intervenes during generation without the need to retrain the base model. ## experiments data. we use the extreme summarization (xsum) dataset @xcite , which is composed of 226,711 british broadcasting corporation (bbc) articles and their single-sentence summaries. we use the same train-valid-test splits as the authors. a subset of these articles (500) from the test set are annotated, i.e., reference spans are labeled as faithful or not to the source article @xcite . we further process these labels to obtain token-level hallucination labels. ## conclusion in this work, we link the start of a hallucination in abstractive summarization during generation to model uncertainty, as quantified by high conditional entropy, about next-token predictions. we then propose a decoding procedure, cpmi, which switches the decoding objective to pointwise mutual information when model uncertainty is high to prevent hallucinations. our method reduces the likelihood of generating unfaithful tokens while still outputting high-quality summaries. in the future it would be interesting to combine cpmi decoding with post-hoc correction methods and other decoding procedures, to investigate if we can complement existing techniques mentioned in §4.