Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization (2022)

Abstract: State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text. Despite being assumed to be incorrect, we find that much hallucinated content is actually consistent with world knowledge, which we call factual hallucinations. Including these factual hallucinations in a summary can be beneficial because they provide useful background information. In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. Our method is based on an entity’s prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our method vastly outperforms two baselines in both accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks. Furthermore, we use our method as a reward signal to train a summarization system using an off-line reinforcement learning (RL) algorithm that can significantly improve the factuality of generated summaries while maintaining the level of abstractiveness.

## introduction state-of-the-art abstractive summarization systems can generate fluent summaries with high automatic evaluation scores in terms of rouge @xcite . however, recent studies have shown that these systems are prone to hallucinate content that is not supported by the source document @xcite @xcite @xcite . for instance, @xcite discovered that 64.1% of the summaries generated by a bert-based abstractive summarization model on xsum @xcite contain hallucinations. previous studies commonly assume that hallucination is an undesirable behavior in abstractive summarization systems. they investigate the source: under the proposals, 120,000 additional asylum seekers will be distributed among eu nations, with binding quotas. (...) mr juncker told the european parliament it was "not a time to take fright". (...) he said tackling the crisis was "a matter of humanity and human dignity". "it is true that europe cannot house all the misery in the world. but we have to put it into perspective." (...) generation: european commission president jean-claude juncker has set out his proposals for dealing with the migrant crisis in a speech to meps, saying europe cannot house all the misery in the world. our stance in this paper is that hallucinations are not always undesirable: many factual hallucinations provide additional world knowledge that is important for summary comprehension. we thus argue that not all hallucinations should be treated equally; in particular, factual hallucinations may be less deleterious or even potentially beneficial to to be included in a summary, as opposed to non-factual ones. we propose a method to classify entities according to whether they are hallucinations and whether they are factual (if hallucinated). we focus on entities (e.g., persons, locations, dates, cardinal numbers) because they are necessary to express the most salient pieces of in-formation in a summary. moreover, entity hallucinations are common in generated summaries. as we will show later in our work, about 30% of entities generated by bart @xcite on xsum test set are hallucinated. our approach is inspired by the observation that many hallucinated entities are generated with low probabilities. this observation suggests that the summarization model's confidence correlates with the factuality statuses of generated entities. in other words, the uncertainty is indicative of the likelihood of whether generated entities are hallucinated and non-factual. we refer to the probability of an entity being in a summary without considering the source document as its prior probability, and its probability given the document as its posterior probability. our assumption is that if an entity in a generated summary results in a factual error, giving the source should not provide more evidence for it, resulting in a small change in probability between the prior and the posterior. based on this assumption, we propose to use the prior and posterior probabilities as the key features in a simple classifier that predicts an entity's hallucination status and factuality. due to the lack of fine-grained hallucination annotation, we create an entity-level hallucination and factuality annotation on the xsum dataset. we evaluate our detection method on this annotated dataset as well as annotations from @xcite . on both datasets, our approach outperforms five baseline models at identifying nonfactual hallucinations. in addition, our approach has a strong correlation with the factuality scores given by human judges. besides, we show that our detector, when used as a reward signal in training neural-based summarizers with the off-line rl algorithm, significantly improves the factuality of generated summaries even when the underlying dataset is noisy. our contributions are the following: (i) we demonstrate that an entity's prior and posterior probabilities can be used to infer whether it is hallucinated and factual. based on this hypothesis, we propose a novel approach for entity-level hallucination detection and factuality checking. our approach outperforms five baselines from previous work on two human-annotated datasets, in addition to having a strong correlation with summarylevel factuality scores given by human judges. (ii) we empirically demonstrate that our classifier can provide reliable reward signals for rl algorithms, leading to improved factuality while maintaining the level of abstractiveness in generated summaries. (iii) we create a set of entity-level hallucination annotations. ## related work the correctness of summarization systems' outputs has been evaluated as one aspect of content selection in the past, for example using the pyramid method @xcite . as neural abstractive summarizers have become popular, their issues with correctness have sparked much recent work that focus specifically on model hallucinations and summary factuality @xcite . @xcite conducted a large-scale human evaluation of several neural abstractive summarization systems, and found that hallucinations are common among the outputs of different summarization models. ## method in this section, we propose a novel detection approach that separates factual from non-factual hallucinations of entities (section 3.2), and present a factuality-aware training framework for summarization models trained on noisy dataset (section 3.3). ## analysis 7.1 ablation studies to explore the effect of each feature, we conduct an ablation study by training the knn classifier with fewer features. the results are illustrated in ## conclusion in this paper, we investigate the hallucination and factuality problems in abstractive summarization. we show that about 30% of entities generated by state-of-the-art summarization model are hallucinated. more interestingly, more than half of the hallucinated entities are factual with respect to the source document and world knowledge. we propose a novel method based on the entity's prior and posterior probabilities according to masked language models. our approach outperforms five baseline models on both factuality classification and hallucination detection tasks on human-annotated datasets. in addition, using our classifier as a reward signal vastly improves the factuality of summarization systems. our approach is limited to entity-level hallucination and factuality classification. in the future, we are interested in extending our work to arbitrary text spans. a.6 where does the model learn to hallucinate? figure 6 shows the entity distribution from the two cmlm models. for non-hallucinated entities, the distributions are similar; for factual hallucinations, we can find that a large portion of them has very low posterior probabilities under cmlm cnn/dm , but high posterior under cmlm xsum . this pattern suggests that the knowledge of many factual hallucinations comes from the xsum training set.