On Hallucination and Predictive Uncertainty in Conditional Language Generation (2021)

Abstract: Despite improvements in performances on different natural language generation tasks, deep neural models are prone to hallucinating facts that are incorrect or nonexistent. Different hypotheses are proposed and examined separately for different tasks, but no systematic explanations are available across these tasks. In this study, we draw connections between hallucinations and predictive uncertainty in conditional language generation. We investigate their relationship in both image captioning and data-to-text generation and propose a simple extension to beam search to reduce hallucination. Our analysis shows that higher predictive uncertainty corresponds to a higher chance of hallucination. Epistemic uncertainty is more indicative of hallucination than aleatoric or total uncertainties. It helps to achieve better results of trading performance in standard metric for less hallucination with the proposed beam search variant.

## introduction modern deep neural network models have brought drastic improvements of generation quality measured by standard metrics on different natural language generation (nlg) tasks. however, along with these improvements, researchers find that neural models are more prone to a phenomenon called hallucination, where models generate description tokens that are not supported by the source inputs. this phenomenon seriously damages the applicability of neural language generation models in practice where information accuracy is vital. hallucination has been observed in various conditional nlg tasks such as image captioning @xcite , data-to-text generation @xcite @xcite , abstractive summarization @xcite , and neural machine translation (nmt) @xcite . these studies tackle hallucinations within a specific task and give possible explanations of why hallucinations occur. for example, @xcite attributes object hallucination in image captioning to visual misclassification and over-reliance on language priors; @xcite believes hallucination in neural surface realization comes from the misalignment between meaning representations and their corresponding references in the dataset; @xcite claims that hallucinations in nmt are mainly due to domain shift. we believe that there is a common theme across all the hallucination explanations in conditional nlg tasks: predictive uncertainty. in language generation, predictive uncertainty quantifies the entropy of the token probability distributions a model predicts. there are multiple sources of uncertainty. two major ones frequently studied are aleatoric and epistemic uncertainties, where the former comes from the data or measurements, and the latter is concerned with the model. with recent progress in bayesian neural networks (bnns) @xcite and uncertainty quantification @xcite @xcite , we are able to quantify both parts of predictive uncertainty in neural nlg. this study draws connections between hallucination and predictive uncertainty and empirically investigates their relationship in image captioning and data-to-text generation tasks. we propose an uncertainty-aware beam search algorithm to reduce the chance of hallucination by penalizing parts or the entirety of the predictive uncertainty during model decoding. we find that the choice of uncertainty matters, and penalizing epistemic uncertainty yields better results compared to penalizing aleatoric or total uncertainty. our contributions are: ## case study: image captioning in this section, we analyze image captioning models trained on mscoco @xcite data set. ## case study: data-to-text generation data-to-text generation @xcite is a task to generate textual content conditioned on input content in the form of structured data such as tables. neural models are prone to hallucination in data-to-text generation tasks compared to traditional template-based systems, and methods are proposed to improve faithfulness @xcite @xcite . in this section, we discuss the relationship between predictive uncertainty and hallucination in data-to-text generation with totto dataset (parikh et al., 2020). ## discussion and conclusions we investigate the relationship between hallucination and predictive uncertainty in image captioning and data-to-text generation tasks and show that predictions with higher uncertainty are more prone to hallucination. in particular, epistemic uncertainty is more indicative of hallucination than aleatoric uncertainty. we propose uncertainty-aware beam search to incorporate uncertainty into the decoding process to reduce hallucination. we show that uncertainty decomposition helps the proposed beam search variant to achieve a better performancehallucination trade-off. specifically, penalizing epistemic uncertainty yields better results compared to penalizing aleatoric or total uncertainty. in this work, we analyze uncertainty from the token level. this might be restrictive because uncertainty corresponds to the current prediction context instead of the predicted token. the relationship between hallucination and uncertainty, therefore, can be much more complicated than a linear one. it is still possible to produce hallucinated information with a very confident model. the proposed uabs reduces hallucination by limiting the total uncertainty of the generated text. as a result, it might lead to shorter generations and lower generation quality. devising more sophisticated uncertaintyaware training and decoding methods with less adverse effects on the generation quality is a future direction to explore.