Pre-trained Language Models Return Distinguishable Probability Distributions to Unfaithfully Hallucinated Texts (2024)

Abstract: In this work, we show the pre-trained language models return distinguishable generation probability and uncertainty distribution to unfaithfully hallucinated texts, regardless of their size and structure. By examining 24 models on 6 data sets, we find out that 88-98% of cases return statistically significantly distinguishable generation probability and uncertainty distributions. Using this general phenomenon, we showcase a hallucination-reducing training algorithm. Our algorithm outperforms other baselines by achieving higher faithfulness metrics while maintaining sound general text quality measures.

## introduction hallucination is one of the key phenomena that undermine the reliability of large language models (llms), which recently gained large popularity in real-world applications @xcite . @xcite characterized hallucinations with two perspectives: faithfulness and factuality. the former represents consistency to the provided source text, while the latter is consistency to the world knowledge. for example, if a user asks a machine to recommend a dinner menu and a machine answers that 'cereal is a breakfast menu enjoyed by many people', then the answer is factual but not faithful to the user's request. before the pre-trained language model (plm) era, researchers found out that generation probability and uncertainty measured by a language model are correlated with the faithfulness of a text @xcite @xcite . though their work utilized un-pre-trained models trained on specific tasks, like image captioning, these works hinted at plms' potential to distinguish unfaithfulness. in this paper, we examine three research hypotheses first. (1) does the unfaithfulness distinguishing ability generalize to the various sizes and types of plms? (2) does the model size affect the ability? (3) how does the fine-tuning affect the ability? we examine these hypotheses with 24 pre-trained language models of various sizes and types on 6 data sets. from massive experiments, 88-98% cases return significantly distinguishable generation probability and uncertainty distributions. using this phenomenon, we showcase a simple training algorithm that effectively reduces hallucination. ## suggested metrics notations let f be a plm. f can be an encoder model pre-trained on a masked language modeling task like bert, a decoder model pre-trained on an autoregressive language modeling task like gpt2, or an encoder-decoder model like t5. for an encoder model, f (x i , y i,!j )[v] ∈ [0, 1] is a probability of a token v at masked position j given reference text and masked target text. so f (x i , y i,!j )[y i,j ] is the probability for the right token. for decoder and encoder-decoder models, f (x i , y i,<j )[v] ∈ [0, 1] is a probability of a token v at truncated position j given reference text and truncated target text. hence f (x i , y i,<j )[y i,j ] is the probability for the right token. ## distribution distinguishability let p be a distribution of a metric based on a plm f , and d be a statistic computing distinguishability between two distributions. in this section, our goal is to verify (1) whether d(p(d hallucinated )||p(d entailed )) is statistically significant, and how the (2) model size and (3) finetuning of plms affect the distinguishability. ## hallucination reduction with weighted training in this section, we showcase a weighted training method to mitigate hallucination. the idea is sim- we compare four baseline training methods on three data sets: the usual unweighted training, a control-token method (ctrl, rashkin et al. ( 2021 )), and other loss weighting methods (truncation @xcite ) truncate high loss points and mfact @xcite weight the loss with faithfulness score). we compare four faithfulness metrics and three general text quality metrics. a more detailed explanation is in appendix e. the results are on for knowledge-grounded dialogue data sets, our algorithms, with both entropy and logprob, improve faithfulness compared to unweighted by a large margin, in all cases. the same phenomenon occurs when compared with other baselines. it is interesting since ctrl is designed to mitigate hallucination, especially in the knowledge-grounded dialogue task. for mediqa, ours-logprob outperforms unweighted on most metrics. ours-logprob outperforms truncation and mfact on q 2 and summac, though both are designed to reduce hallucination in the summarization task. the result shows not only powerful hallucination reduction performance compared to other task-specific methods but also the general applicability of our methods through various tasks. our algorithm maintains general text quality measures, which we do not directly target. more interestingly, our method often outperforms other baselines. it may indicate its potential for enhancing not only faithfulness but also the overall fidelity and quality of generated text across diverse evaluation metrics. ## conclusion our work is the first comprehensive analysis of the plms' unfaithful hallucination-distinguishing ability. we compare plms' generation probability and uncertainty distributions of unfaithful and entailed texts. regardless of the model type and size, plms return statistically distinguishable distributions to unfaithfully hallucinated and entailed texts for 88-98% cases. unlike usual practice, the smaller models show comparable (and sometimes better) distinguishability to the largest models, while the distinguishability of entropy declines while logprob increases after fine-tuning. utilizing this phenomenon, we showcase a hallucinationreducing training algorithm that outperforms other baselines with hallucination reduction while maintaining sound general text quality measures. we hope these findings lead to a deeper understanding of the hallucination phenomenon and more reliable hallucination mitigating techniques.