Sources of Hallucination by Large Language Models on Inference Tasks (2023)

Abstract: Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI), necessary for applied tasks like question answering and summarization. We present a series of behavioral studies on several LLM families (LLaMA, GPT-3.5, and PaLM) which probe their behavior using controlled experiments. We establish two biases originating from pretraining which predict much of their behavior, and show that these are major sources of hallucination in generative LLMs. First, memorization at the level of sentences: we show that, regardless of the premise, models falsely label NLI test samples as entailing when the hypothesis is attested in training data, and that entities are used as “indices’ to access the memorized data. Second, statistical patterns of usage learned at the level of corpora: we further show a similar effect when the premise predicate is less frequent than that of the hypothesis in the training data, a bias following from previous studies. We demonstrate that LLMs perform significantly worse on NLI test samples which do not conform to these biases than those which do, and we offer these as valuable controls for future LLM evaluation.

## introduction large language models (llms) such as llama, gpt-3/4, palm, and more @xcite @xcite , have been trusted by many to perform language understanding in downstream tasks such as summarization, question answering, and fact verification, among others @xcite . however, due to the large-scale nature of llm training on vast, often proprietary data, and the inherent opacity of llm parameters, it is difficult to explain their behavior when answering user queries and the corresponding risks in terms of bias and robustness. in particular, one llm behavior poses a significant challenge: "hallucination," the phenomenon in which llms provide information which is incorrect or inappropriate, presented as fact. this paper investigates two biases driving llm performance in natural language inference, sometimes called textual entailment. this is a basic component of language understanding which is critical in applied tasks, and we offer these two biases as explanations of general false positive hallucination in everyday use. we examine broader nli, and especially directional entailments, which hold in one direction, but not both. for example, defeat entails play but play does not entail defeat. inferring directional entailment is more difficult than that of symmetric paraphrase, so it more deeply probes understanding. our approach is a behavioral study of prompted llm decision-making. we alter existing nli datasets in targeted ways while measuring how predictions change, across several major llm families (llama, gpt-3.5, and palm). we demonstrate two sources of llm performance on the nli task, which we offer as explanations of general false positive hallucination: (1) llm bias toward affirming entailment when the hypothesis may be attested in the training text, including reliance on named entity identifiers; and (2) a corpus-frequency bias, affirming entailment when the premise is less frequent than the hypothesis. we establish that these biases originate from the llm pretraining objective, in which statistical modeling of the natural distribution of humangenerated text leads to (at the level of sentences) memorizing individual statements, and (at the level of corpora) learning typical patterns of usage. though they are superficially performant, our experiments show that even powerful llms still use unsatisfactory tools instead of robust reasoning. we present three contributions, the demonstrations of both factors and an analysis of their impact: ## related work addressing task robustness, @xcite found a range of nli datasets contain artifacts which are learned by supervised models trained on only sample hypotheses. in our work we design a similar hypothesis-only test with llms, but we use it to probe model memory without any training. by conditioning on the attestation of hypotheses, we show that llms are inherently sensitive to attestation, separate from the statistical idiosyncrasies of nli datasets. 2 additionally, @xcite report generalization failure among many models supervised for nli -models fail to generalize between nli datasets, even if the task is formatted the same. on smaller language models such as roberta @xcite ; 355m params), @xcite also observed a reliance on dataset artifacts when performing directional nli on predicates. we now study the behavior of much larger lms, which have demonstrated more robust performance across nlp tasks. recent work has also explored llm memorization and generalization. @xcite establish that llms are able to memorize more data than small lms, whereas @xcite further demonstrate that llms pay special attention early in training to numbers and nouns, which act as unique identifiers for individual training sentences. we also show that memories used in language inference are tied to specific named entities. and while @xcite and @xcite find that entity frequency in training data is correlated with performance in factual recall about them, we find that entity frequency is anti-correlated with hypothetical generalization performance ( §6). @xcite argue that gpt-4 understands language "beyond memorization". we do not disprove generalization, but we show that gpt-4 shows the same hallucinations in appendix f. ## experimental design we design behavioral experiments on llms by modifying nli datasets with rigorous controls. we observe large behavior changes across three major llm families due to propositional-memory effects in §5 and §6, and corpus frequency in §7. finally, we show the impact on real performance in §8. ## experiment 1: attestation bias we begin our experiments by assessing llms' reliance on their propositional memory of training text by conditioning each model's entailment task predictions i on its own predictions of attestation λ. we do this by comparing estimated probabilities of predicting entail conditioned on whether the hypothesis is predicted attested or not. further, we test a setting which controls for the possibility that original levy/holt entailments may coincidentally refer to attested facts, which could lead to spurious correlation between inference and attestation scores without clearly demonstrating use of memory versus true entailment. this controlled setting is the random premise task i randp rem , which converts entailments into non-entailments without altering the hypothesis. an ideal model capable of drawing inferences from information in context should detect that in the i randp rem task it is no longer possible to infer the hypothesis based on the premise (even if the hypothesis is itself attested in training), and never predict entail. thus, in i randp rem , all entail predictions are assumed to be false positive hallucinations. ## experiment 3: relative frequency bias we continue the conditioning experiments from §5, now exploring the relative frequency bias. sample labels for this bias are denoted by the modelagnostic φ as described in §3.1. φ labels the conformance of sample predicates to the bias: φ < means p is less corpus-frequent than h by a margin (positive class), φ > means p more frequent than h by the margin (negative class). to control for differences between datasets, the margin is set so that 1/3 of samples are classed as "roughly equal" (φ ≈ ), which we discard. following the observations in §6, we further apply a generic-argument transformation to control for attestation, yielding i genarg randp rem . with the entities masked, models cannot recall propositional memory for this task: by re-calculating the λ measure with generic arguments, only 2 hypotheses are still predicted as attested by gpt-3.5, whereas for llama and palm, the numbers are also only 6.2% and 3.9%. additionally, as with i randp rem , here the entailment label of each sample remains no-entail, so any entail prediction is false positive hallucination. ## impact of bias on performance we have demonstrated two sources of hallucination by llms on inference tasks. we now assess their impact on model performance to quantify their risk. we compare llms' performance between nli subsets that are consistent or adversarial to each bias. a sample p ⊨ h? is consistent with a bias when the prediction by the bias agrees with the gold entailment label; conversely, it is adversarial to a bias when the prediction by the bias disagrees with the label. for example, "google bought youtube ⊨ google owns youtube" is consistent with the attestation bias of every model, because the conclusion google owns youtube is attested in every llm's training data, and the sample label is entail; "apple owns samsung ⊭ apple bought samsung" is also consistent, because its conclusion is not attested and the sample label is no-entail. the reverses of these two samples are adversarial, since their respective attestedness (unchanged) does not agree with the entailment labels (now flipped). for each subset, there is substantial representation in both levy/holt and rte-1 (see appendix while earlier experiments inspected model textual responses to characterize behavior change, we now use area under the precision-recall curve (auc) to summarize model performance over a tunable confidence threshold (scoring described in §4.2), which is better for measuring practical discriminative power. following @xcite , we re-scale auc values to normalize over the label distribution, yielding au c norm values that assign random classifiers 0% and perfect classifiers 100%. we report results in on the other hand, with the generic arguments in i genarg , llms are forced to focus on the predicates in each proposition. as a result, the impact of the relative frequency bias is intensified. from the standard inference task i to i genarg , the average performance drop from the cons. to adv. ## conclusion across several major llm families and experimental settings, we demonstrate two important biases in the performance of llms on natural language inference tasks, which may also manifest in applied tasks as hallucination. contrary to claims of llm general reasoning capabilities, we show that much of this performance is achieved by (1) recall of relevant memorizations and (2) corpus-based biases like term frequency. since these factors are reproduced in all models, we establish that they originate in llm pre-training, and are not corrected during gpt-3.5 fine-tuning. we conclude that llms, though powerful, use unsatisfactory tools for the basic tasks of language understanding and inference. we propose several approaches to control for these biases in evaluation, and ultimately conclude that further attention on alleviating these biases are needed, before llms may be trusted to reason robustly about language.