Why LLM s Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generation (2023)

Abstract: We show that LLMs hallucinate because their output is not constrained to be synonymous with claims for which they have evidence: a condition that we call evidential closure. Information about the truth or falsity of sentences is not statistically identified in the standard neural language generation setup, and so cannot be conditioned on to generate new strings. We then show how to constrain LLMs to produce output that satisfies evidential closure. A multimodal LLM must learn about the external world (perceptual learning); it must learn a mapping from strings to states of the world (extensional learning); and, to achieve fluency when generalizing beyond a body of evidence, it must learn mappings from strings to their synonyms (intensional learning). The output of a unimodal LLM must be synonymous with strings in a validated evidence set. Finally, we present a heuristic procedure, Learn-Babble-Prune, that yields faithful output from an LLM by rejecting output that is not synonymous with claims for which the LLM has evidence.

## introduction there is a growing body of evidence that llms systematically hallucinate @xcite @xcite @xcite . hallucinations may limit the utility of llms, in addition to having significant implications for safety @xcite @xcite . it has been suggested that hallucinations occur because language models do not interpret training inputs semantically @xcite @xcite . we offer a new formalization of this notion that allows us to explain why llms are inherently prone to hallucination, and what any faithful llm must do: its output must be closed under synonymy with its evidence about the world, a condition we call evidential closure. an llm is factual if it is faithful, and, in addition, its evidence about the world is correct. 1 many of the conceptual issues now studied in natural language processing have received extensive treatment in the analytic philosophy of language @xcite @xcite @xcite . conveniently, these treatments are often mathematically tractable. one set of fundamental distinctions is between intension or meaning; extension or reference; and facts or states of the world, respectively. 2 words and sentences have meanings, which are equivalence classes of other words or sentences with which they are synonymous. they also have referents: states of the world that they map onto. finally, there is an external reality that the agent has access to, equipped with a valuation function that assigns states of the world to true or false. sentences are true when they correctly refer to states of the world that are true. a popular theory of meaning in the philosophy of language that links these three notions is the extensional semantics of @xcite . this theory holds that the meaning of a sentence is just the set of states of the world in which that sentence is true. using this account, we can characterize a faithful speaker of a language as one who 1) uses their knowledge about the world 2) to describe states of the world 3) using a variety of equivalent sentences. this entails that a faithful speaker must perform three tasks: they must learn about the world (perceptual learning); they must learn which sentences map onto which states of the world (extensional learning); and they must learn which sentences have the same meaning (intensional learning). a factual speaker performs the same tasks, with the difference that their evidence about the world is correct. here, faithfulness to model inputs is conceptually prior to factuality, since, definitionally, the information the model has about the world is contained in its inputs. we use this setup to state an impossibility result: neural probabilistic language models in the vein of @xcite are not factual (theorem 4.5). llms maximize the conditional probability of the generated strings given the corpus and the prompt. they do not explicitly learn states of the world, do not explicitly learn the meanings of words, and do not explicitly learn the referents of sentences. each of these types of information is unobserved. as a result, the conditional distributions learned by llms can be statistically independent of, or invariant to, their semantic content: that is, of the referents and the truth-conditions of the sentences in the corpus and prompts. so we may have variation in the truth or falsity of a state of the world without variation in the solution to a predictive model that generates the next sentence given a prompt. because this semantic information is not contained in the conditional distribution from which an output string is generated, simulating from the learned distribution does not preserve this information, even when it is contained in the corpus (theorem 4.6). hence there is no guarantee that llms are faithful to the semantic content of their inputs, either. we can think of this as the cause of hallucination in llms. second, we show conceptually how to build a faithful or factual llm. the output of such an llm must satisfy evidential closure: that is, its output must be synonymous with claims for which the llm has evidence. this ensures that every claim made by the model is either directly corroborated by evidence, or is a paraphrase of a directly corroborated claim. we first define the objective functions for faithful and factual llms using theory from the philosophy of language. we then decompose those objective functions into learnable distributions (model 5.9). we show that the output of these models is faithful or factual, because it solves the constrained learning task that incorporates semantic information about the truth or fal-sity of sentences for which the model has evidence (theorem 5.10 and theorem 5.11). third, we provide heuristic framework for building factual or faithful llms, which we call learn-babble-prune. in this setup, the output of an llm is cross-checked against its evidence, and discarded if it is not a paraphrase of a claim that it has evidence for. this ensures that the output of an llm is consistent with its evidence. this ensures that the llm is faithful to its evidence, and hence, does not hallucinate. we consider two applications: when the evidence is a corpus of strings, and when the evidence is in the form of sensor information. if the evidence is in the form of text, as in some retrieval-augmented language models @xcite , then an llm's output is consistent with its retrieved body of evidence if its output is a paraphrase of a string in its evidence base. in this application, the model must learn paraphrases of sentences in order to cross-check the output of an llm with its evidence base. 3 if the evidence is in the form of information about the world gathered by sensors, as in some multimodal llms @xcite @xcite , then, in addition to intensional learning, the llm must learn a mapping from perceptual information to strings. thus a multimodal llm is faithful if its output paraphrases information acquired by its perceptual learner, and factual if that perceptual learner is unbiased and consistent. a final point is that, like any language speaker, an llm can only make as many true claims as are semantically entailed by the evidence it possesses @xcite . collecting and interpreting large bodies of evidence is a vitally important task for the next generation of llms. ## building factual/faithful llms: how to get (evidential) closure how do we go beyond the negative result above? we require that the output of an llm is equal to the closure under synonymy of strings that refer to verified information. that is, every string output by the llm either refers to a claim that is verified, or is a synonym of a claim that is verified. by proposition 3.5, all such sentences will be true. any llm that satisfies this property is then factual. if we require only that the output is closed under synonymy with strings that are synonymous with the agent's evidence, irrespective of its credibility, the llm is faithful. ## conclusions llms hallucinate because their output is not constrained to be semantically consistent with their inputs, so there is no guarantee that any evidence about the world contained in their inputs is preserved in their outputs. to build a faithful or factual llm it is necessary to constrain the output of a model to be consistent with claims for which the model has explicit evidence. in practice, this means acquiring large bodies of machine-readable string evidence or using sensors (perceptual learning) combined with sensor-to-text encoders (extensional learning) to validate the output of an llm with evidence. paraphrase learning methods can be used to expand the llm's vocabulary (intensional learning). we propose a simple method to implement this in practice via rejection sampling. any input-faithful llm is limited in what it can say by what it has evidence for. generating large-scale evidence bases is likely to be a much bigger binding constraint that the parameter size of the model, for instance, and may require a rethink of how computational and financial resources are allocated to the design of llms. this is a challenge for the next generation of llms. algorithm 2 lbp: multimodal-to-text enforcing consistency with evidence via rejection sampling is a relatively inefficient way to constrain output to be factual or faithful. we expect that rl approaches could implement a procedure analogous to learn-babble-prune more efficiently, with tolerable loss of accuracy. the purpose of this framework is to highlight, at a high-level, the kind of tasks that any such rl approach would have to do. as such, it is primarily intended as a conceptual contribution. further, strict factuality may be an undesirably high bar in some applications: clearly it depends on the use case.