Strong hallucinations from negation and how to fix them (2024)

Abstract: Despite great performance on many tasks, language models (LMs) still struggle with reasoning, sometimes providing responses that cannot possibly be true because they stem from logical incoherence. We call such responses strong hallucinations and prove that they follow from an LM’s computation of its internal representations for logical operators and outputs from those representations. Focusing on negation, we provide a novel solution in which negation is treated not as another element of a latent representation, but as an operation over an LM’s latent representations that constrains how they may evolve. We show that our approach improves model performance in cloze prompting and natural language inference tasks with negation without requiring training on sparse negative data.

## introduction this paper investigates and addresses reasoning problems for language models (lms) that stem from their representation of the meaning of logical terms. underlying such problems, which we call strong hallucinations, are faulty representations that lead lms into logical incoherence, and hence into error, regardless of the facts. concentrating on negation, we prove that how an lm computes meaning representations for negation and how that affects outputs inevitably lead to logical error. we argue that we can eliminate these errors and strong hallucinations by altering the way an lm treats and hence interprets logical operators. we illustrate our approach with negation. our idea is simple but radical: whereas for lms all tokens in a context play the same role in determining an attention based probability distribution over next tokens or "continuations" of the context, we claim logical operators should operate differently from non logical expressions. for instance, negation does not contribute just another token to a latent representation via linear algebraic operations; it provides constraints that shift an input probability distribution π that is determined by the tokens ϕ in its scope to a distribution π ′ , with π ′ (¬ϕ) = 1 -π(ϕ). these constraints, along with others for sequences or conjunction, allow us to calculate algorithmically and recursively transitions from input distributions over only positive content to appropriate distributions for a continuation containing logical operators. the output distribution not only faithfully reflects the logical meaning of negation but also other first order logical operators. while hallucinations are often observed for text generation, these settings are difficult to evaluate. thus, we evaluate our algorithmic or hybrid approach, we call it λ for negation on three easily evaluable tasks: yes|no question answering, masked knowledge retrieval (mkr) and natural language inference (nli) with negation, using both encoder lms and llms. nevertheless, we will illustrate λ on text generation in nli. λ yields good performance on mkr and nli; we increase accuracy by 13% on one nli dataset and by 91% on another. our datasets feature negation and are modified, linguistically validated versions of @xcite . λ requires an lm to train and infer only on positive data, which avoids the problem of finding sparse negative data. section 2 provides background on hallucinations, distributions and linguistic meaning. section 3 proves strong hallucinations follow from basic assumptions about an lm's treatment of negation. section 4 details our positive proposal, and section 5 provides experiments with our approach on q&a, mkr and nli. section 6 discusses related work. ## background lms. we assume lms trained on transformer architectures @xcite with multiple layers i over very large corpora using masked or causal language modeling. in decoder models, the decoder is trained to predict next token y i given a context matrix c i and previously predicted tokens {y 1 , , y i-1 } using a conditional distribution µ(y i /y 1 , ..., y i-1 , c i ). in encoder models, c i is typically the input to an mlp layer to make predictions. c i is the lm's internal representation at i of all the input tokens in its context and is a concatenation of multiple hidden states z h j , one for each attention head such that: generative lms and encoders use this distribution over strings to select ouputs or continuations given a context c. various methods m (maxarg, greedy, beam search, or sampling) over the distribution µ(.|c) have been proposed to find the maximally plausible continuation ϕ ∈ x n of c. an lm's affirmations, which we use in defining hallucinations below, are directly linked to this distribution. definition 1. f affirms ϕ given c, if ϕ = m{@xmath0: µ(x|c)} strong hallucinations. in this paper, we examine a special kind of hallucination we call strong hallucinations. to clarify strong hallucinations, we first provide a gloss for hallucinations. hallucinations involve an attitude like belief or affirmation towards a proposition or sentential content p. an agent a's degree of belief in p, µ(p), which is dependent on a's evidence e, should determine their sincere assertion of p. this holds for lms too; they use probability distributions to determine their "affirmations" (output). now a's degree of belief in p may or may not correlate with a "reasonable" degree of belief in p, π(p)-a degree of belief that one expects a rational person to have. on one type of hallucination, then, an agent a with grounds or context e hallucinates that p if µ a (p|e) diverges significantly from π(p|e). a strong hallucination by a involves a reasoning mistake given evidence. for an lm this evidence is a context or prompt. the result of a strong hallucination is that µ leads a to affirm a claim or set of claims that strongly diverge from any possible reasonable degree of belief given the evidence. a special case is when the set of claims together with the evidence is logically unsatisfiable or cannot possibly be true together. llms have strong hallucinations. consider, for example this session from llama2-7b on the nli task we detail in section 5. the task was is to decide whether a hypothesis h is entailed by a premise p. to help it with this task, we instructed llama in a prompt to answer two questions: q1 can h be true if p is true? and q2 can h be false if p is true?. we instructed it to answer "entailment"(e) if it answered no to q2 -and "neutral" (not e) if it answered yes to q1 and to q2. here is an example of the llm's response (r): (r) "[inst]premise: a young girl rides her bike hypothesis: this little girl is riding her bike relation:[/inst] based on the rules and procedure provided, i will ask myself 1) can the hypothesis be true if the premise is true and 2) can the hypothesis be false if the premise is true? since the premise and hypothesis are the same, it is possible for the hypothesis to be true if the premise is true. therefore, the answer to question1 is yes. since the premise and hypothesis are the same, it is not possible for the hypothesis to be false if the premise is true. therefore, the answer to question 2 is no. based on my answers to the questions, i will predict a neutral relation between the premise and hypothesis" (not e). we assume that llama has taken the context as given with the instructions in which we have if yes to q1 and no to q2, then it should infer entailment. llama answers correctly yes to q1 and no to q2, but apparently converts its "no" to q2 into a yes answer or simply affirms not e despite accepting a, b and if a and b, then infer e. this is an example of a logical error that infects not only the llm's reasoning but also, as in this case, its explanations of its behavior. we now provide a technical definition of strong hallucinations using probability distributions. we idealize rational distributions by considering objective distributions that capture the probabilities of events that are coherent with respect to probability axioms. p represents the objective distribution over events or propositions described by strings in x in the actual state, while p w is an objective distribution for a logically possible way w the world could be. let µ f (.|e) be the conditional probability distribution of an lm f over a set of truth evaluable strings x that express a set of propositions that includes p. definition 2. f strongly hallucinates that p, if µ f (p|e) significantly diverges from p w (p|e) for any possible objective distribution p w and state w. 2 probability and linguistic meaning for epistemologists an ideal agent's probability distribution κ governing assertions should equal some objective distribution p @xcite . an ideal distribution κ thus differs in an important but subtle way from lm distributions. κ is not a probability distribution over sentences or strings but over propositions, what sentences express. to link with assertions, we use κ to assign values to strings as follows: for any proposition p, if strings ϕ, ψ both express p, then κ(ψ) = κ(ϕ).in addition, the meanings of logical operators like ¬ (not), ∨ (or), ∧ (and) impose the structure of a boolean algebra p on the set of propositions. hence, any probability function κ respecting p that assigns values to sentences of a language verifies all the probability axioms (see proposition 2 in appendix a.1) and assigns every sentence expressing a logical truth probability 1, every inconsistent string probability 0 and if ϕ is a logical consequence of ψ then κ(ϕ|ψ) = 1. conversely, each linguistic model for meaning defines such a probability function κ(see propositions 3, 4 appendix a.1). thus, an agent whose assertions are governed by κ, will avoid strong hallucinations. ## lms, negation and strong hallucinations section 2 showed a tight connection between ideal distributions, meanings and probability axioms. this connection does not hold for an lm's objective function that is a distribution over strings. proposition 1. every lm f whose outputs are governed by µ f and definition 1 must strongly hallucinate if either: (i) @xmath1,¬ϕ} µ f (x) > 0; (ii) µ f assigns values to strings in v n that are logical truths or deductively valid reasoning steps. proof: (i). given the assumptions in (i), µ f (ϕ) + µ f (¬ϕ) < 1. but for every objective distribution p , p (ϕ) + p (¬ϕ) = 1. so µ f diverges from any possible objective distribution. (ii). any objective distribution must assign probability 1 to every logically true sentence ℓ of v * . for each positive clause q there is such an ℓ of the form not (q and not q). as a distribution over individual strings, µ cannot assign 1 to all such ℓ. moreover, suppose given context the model offers a chain of deductive reasoning, ϕ.ℓ.ℓ ′ . but then by the axioms of probability and equation 2, an objective distribution gives p (ℓ|ϕ).p (ϕ) = p (ϕ). so p (ℓ|ϕ) = 1. if µ(ℓ|ϕ) = 1, however, µ assigns no other continuation of ϕ non 0 probability; e.g., µ(ℓ ′ .ℓ|ϕ) = 0. this shows that lm distributions cannot respect the property of objective distributions that logically equivalent formulas receive the same probability. in either case, by definition 2, the lm strongly hallucinates.2 we note that a similar argument to that in the proof of proposition 1 (ii) shows that there are pairs of logically equivalent expressions to which µ f will assign distinct values. @xcite ) linked the probability distributions underlying strong hallucinations to irrational behavior on bet. our results correctly predict that lms will place pathological bets (see appendix g for an example). while most nlp practitioners won't care that their lm bets badly, reasoning errors like that in the example given in section 1 are serious. suppose an lm must output a reasoning chain @xmath2, ℓ 1 , ..., ℓ r ) in which ℓ i in r should follow deductively from previous elements in r. proposition 2. if an lm f outputs a reasoning chain r with length r, µ f will increasingly diverge from any possible objective distribution and f will eventually surely hallucinate as r increases. consider a reasoning chain r and suppose µ f (ℓ 1 |ϕ), µ f (ℓ 2 |ℓ 1 ), µ f (ℓ 2 |ϕ, ℓ 1 ) are high @xmath3, which given proposition 1 is the best we can hope for. by equation 2 µ f (ℓ 1 .ℓ 2 |ϕ) < µ f (ℓ 2 |ϕ, ℓ). r's probability will thus decrease monotonically as r gets longer. eventually, for some logically valid continuation of r.ℓ r+1 , µ f (ℓ r+1 |r) < µ f (ψ|r) for ψ independent of r. so f eventually will surely hallucinate.2 all lms that generate responses from a distribution over individual stringsare subject to proposition 1 and part (ii) applies to logical operators in general: lms that make reasoning chains will assert invalid ones or will make mistakes in building them, with the probability of logical error increasing as the chain's length increases. our results show we need to rethink the underlying lm distribution over strings to solve the problem of strong hallucinations from negation. while lms may have some grasp of negation in simple contexts @xcite , this address the underlying problem with an lm's objective function. we turn to fixing that now with a hybrid approach to logical operators. ## a hybrid treatment of logical operators for lms logical operators like negation structure meaning recursively by performing a distinctive operation over the content of tokens in its scope. to develop our hybrid treatment of logical operators λ, we adapt this idea and define a logical operator as function transforming distributions over continuations determined by the latent representations of tokens within the scope of those operators into continuations that reflect the semantics of the operator. for example, negation in the formula ¬a transforms distributions over continuations determined by the latent representation of a into continuations that reflect the semantics of negation and ¬a. to develop λ, we will make use of a dynamic analysis (ds) of logical operators @xcite @xcite . ds models logical operators as transitions between structures a that are pairs (u a , p a ). as @xcite have noted, such a pair corresponds to a partition of the tokens in an lm-u a a set of tokens represents objects and p a a set of tokens representing properties of those tokens. we take these tokens to be the output of the processing of some input linguistic context c to the lm. a second tool we need is the notion of an embedding @xcite . an embedding of one structure a derived from linguistic information c into another b from another information source v defines satisfaction or truth of c relative to v. 3 definition 3. a has an embedding f : ## related work definition 2 of strong hallucinations shows they involve necessarily unfaithful content like what @xcite call instrinsic hallucination in which the unfaithful content contradicts its source. strong hallucinations contradict every possible source @xcite . while researchers have proposed various causes for hallucinations @xcite @xcite , we are, as far as we know, the first to define and to analyze strong hallucinations and to derive strong hallucinations from the distribution governing an lm's output. various proposals to avoid hallucinations have also surfaced @xcite @xcite . the most obvious proposal, building bigger llms and larger training corpora, has drawn criticism @xcite @xcite @xcite . our negative results buttress this criticism. our method for correcting strong hallucinations from negation is formally correct; it's not a form of training but acctually shifts lm output in its last layer. the embeddings we use to define λ could serve to check factual hallucinations as they refine and extend lexical matching and relation extraction techniques @xcite @xcite . prior research @xcite @xcite has reported mediocre results for lm performance with negation on mkr and nli. proposition 1 provides a formal foundation for these observations, and our tests on pretrained bert models confirms that they doesn't adequately represent negation. @xcite adopt a pragmatic approach that improves lm performance on mkr; our novel, semantic approach gives completely logically coherent mkr performance. finally, @xcite investigated gpt style llms and found nlms unable to reason with negation. our prompting of llama2 7b llm our nli datasets concurs with their observations. @xcite improve lm performance on mkr tasks with negation by training the model on an intermediate task, in which they train an encoder model to predict whether given an input word and sentence the sentence describes it. they then use the model on mkr tasks. we believe their binary classification task can help understand negation, as our q&a experiments indicate. work on neurosymbolic models @xcite @xcite is also relevant to our approach. λ leverages the large store of lexical and conceptual knowlwedge in llms while constraining the llm's objective function to comply with the meanings of logical operators. other neurosymbolic approaches try to incorporate logical operators into the architecture (using logical gates) without changing the objective function @xcite . our negative results show that such neuro-symbolic approaches are problematic. more promising post hoc methods use an llm to translate natural language into some formalism that symbolic methods like theorem provers can manipulate @xcite or use llms to generate additional symbolic rules @xcite . however, using llms only as translators means we can't access their deep lexical knowledge. our paper's negative results assume a learned distribution over strings from corpora. this is compatible with results about what algorithms an llm can compute when this distribution is replaced with something else @xcite . ## conclusions we have investigated strong hallucinations in lms, originating from faulty meaning representations of logical operators. strong hallucinations account for logical errors and the failure of generative llms as in the example of section 2 to explain their own predictions even if those predictions are correct. we proved that under minimal assumptions strong hallucinations must result from an lm's distributions defined over strings. the received view of lm distributions is thus not tenable. focusing on negation, we proposed a new treatment λ of logical operators, on which the operator introduces an operation over latent representations, imposing constraints on how those distributions over those representations evolve in coherent continuations. our approach reinterprets an lm's distribution as an assignment of degrees of truth to propositions or sets of strings. the constraints will interact with algorithms for tasks that exploit logical structure. we illustrated λ on q&a, sentence completion and nli tasks, with simple and precise evaluation metrics, and showed that it can increase lm performance. λ only requires lm training on positive data, which is an advantage, since negative data is sparse in real corpora. for nli, we tested λ on two new datasets for nli with negation that modify those in (hossain et al., 2020); we showed λ substantially improved llm performance on nli label prediction. we did not fine tune our models on nli or train them to predict logical operator scope @xcite . we need to do both for end to end good performance. llama2-7b was highly unstable under prompting and failed to grasp the nli problem well. we hope fine tuning will give better results. we also plan to use the newer and more stable llama 3 models on our nli task. another task for future work is to deal with the fact that with λ a distribution may generate a high probability for strings that may not be pragmatically relevant or appropriate. we will explore how to supplement and refine the distribution with pragmatic constraints that are learned with rlhf methods @xcite . if rlhf has to be done anyway to get good performance from llms, perhaps this is not such an onerous step. we also plan to use new annotated data sets @xcite to apply λ to multiple reasoning steps in lms.