Chain-of-Verification Reduces Hallucination in Large Language Models (2024)

Abstract: Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models. We study the ability of language models to deliberate on the responses they give in order to correct their mistakes. We develop the Chain-of-Verification (CoVe) method whereby the model first (i) drafts an initial response; then (ii) plans verification questions to fact-check its draft; (iii) answers those questions independently so the answers are not biased by other responses; and (iv) generates its final verified response. In experiments, we show CoVe decreases hallucinations across a variety of tasks, from list-based questions from Wikidata, closed book MultiSpanQA and longform text generation.

## introduction large language models (llms) are trained on huge corpora of text documents with billions of tokens of text. it has been shown that as the number of model parameters is increased, performance at tasks such as closed book qa improve in accuracy, and larger models can generate more correct factual statements @xcite . however, even the largest models can still fail, particularly on lesser known torso and tail distribution facts @xcite , i.e. those that occur relatively rarely in the training corpora. in those cases where the model is incorrect, they instead generate an alternative response which is typically plausible looking (e.g., a similar entity, but an incorrect one). these factually incorrect generations are referred to as hallucinations @xcite . further, in longform tasks consisting of generating multiple sentences or paragraphs, the hallucination problem can be exacerbated due to the issue of exposure bias @xcite . the current wave of language modeling research goes beyond next word prediction, and has focused on their ability to reason. improved performance in reasoning tasks can be gained by encouraging language models to first generate internal thoughts or reasoning chains before responding @xcite @xcite , as well as updating their initial response through self-critique @xcite . in this work we follow this line of research to study how and when languagemodel-based reasoning can be used to reduce hallucinations. we develop an approach, called chainof-verification (cove) which, given an initial draft response, first plans verification questions to check its work, and then systematically answers those questions in order to finally produce an improved revised response. we find that independent verification questions tend to provide more accurate facts than those in the original longform answer, and hence improve the correctness of the overall response. we study variations on this recipe across a range of tasks: from list-based questions, closed booked qa and longform text generation. we first propose a joint approach for generating the entire verification chain left-to-right, which improves performance and decreases hallucinations compared to the baseline language model. however, models that attend to existing hallucinations in the context from their own generations tend to repeat the hallucinations. hence we also introduce further improvements with factored variants which separate out the verification chain steps, in terms of which context is attended to. we show how these factored variants give further performance gains across all three tasks considered. ## related work hallucination is a general problem in language model generations that appears across many tasks, from summarization @xcite to opendomain dialogue @xcite , and has not been resolved by simply scaling up training data or model size @xcite . for a survey of the hallucination issue, see @xcite and @xcite . a majority of the methods for reducing hallucination can be divided into roughly three categories: training-time correction, generation-time correction and via augmentation (tool-use). in training-time correction methods, an attempt is made to improve the raw left-to-right generations of an encoder-decoder or decoder-only language model by either training or otherwise adjusting the model weights to decrease the probability of hallucinated generations. this includes using reinforcement learning @xcite , constrastive learning @xcite and other methods @xcite . in generation-time correction, a common theme is to make reasoning decisions "on top of" the base llm in order to make them more reliable. for example, by considering the probabilities of the generated tokens @xcite . in @xcite multiple samples are drawn from the model to detect hallucinations. in @xcite hallucinations are identified using low confidence scores, and their correctness is checked through a validation procedure, mitigated, and then the generation is continued. an alternative to using the confidence scores is to leverage inconsistencies in the llms output to detect hallucination. @xcite show that using inconsistencies for qa tasks can outperform using confidence scores for hallucination detection. @xcite simulate an interactive multi-agent lm vs. lm debate setup to detect hallucinations for factoid qa, @xcite check for hallucinated references, while @xcite extract relational triples from generations and verify them against another llm. cove also uses a related self-consistency approach but we show our approach can correct hallucinated facts in longform generations by generating and answering verification questions by solely using the same llm. a third approach is to use external tools to help mitigate hallucinations, rather than relying solely on the abilities of the language model itself. for example, retrieval-augmented generation can decrease hallucinations by using factual documents for grounding @xcite @xcite or chain-of-thought verification @xcite . other approaches include using tools for fact-checking @xcite @xcite , or linking to external documents with attribution @xcite @xcite . there are also a number of related works in improving reasoning for logical and mathematical tasks, even if they do not address reducing hallucination explicitly. several approaches have been shown to improve results with extended reasoning steps by the system, such as chain-of-thought @xcite , deductive verification @xcite , and self-verification @xcite @xcite . the latter tries to predict the (masked) question given the answer for math problems, and use that as evidence that this is the correct solution. our overall process, which we call chain-of-verification (cove), thus performs four core steps: ## experiments we use various experimental benchmarks to measure the efficacy of cove in reducing hallucination, comparing against a number of baselines. ## conclusion we introduced chain-of-verification (cove), an approach to reduce hallucinations in a large language model by deliberating on its own responses and self-correcting them. in particular, we showed that models are able to answer verification questions with higher accuracy than when answering the original query by breaking down the verification into a set of simpler questions. secondly, when answering the set of verification questions, we showed that controlling the attention of the model so that it cannot attend to its previous answers (factored cove) helps alleviate copying the same hallucinations. overall, our method provides substantial performance gains over the original language model response just by asking the same model to deliberate on (verify) its answer. an obvious extension to our work is to equip cove with tool-use, e.g., to use retrieval augmentation in the verification execution step which would likely bring further gains.