M ini C heck: Efficient Fact-Checking of LLM s on Grounding Documents (2024)

Abstract: Recognizing if LLM output can be grounded in evidence is central to many tasks in NLP: retrieval-augmented generation, summarization, document-grounded dialogue, and more. Current approaches to this kind of fact-checking are based on verifying each piece of a model generation against potential evidence using an LLM. However, this process can be very computationally expensive, requiring many calls to a model to check a single response. In this work, we show how to build small fact-checking models that have GPT-4-level performance but for 400x lower cost. We do this by constructing synthetic training data with GPT-4, which involves creating realistic yet challenging instances of factual errors via a structured generation procedure. Training on this data teaches models to check each fact in the claim and recognize synthesis of information across sentences. For evaluation, we unify datasets from recent work on fact-checking and grounding LLM generations into a new benchmark, LLM-AggreFact. Our best system MiniCheck-FT5 (770M parameters) outperforms all systems of comparable size and reaches GPT-4 accuracy. We release LLM-AggreFact, code for data synthesis, and models.

## introduction freeform generation of responses is a flexible way to employ large language models (llms) for question answering, summarization, and beyond. however, this kind of generation can lead to factual errors, the "hallucination" problem in llms @xcite @xcite . such errors arise in generation settings where an llm is prompted closed-book, but its parametric knowledge may be insufficient to produce the right facts @xcite ., 1 https://github.com/liyan06/minicheck ; see latest leaderboard at llm-aggrefact.github.io. fixed-doc genera/on (e.g., tofueval) post-hoc grounding (e.g., factcheck-gpt) retrieve-then-generate (e.g., claimverify) summarize the document(s): what/why/… ? what/why/… ? retrieved docs to verify answer the doc describes… according to [1], the answer is … retrieved web docs the answer is… sentences fact-check ( , ) this work: generate synthe/c data to train a sentence-level fact-checker ## methodology: training data synthesis to address these challenges, new data is required. existing datasets like mnli @xcite and anli @xcite do not feature instances that reflect the complexity of llm factchecking. annotation of real errors is challenging to scale; datasets of such errors (including those in llm-aggrefact) are largely test-only. our goal is to construct a dataset {(d i , c i , y i )} n @xmath0, 1}, using two novel synthetic data generation methods (figure 3 ). statistics about our final synthetic training data can be found in ## llm-aggrefact benchmark we construct a fact verification benchmark, llm-aggrefact, by aggregating 10 of the most upto-date publicly available datasets on factual consistency evaluation across both closed-book and grounded generation settings. characteristics in llm-aggrefact, all datasets contain human-annotated (document, claim, label) tuples. the documents come from diverse sources, including wikipedia paragraphs, interviews, web text, covering domains such as news, dialogue, science, and healthcare. the claims to be verified are mostly generated from recent generative models (except for one dataset dataset feature fixed-doc generation type post-hoc grounding retrieve-then generate written claims claimverify expertqa factcheck-gpt wice aggrefact (cnn/xsum) tofueval (medias/meetb) (check-worthy) sentences from llms/ search engines' responses to search queries summaries from sota fine-tuned summarizers topic-focused dialogue summaries from llms wikipedia claims with citations lfqa reveal expertqa figure 4: 10 datasets in llm-aggrefact. details of these datasets as well as related but excluded datasets can be found in appendix c. an overview of the llm-aggrefact is shown in figure 4 , with statistics and detailed dataset descriptions in appendix c. ## experimental setup we include the following specialized fact-checkers: t5-nli-mixed @xcite , dae @xcite , qafacteval @xcite , summac-zs and summac-cv (laban et al., 2022), alignscore @xcite , and ft5-anli-l that fine-tunes flan-t5-large on the full anli training set. a meta-comparison of those specialized fact-checkers and our models can be found in we also include the following llms as factcheckers: gemini-pro @xcite , palm2-bison @xcite , mistral-8x7b, mistral-large @xcite , claude 2.1, claude 3 opus @xcite , gpt-3.5 and gpt-4 (openai, 2023). more details about the models can be found in appendix e.1. for the llm-based fact-checkers, we adapt a prompt from @xcite for zero-shot prediction, which can be found in appendix h. ## rethinking llm fact-checking we now revisit two other stages of the typical llm fact-checking pipeline: claim decomposition and decontextualization. surprisingly, we find that claim decomposition is not needed in our settings, contradicting prior work @xcite . furthermore, we find that decontextualization doesn't help on our benchmark, ## conclusion in this work, we introduce two synthetic data generation methods that address key limitations of specialized fact-checkers by encouraging models to verify each atomic fact within a claim and reason across multiple sentences. we also present llm-aggrefact, a new factual consistency evaluation dataset covering both closed-book and grounded generation settings. a model fine-tuned on our synthetic data outperforms all prior specialized factcheckers on llm-aggrefact, while being much cheaper than llm-based fact-checkers.