Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output (2024)

Abstract: We present a light-weight approach for detecting nonfactual outputs from retrieval-augemented generation (RAG). Given a context and putative output, we compute a factuality score that can be thresholded to yield a binary decision to check the results of LLM-based question-answering, summarization, or other systems. Unlike factuality checkers that themselves rely on LLMs, we use compact, open-source natural language inference (NLI) models that yield a freely accessible solution with low latency and low cost at run-time, and no need for LLM fine-tuning. The approach also enables downstream mitigation and correction of hallucinations, by tracing them back to specific context chunks. Our experiments show high ROC-AUC across a wide range of relevant open source datasets, indicating the effectiveness of our method for fact-checking RAG output.

## introduction with natural language understanding applications increasingly relying on large language models (llms) to answer questions, summarize texts, and perform other tasks, detecting nonfactual claims in the generated text has become critical from an ethical and compliance standpoint. llms, while powerful, are prone to generate nonfactual or "hallucinated" information that can lead to misinformation and introduce errors in business processes. to address this problem, we present provenance, a fact-checking method for output generated by llms, with respect to a given context that provides the factual basis for the output. provenance leverages compact cross-encoder models that offer substantial advantages over conventional llm-based methods. these advantages include accessibility, low latency/high throughput, and interpretable judgments. provenance is evaluated on diverse open-source datasets, including the true dataset @xcite , msmarco @xcite , truthfulqa @xcite , hotpotqa @xcite , halueval @xcite and @xcite . these datasets encompass a variety of question-answering contexts, providing a robust testbed for our methods. we assess performance using standard detection metrics to demonstrate our method's efficacy as a factuality checker for llm-generated content. our findings show that provenance achieves competitive hallucination detection performance (as measured by auc) across different datasets, thus contributing to improved trustworthiness and utility of llms in real-world applications. ## related work in prior work, three main approaches to factuality evaluation have been used: 1. llm ablation, 2. llm introspection, and 3. nli methods. llm ablation refers to approaches such as self-checkgpt @xcite and @xcite that measure the consistency of multiple candidate generations for a given prompt. methods such as @xcite that gauge factuality based on the language model's output distributions also fall in this category. llm introspection refers to techniques that use the reasoning ability of modern language models to evaluate their own or another model's output. work by @xcite , @xcite and @xcite are examples of this. natural language inference (nli) methods exploit special-purpose cross-encoder models that indicate whether a claim is supported by a premise. this approach usually involves breaking down the context into a list of premises (context items), and the generation into a list of claims. @xcite is a representative method that chunks the generation and context at the sentence level and computes pair-wise entailment judgments, which are then aggregated. however, this approach has some shortcomings: 1. the original prompt/query is ignored when evaluating entailment, and 2. context and generation chunking is overly simplistic. our method falls into the nli-based category, but addresses these shortcomings. broadly speaking, there are four scenarios (fig. 1 ) in which a fact-checker may operate: 1. short context/short answer, 2. short context/short answer, 3. short context/long answer, and 4. long context/long answer. when the answer or context are long, we need a mechanism to break them into smaller units. we narrow our focus based on the following observations and practical considerations: while we have a straightforward way to break down the contexts, it is still hard to chunk generated answers meaningfully. the chunking of information is an area for further research, since context and answers come in many forms, such as text, conversations, and tables. we limit the scope of this paper to text source for scenarios in the first row of figure 1 , namely, short context/short answer and long context/short answer. we also need to ensure that the chunk length chosen is viable for all the models in the system. ## system description contemporary fact-checking systems employ approaches based on llms as a judge @xcite to validate the generations of other llms. by virtue of being auto-regressive, the judge-llms themselves are prone to hallucinate. by contrast, provenance (figure 2 ) uses two cross-encoder based models that do not suffer from this tendency. as input, provenance expects 1. a list of context items used by the generating llm in the upstream rag, 2. the user's original question or query, and 3. the generated text to fact-check. the first cross-encoder model determines which of the context items are relevant to the given query and generates a score. this score is then used to select context items to build a smaller and more focused context, which we refer to as the sources. the selection process also produces a weight associated with each source. in parallel. we construct the claim by inserting the query and generation into a claim prompt. the claim and sources are then passed to the second cross-encoder model for validation, generating a factuality score for each claim/source pair. these scores are then aggregated using the source weights generated earlier to produce a single score for the llm's output. this score can be thresholded to produce a binary factuality decision, with the threshold being tuned for a target dataset and task. here we used thresholdinvariant evaluation methods, such as receiver operator characteristics (roc) and area under the curve (auc). ## data we utilize several open-source datasets to evaluate the effectiveness of our approach in detecting nonfactual texts generated by llms. these datasets provide a diverse range of question-answering contexts and candidate answers, ensuring a comprehensive assessment. ## data preparation the msmarco and hotpotqa datasets each contain 10 sources per question, with one relevant source per question in msmarco and multiple relevantsources per question in hotpotqa. other datasets have a single source paragraph given for each question. all sources were split into individual sentences, and all datasets were converted into triplets with the query and answer as strings, and the sources as a list of strings.our framework processes these triplets and returns a score, which, combined with a set threshold, classifies the generated answer as hallucinated or factual. to calculate auc, we ensured representation of the two classes by generating hallucinated answers for datasets lacking them. for the msmarco dataset @xcite , we randomly selected 252 out of 100,000 datapoints and generated hallucinated answers using the gpt-3.5-turbo model, which were verified manually. for hotpotqa @xcite , we appended the qa data from halueval @xcite , which includes 10k hallucinated samples based on hot-potqa. ## experiments 6.1 preliminary experiments before developing our final provenance framework, we also experimented with a bert-based relevancy scorer using topp selection and a debertabased nli model for computing factuality scores. these preliminary experiments showed the importance of (1) sorting of selected sources into ## results we report the roc auc of our system for all datasets mentioned in section 4. the roc curves in figures 3 and 4 show the trade-off between false versus missed hallucination detections for the least figure 4: roc curve for halueval-general task @xcite . and the most difficult of the test sets, respectively. note that we did not reproduce the evaluations of the llm-based methods listed in tables 4 and 6 , and simply copied the results reported in the respective references. ## conclusion we have presented provenance, a practical approach to fact-checking of llm output in rag scenarios, based on light-weight cross-encoder models for relevance scoring and natural language inference. the factuality scoring takes the query into account when judging a generated answer against the retrieved information sources. evaluation on a variety of open-source datasets shows our method to be effective for hallucination detection across a variety of tasks, at a model size that is a fraction of that of llms that are commonly used for this task. we expect our method to make the fact-checking of llm output more accessible and scalable, contributing to the reliability and trustworthiness of llm-based applications.