Detecting Errors through Ensembling Prompts ( DEEP ): An End-to-End LLM Framework for Detecting Factual Errors (2024)

Abstract: Accurate text summarization is one of the most common and important tasks performed by Large Language Models, where the costs of human review for an entire document may be high, but the costs of errors in summarization may be even greater. We propose Detecting Errors through Ensembling Prompts (DEEP) - an end-to-end large language model framework for detecting factual errors in text summarization. Our framework uses a diverse set of LLM prompts to identify factual inconsistencies, treating their outputs as binary features, which are then fed into ensembling models. We then calibrate the ensembled models to produce empirically accurate probabilities that a text is factually consistent or free of hallucination. We demonstrate that prior models for detecting factual errors in summaries perform significantly worse without optimizing the thresholds on subsets of the evaluated dataset. Our framework achieves state-of-the-art (SOTA) balanced accuracy on the AggreFact-XSUM FTSOTA, TofuEval Summary-Level, and HaluEval Summarization benchmarks in detecting factual errors within transformer-generated text summaries. It does so without any fine-tuning of the language model or reliance on thresholding techniques not available in practical settings.

## introduction the advancement of cutting-edge large language models (llms) like gpt-4, claude 3, llama-2, and gemini variants introduces a significant challenge: despite producing content that is linguistically coherent, their outputs frequently contain misleading or false information, often referred to as hallucinations or factual inconsistencies. hallucinations in large language models refer to instances where the model generates usually plausible but entirely fabricated information. factual inconsistencies, a specific type of hallucination, 1 code and data are available on github. traditional automatic evaluation methodologies like rouge @xcite , meteor @xcite , and bleu @xcite have been instrumental in assessing natural language generation tasks. however, numerous studies demonstrate the lack of correlation between initial automatic evaluation models and human judgment in tasks such as machine translation @xcite , image captioning @xcite , and notably, factuality @xcite . in particular, these models struggle to capture semantic equivalence when there are substantial discrepancies in length, syntax, and wording between two texts @xcite . consequently, specialized models @xcite @xcite have been developed to assess textual factual consistency, verifying the truthfulness of a claim or summary based on given ground truth textual content. however, existing models, often fine-tuned variants of @xcite for assessing factual consistency, exhibit significant limitations. as highlighted in @xcite , these models show reduced effectiveness in detecting factual inconsistencies in content produced by recent state-of-the-art text-generating models. ensemble learning is the practice of merging the outputs of multiple models to produce a more accurate prediction @xcite . @xcite demonstrated that ensembling factual consistency models by calculating their weighted mean surpassed the performance of individual models in detecting hallucinations within a small dataset of gpt-3generated wikipedia abstractive summaries. in light of these limitations, this study evaluates benchmarks exclusively featuring summaries from recent transformer-based language models. this approach more accurately reflects actual usage scenarios, where users commonly need to validate texts generated by newer llms rather than texts from older text generation models. we assess factual consistency using the aggrefact-xsum ftsota, tofueval summary-level, and halueval summarization @xcite @xcite benchmarks, consisting of transformer-generated abstractive summaries featuring hallucinations that existing models struggle to identify. existing factual consistency encoder models output numerical scores, requiring thresholding to map the scores to binary labels. however, @xcite demonstrates that the optimal threshold for factual consistency models varies depending on the recentness of the summarization model within the aggrefact dataset. previous studies that report the performance of these factual consistency models have fine-tuned each model's threshold using the development subset of the same dataset under evaluation @xcite @xcite . this approach is problematic and unrealistic, as it assumes access to labeled data from the target dataset, which may not be available in real-world scenarios. in this study, we benchmark five popular state-of-the-art factual consistency models and demonstrate a substantial decline in performance when thresholds are learned from different datasets or set to their default midpoint. we extend the findings of @xcite and discover that the optimal threshold for each factual consistency model varies widely across different datasets, even when evaluating text generated solely from recent summarization models. figure 1 reveals that, even when considering only datasets with summaries from recent transformer models, the optimal linear threshold for maximizing balanced accuracy differs widely for each factual consistency model, covering a broad range of their possible output scores. 2 moreover, figure 2 demonstrates that optimizing thresholds on nontest data or setting them to the midpoint of each model's score range substantially reduces balanced accuracy compared to test-set optimization. the reliance of these models on dataset-specific thresholds, as demonstrated by our findings, limits their practical utility in evaluating factual consistency 2 questeval, summac-conv, and alignscore (with mode nli_sp) generate scores from 0 to 1. summac-zs scores range from -1 to 1. qafacteval scores range from 0 to 5. across a diverse range of text without further finetuning or adjustments. showed that these existing prompts had shown poor performance on detecting factual consistencies over the aggrefact ftsota benchmark of transformergenerated summaries. . however, reliably mapping the generated token probabilities from the reasoning steps to the final answer's confidence remains unclear, especially considering factors like temperature and sampling methods that influence token generation. existing llm solutions are frequently overconfident in their assessments of a text's factual consistency. @xcite showed that their summary-level binary factual consistency prompts, when used with gpt-4, frequently failed to identify sentences with factual errors, resulting in false positive rates of 69% and 46% on the mediasum and meetingbank subsets of the tofueval summary-level dataset, respectively. this issue of overconfidence is not unique to their approach but rather a general problem that neural networks, including llms, tend to be overconfident in their predictions @xcite @xcite ). confidence elicitation, an increasingly popular method, involves prompting the llm to output its uncertainty along with its prediction @xcite @xcite . despite its potential, no confidence elicitation approach has yet consistently yielded accurate confidence estimates across diverse llms and tasks, limiting its current practical utility. calibration, the process of adjusting model-predicted probabilities to match their empirical accuracies, has remained the leading solution to correct model overconfidence @xcite . we propose a novel approach -crafting a diverse set of llm prompts that each output a binary score, indicating each prompt's belief of whether the summaries contain any factual errors. these bitributing to the overconfidence observed in their responses. the primary contributions of our work can be outlined as follows: 1: we demonstrate that prior methods for detecting factual errors in summarizations perform significantly worse without the common practice of an optimized threshold on subsets of the dataset under test. 2: we introduce a large and diverse set of prompts, each employing unique methods and evaluation protocols to detect hallucinations and factual inconsistencies in generated summaries. 3: our end-to-end framework achieves state-of-the-art balanced accuracy on the aggrefact-xsum ftsota, tofueval summary-level, and halueval summarization benchmarks in detecting factual errors in transformer-generated text summaries, all without fine-tuning the language model or relying on impractical thresholding techniques. ## datasets the aggrefact ftsota benchmark @xcite tests a model's ability to identify factual inconsistencies in summaries produced by finetuned transformer-based summarization models. the dataset combines nine existing annotated factuality datasets, converting all factual consistency scores to binary. aggrefact is categorized based on the development timeline of the underlying summarization models into ftsota, exformer, and old categories, and is divided into aggrefact cnn/dm and aggrefact-xsum subsets. we benchmark aggrefact-xsum ftsota and exclude its cnn/dm counterpart due to its insufficient number of factual inconsistencies and its more extractive summary style, compared to the more abstractive style of current sota llms. halueval @xcite , a comprehensive benchmark for assessing hallucination in llms, uses a chatgpt-based two-step sampling-thenfiltering framework to create the dataset. we focus determine if the summary is consistent or inconsistent with the provided context. context: {...} summary: {...} output 0 if the text is inconsistent and 1 if the text is consistent. ## conclusion we introduce detecting errors through ensembling prompts (deep), a state-of-the-art llm-based method for detecting factual consistencies and hallucinations in summaries. our findings reveal that factual consistency encoder models exhibit a prothereby ensuring that differing sample sizes do not affect our statistical significance testing. nounced sensitivity to threshold settings, with their performance markedly declining if the threshold is not adjusted based on the test datasets. we demon-strate that deep surpasses the performance of existing methods and models in evaluating the factual consistency of summaries produced by recent trans-former models. finally, we show that by calibrating ensemble models using binary input features derived from llm prompts, deep achieves reliable probabilities indicating a text's factual consistency or presence of errors.