FENICE : Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (2024)

Abstract: Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims , extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization. In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at https://github.com/Babelscape/FENICE .

## introduction in recent years, natural language generation (nlg) approaches have achieved impressive results in many areas of natural language processing (nlp), including machine translation (mt), and question answering (qa). when it comes to text summarization, the landscape differs somewhat. despite the high performance boasted by conventional metrics such as rouge @xcite , several empirical investigations @xcite @xcite revealed that a significant portion of automaticallygenerated summaries contains factual inconsistencies, e.g., hallucinations. addressing this issue is crucial, as it stands as a barrier preventing the widespread application of such systems in realworld scenarios. with the emergence of large language models (llms) the challenge still persists, as their outputs continue to exhibit inconsistencies @xcite . what is concerning is that the heightened fluency boasted by these models @xcite may inadvertently lead inexpert users to place excessive trust in their outputs, thereby fostering the spread of misinformation. for these reasons, the introduction of reliable automatic metrics that detect factual inaccuracies in summaries is becoming increasingly urgent. as a response to this need, various factuality metrics have emerged, leveraging methodologies rooted in question answering @xcite @xcite , natural language inference @xcite @xcite , and semantic representations @xcite @xcite . the latest trend in this domain involves works @xcite @xcite that introduce prompt engineering approaches using llms. however, current factuality metrics suffer from one or more of the following drawbacks: i) they often assign a broad score to the entire summary without specifying which parts are factual or hallucinated, thereby lacking interpretability; ii) they are mainly designed for evaluating summaries of short documents, such as news articles, and iii) they are impractical from a computational standpoint, specifically in the case of llm-based metrics. to overcome these limitations, we introduce factuality evaluation of summarization based on natural language inference and claim extraction (fenice), a state-of-the-art factuality metric that is more interpretable and efficient. as depicted in figure 1 , fenice evaluates the factuality of a summary by leveraging an nli-based alignment between information in the source document and a collection of atomic facts, referred to as claims, extracted from the summary. our metric aligns claims from the summary with text from the input at three different levels of granularity: sentence, paragraph, and document. this novel alignment strategy facilitates interpretability: when presented with a claim, the metric user can either verify which specific part of the input document entails it, or, in the presence of hallucinations, they can observe that no alignment is provided by the metric. not only does our approach enhance interpretability but it also yields a positive impact on overall performance: fenice achieves the highest average result in the aggrefact benchmark @xcite , the de-facto benchmark for summarization factuality evaluation. furthermore, we evaluate the performance of our metric in a more challenging context, specifically targeting long-form summarization. to accomplish this, we conduct a human annotation process aimed at assessing long-form summarization factuality. when confronting the metrics outputs with the human labels, fenice exhibits higher accuracy compared to its competitors, paving the way for future approaches for long document summarization evaluation. in summary, our contributions are the following: in the same context, alignscore @xcite provides a novel alignment system, which is trained on several verification tasks, including nli. while it achieves high performance in factuality evaluation, alignscore also falls short in terms of interpretability, providing an overall score for the entire summary, which prevents users from identifying specific factual or non-factual spans within it. moreover, it aligns the summary with large blocks of text from the input document, making it hard for users to gain insights from the output. the dependency arc entailment (goyal and durrett, 2020, dae) metrics leverage the entailment between the dependency parse trees of document sentences and the summary. however, the highly abstractive nature of summaries, often encapsulating multiple facts in a single sentence, introduces a drawback, making their sentence-level comparison suboptimal. despite the significant progress achieved so far, we posit that there is still considerable room for enhancing nli-based metrics, encompassing improvements in both performance and interpretability. llm-based metrics. several studies @xcite @xcite have recently analyzed the efficacy of llms as summarization evaluators in multiple evaluation settings. their experiments demonstrate that, while llms outperform existing automatic metrics, they are not yet ready to be considered reliable alternatives for human evaluation. starting from these findings, trueteacher @xcite analyzed the llms' capability of generating large-scale factuality datasets and then trained smaller student models on such data, outperforming existing nli metrics. another trend of approaches revolves around prompt engineering strategies @xcite @xcite , where llms are typically instructed with the evaluation task description, the input document, and the system-generated summary. specifically, @xcite demonstrated the capabilities of such models in achieving multi-aspect, customized, and training-free evaluation. relatedly, the recentlyintroduced g-eval @xcite experimented with chain-of-thoughts (cot) and form-filling instruction paradigms. they showed that g-eval, configured with gpt-4, outperforms all previous methods in terms of correlation with the human consistency annotations in summeval @xcite . while achieving high correlations with human judgments, llm-based metrics raise concerns about their real-world applicability. firstly, many of these studies rely on closed-source llms, and the lack of information about the composition of their training dataset raises uncertainty: there is no guarantee that the test examples used in these studies are actually unseen at training time for these models. additionally, their exceptionallyhigh computational cost poses a significant constraint on their practical deployment and hampers their reproducibility for research purposes. our approach differs from previous ones in that it utilizes an llm solely for the task of claim extraction, rather than also utilizing it for generating the factuality annotations, which it may have already seen during training. furthermore, we provide a distilled version of this llm-a more compact generative transformer-trained to emulate the output of the llm, which ensures the practicality and reproducibility of our approach. ## fenice we now provide an overview of the fenice algorithm. initially, we extract a list of atomic facts, referred to as claims, from a given summary (section 3.1). subsequently, we detail the process of obtaining a factuality score for each extracted claim. this involves leveraging nli, with the claim serving as the hypothesis, and the document sentences (section 3.2) or longer blocks of text at different levels of granularity (section 3.4) serving as premises. in section 3.3, we elaborate on a refinement of the obtained scores through coreference resolution. ## conclusion and future work we introduced fenice, a novel metric designed for the factuality evaluation of summarization. by leveraging nli-based alignments between claims extracted from the summary and the source text at multiple levels of granularities, fenice facilitates interpretability. our experiments demonstrate that our metric attains state-of-the-art performance in standard summarization evaluation datasets. additionally, we showcase the effectiveness of our metric in a more-challenging setting, i.e., the evaluation of long-form summarization. to do so, we conducted a human annotation process for this task. our work paves the way for future research focused on assessing long-form summarization factuality. as future work, we plan to improve our zero-shot alignment system's performance with a tailored fine-tuning on summarization datasets.