F ine S ur E : Fine-grained Summarization Evaluation using LLM s (2024)

Abstract: Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and time-consuming nature of human evaluation. Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likert-scale scores. This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations. To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs). It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment. We compare various open-source and proprietary LLMs as backbones for FineSurE. In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions. The code is available at https://github.com/DISL-Lab/FineSurE.

## introduction text summarization stands out as an important task in natural language processing, aiming to generate a condensed summary of a provided text while retaining its essential information @xcite . despite the enhanced quality of summaries produced by llms, the development of automated methods for evaluation remains a challenge @xcite . conventional reference-based metrics, such as rouge @xcite , have exhibited a weak correlation with actual human judgments @xcite . consequently, human evaluation remains an essential step for accurately assessing the quality of generated summaries, even considering its inherent costs and time-consuming nature. recently, the need for better automatic evaluators has become an important research topic, aiming to streamline evaluation processes and ease manual efforts in model development @xcite . this effort provides valuable insights into whether generated summaries align with predefined quality standards, including aspects like faithfulness. various approaches have been explored, including approaches based on neural language inference (nli) @xcite and questionanswering (qa) @xcite . in addition, llms have recently proven their potential to be an automated tool for humanlike evaluation @xcite . the latest llm-based method, g-eval @xcite , demonstrated a spearman correlation coefficient of over 0.5 with likert-scale human judgments on the news domain using gpt-4. despite these advancements, we contend that the current llm-based automated methods still fall short in achieving precise evaluation, primarily attributed to the coarse-grained evaluation pipeline and the ambiguity in evaluation dimensions. specifically for coarse-grained evaluation, the evaluation dimensions-namely faithfulness, coherence, and relevance foot_1 -are frequently assessed at the summarylevel, resulting in likert-scale scores for each summary @xcite @xcite . this likertscale scoring method lacks fine-grained information on errors in generated summaries. for instance, it does not provide a breakdown of the number of summary sentences with quality issues or specify bulgaria might not be the first hotspot that springs to mind for a summer holiday 2013 but that could change after it was named europe 's cheapest destination yesterday . the former soviet state has gained the most from the strong pound , which has cut the cost of visiting the continent . resorts on its black sea coast offer the best value in terms of a meal out (…) input text generated summary [s1] bulgaria's most popular resort of sunny beach is a carbone copy. figure 1 : finesure framework: the given summary is evaluated by conducting the two tasks of fact checking and keyfact alignment. in this specific example, the faithfulness score is 33%, since only one out of the three summary sentences is factually correct; the completeness score is 75%, since three out of the four keyfacts align with the summary; and the conciseness score is 66%, since two out of the three sentences are related to the keyfacts. the types of mistakes present in each sentence. furthermore, regarding ambiguity, the evaluation of coherence and relevance is hindered by the lack of clarity in their definition of "the collective quality of all sentences" and "the selection of important content from the source" @xcite . given that human can encounter challenges in evaluating summaries, it is inappropriate to expect a neural model to provide accurate and objective assessments. hence, there is a need to develop a more precise evaluation framework that results in a more detailed assessment with clearly defined evaluation dimensions. in this paper, we present finesure (finegrained summarization evaluation) using llms, a novel automated approach designed to evaluate the summarization quality at a fine-grained level based on summary sentences or keyfacts foot_3 , as depicted in figure 1 . we aim to evaluate summaries using this framework along three vital criteria: the faithfulness of minimizing factuality errors, the completeness of encompassing the majority of keyfacts, and the conciseness of avoiding unnecessary details. thus, our framework entails executing two finely grained procedures utilizing llms: (1) fact checking involves identifying specific factuality errors present in each summary sentence and (2) keyfact alignment focuses on aligning each keyfact with all summary sentences from which they are inferred. we leverage the outcomes from both procedures to calculate precise percentage scores, offering a more detailed assessment than likertscale scoring. this fine-grained approach enables us to analyze the quality issues of generated texts at both the sentence and keyfact-level. on top of keyfact alignment, two dimensions (completeness and conciseness) can serve as better replacements for coherence and relevance @xcite , as they evaluate two key aspects of a good summary, assessing the comprehensive inclusion and density of key information while excluding irrelevant content. to summarize, our main contributions are as follows: (1) we argue that llm-based summarization suffers from hallucination, information emission and verbosity hence requiring revisiting the evaluation dimensions, (2) we suggest three metrics targeting llm output characteristics and tackling the aforementioned problems including faithfulness, completeness and conciseness, (3) we propose a novel automated evaluation framework -finesure, based on keyfact lists and using llms to generate the keyfacts, align them to the summary sentences and categorize the errors automatically, (4) we compare various open-source and proprietary llms to power finesure and analyze their correlation to human judgment at the summary and system levels, (5) we provide comprehensive results comparing our method with similarity-based, nli-based, qabased, and llm-based automated methods and show improved human correlation for finesure over state-of-the-art methods. ## related work efforts to assess the quality of texts generated by language models have led to numerous initiatives in designing effective automated evaluators across various research directions @xcite @xcite . similarity-based evaluator. the evaluation of generated text can be measured by the n-gram overlap with the reference text, employing met-rics such as rouge @xcite , bleu @xcite , and meteor @xcite . in contrast to relying on exact matches, several evaluators have leveraged token similarity through contextual embeddings like bertscore @xcite , moverscore @xcite , and bartscore @xcite . however, these evaluators lack human correlation and a multi-dimensional assessment of text quality akin to real human evaluation, as they typically produce a single-dimensional score based on text similarity. to assess text quality, primarily focusing on checking factual consistency, task-specific evaluators utilizing nli and qa have been explored. ## evaluation datasets to evaluate the automated evaluator's performance, we need datasets with human annotations for sentence-level faithfulness and keyfactlevel alignment. since no single dataset includes both types of annotations, we opt for two separate datasets. frank @xcite is a benchmark dataset of 2, 246 summaries for factuality evaluation metrics. it encompasses summaries of nine summarization systems on cn-ndm @xcite and xsum @xcite , providing fine-grained annotations of sentence-level factuality error types. on the other hand, realsumm @xcite ) is another dataset of 2, 500 summaries from 25 summarization systems for automated metrics based on cnndm. it includes a list of human keyfacts, along with corresponding annotations indicating their presence in the summary. frank and real-summ obtain the inter-annotator agreement (iaa) scores of 0.58 (cohen's kappa) and 0.66 (krippendorff's alpha) for three annotators, respectively. llms as evaluators we use the gpt-4-turbo (gpt-4-1106-preview) @xcite by default in main evaluation, but test with various open-source and proprietary llms, including mixtral-8x7b (jiang et al., 2024), phi-2, llama-2/-3 @xcite , gpt-3.5-turbo, and gpt-4-omni (gpt-4o-2024-05-13), in section 4.2. we set the temperature to 0 and clear the history for every evaluation instance, following the literature @xcite . we use huggingface models for open-source llms and paid apis for proprietary llms. baselines we compare finesure with five similarity-based methods, rouge-1/-2/-l @xcite , bertscore @xcite , and bartscore @xcite ; a nli-based method, summac-conv @xcite ; two qa-based methods, unieval @xcite and @xcite ; and the latest llm-based method, g-eval @xcite . note that qafacteval and summac-conv are only compared for faithfulness evaluation, as they are limited to factuality. we obtain all the results by executing each metric in our experimental setup. performance each automated evaluator's performance is assessed by comparing estimated scores with ground-truth human judgments using sentence, summary, and system-level measurements. this multi-level analysis is crucial, as we seek to understand the agreement of the evaluator on each sentence, each summary, and the average performance of each summarization system. balanced accuracy (bacc) assesses faithfulness in classifying each summary sentence for the presence or absence of factual errors at the sentencelevel. this is the average of true positive and true negative rates widely used when the two classes are imbalanced @xcite . pearson and spearman correlations assess all three dimensions at the summary-level by comparing percentage scores in eqs. (1)-(2) derived from predicted and human evaluation results. lastly, rank correlation is a system-level measure assessing the alignment of performance rankings across summarization systems (models) calculated by both our evaluator and humans. the details of the measurements are provided in appendix d. ## conclusion we introduce finesure, a novel automated evaluator designed for fine-grained and multi-dimensional text summarization evaluation. the evaluation process is broken down into fact checking and keyfact alignment, providing detailed insights, where keyfacts can be either provided by humans or generated by llms. our experiments include a thorough comparison with existing evaluators, exploration of performance using eight opensource or proprietary llms, and real quality assessment of recent llm-generated summaries. the results indicate the potential effectiveness of finesure as a text summarization evaluator, showcasing its promising capabilities in advancing automated evaluation for text summarization.