U ni S um E val: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLM s (2024)

Abstract: Existing benchmarks for summarization quality evaluation often lack diverse input scenarios, focus on narrowly defined dimensions (e.g., faithfulness), and struggle with subjective and coarse-grained annotation schemes. To address these shortcomings, we create UniSumEval benchmark, which extends the range of input context (e.g., domain, length) and provides fine-grained, multi-dimensional annotations. We use AI assistance in data creation, identifying potentially hallucinogenic input texts, and also helping human annotators reduce the difficulty of fine-grained annotation tasks. With UniSumEval, we benchmark nine latest language models as summarizers, offering insights into their performance across varying input contexts and evaluation dimensions. Furthermore, we conduct a thorough comparison of SOTA automated summary evaluators. Our benchmark data will be available at https://github.com/DISL-Lab/UniSumEval-v1.0.

## introduction despite the enhanced quality of text summarization by large language models (llms), they still face persistent challenges like hallucination, information omission, and verbosity @xcite . this multifaceted nature of text summaries inevitably demands manual evaluation by human experts, a labor-intensive and costly process. to streamline this evaluation process, recent efforts aim to design human-like automatic evaluators, such as g-eval @xcite and finesure @xcite , which achieve a satisfactory correlation with human judgments. such evaluators are typically validated by examining their consistency with human judgments on established benchmark datasets, such as frank @xcite and tofueval @xcite ). yet, these benchmark datasets have limitations in terms of input diversity, granularity of human annotations, and evaluation dimensions. firstly, most existing benchmarks are restricted solely to a single domain. the predominant focus is often on the news domain such as summeval @xcite and aggrefact @xcite . this deficiency constrains the accurate evaluation of automated evaluators by failing to capture diverse input contexts across various domains. secondly, there is a lack of datasets that consider varying input types and lengths simultaneously. while these two factors have a significant impact on the summary quality, existing datasets are often limited to short, non-dialogue texts @xcite @xcite . without considering these factors, they cannot adequately assess distinct perspectives across different input types and lengths. this includes correctly attributing statements to speakers in dialogue, preventing false information in articles with personally identifiable information (pii) redacted, and pinpointing key information in long texts. thirdly, no comprehensive datasets exist for finegrained, multi-dimensional summarization evaluation. some benchmarks offer fine-grained annotations, such as fact verification at the sentencelevel @xcite @xcite and alignment at the key-fact foot_0 level @xcite ), yet they suffer from either a limited evaluation dimension or coarse-grained human labels. in this paper, we create unisumeval in figure 1 , the first one-size-fits-all benchmark for fine-grained, multi-dimensional evaluation of automated evaluators. it includes: text inputs encompassing nine distinct domains (e.g., news, report, booking, meeting) that span from non-dialogue to dialogue, short texts to long texts containing up to 10,462 words, and even with redacted pii to simulate real-world scenarios; summaries generated from nine latest summarizers across three categories, namely non-llms, open-source and proprietary llms; evaluation dimension covering three distinct evaluation aspects -assessing faithfulness, information omission (completeness), and verbosity (conciseness) of the generated summaries; ai-assisted manual evaluation collecting fine-grained, multi-dimensional human annotations with high iaa foot_2 -fact verification at the sentence level for faithfulness, key-fact validation and alignment of validated key-facts to each summary sentence for completeness and conciseness. unlike existing benchmarks, we identify hallucinogenic texts, which can potentially trigger hallucination even for the latest llms. we then add these texts into our dataset, ensuring that our benchmark includes challenging text-summary pairs where the latest models actually generate and incur hallucinations. also, the wide-ranging text diversity within unisumeval enables a comprehensive evaluation of modern automated evaluators. this helps ascertain if the evaluators perform consistently across diverse input scenarios, an aspect overlooked in prior works. using unisumeval's fine-grained human labels, we further benchmark nine latest language models on summarization across multi-dimensional aspects. we group them into three distinct cate-gories: non-llms, open-source and proprietary llms. we then compare their performance across five key evaluation dimensions, including faithfulness, completeness, conciseness, abstractiveness, and domain stability. next, we unveil the current progress of sota automated evaluators, including qa-, nli-, and llm-based methods, by comparing their evaluation scores with human judgments in unisumeval. our main contributions are as follows: ## unisumeval pipeline our data creation pipeline consists of four consecutive steps in the following sections. ## unisumeval quality assessment evaluation metric. we report iaa for three finegrained annotation tasks in unisumeval: fact verification, key-fact validation and alignment. we use @xcite by default, but for key-fact validation, where there is significant label imbalance, we use gwet's ac1 @xcite due to its enhanced robustness. ## benchmarking auto-evaluators evaluator selection. we benchmark sota automated evaluators on unisumeval. the set of compared evaluators varies according to the target evaluation dimension. for faithfulness, we include qa-based models: unieval @xcite and @xcite , nli-based models: summac-conv @xcite , align-score @xcite , and minicheck @xcite , and llm-based models at various levels of granularity: g-eval @xcite for summary level, g-eval+ foot_4 for sentence level and factscore @xcite for unieval and g-eval, we use their relevance scores since conciseness dimension is not directly supported. @xcite , along with unieval and g-eval+. for conciseness, we include the models supporting the evaluation of relevance, such as unieval and g-eval, and our g-eval+ tailored for conciseness. we use gpt-4-turbo for all llmbased models. evaluation metric. following prior works @xcite , we compare the estimated scores with the ground-truth human scores in unisumeval. we report their pearson correlation based on the summary-level percentage score of each evaluation dimension. see appendix d.2 for the details of measurements and appendix f.4 more results using system-level measurement. ## conclusion we introduce unisumeval, a benchmark dataset featuring hallucinogenic texts from nine domains, spanning non-dialogue to dialogue and short to long texts, paired with their summaries generated by nine recent summarizers. built using a data creation pipeline with ai assistance, unisumeval includes high-quality, fine-grained human annotations that enable in-depth studies on the multidimensional performance of summarizers. additionally, based on our benchmark, we provide a thorough assessment of automated evaluators for text summarization, revealing weaknesses related to specific domains and evaluation dimensions.