S umm EQ u AL : Summarization Evaluation via Question Answering using Large Language Models (2024)

Abstract: Summarization is hard to evaluate due to its diverse and abstract nature. Although N-gram-based metrics like BLEU and ROUGE are prevalent, they often do not align well with human evaluations. While model-based alternatives such as BERTScore improve, they typically require extensive labelled data. The advent of Large Language Models (LLMs) presents a promising avenue for evaluation. To this end, we introduce SummEQuAL, a novel content-based framework using LLMs for unified, reproducible summarization evaluation. SummEQuAL evaluates summaries by comparing their content with the source document, employing a question-answering approach to gauge both recall and precision. To validate SummEQuAL’s effectiveness, we develop a dataset based on MultiWOZ. We conduct experiments on SummEval and our MultiWOZ-based dataset, showing that SummEQuAL largely improves the quality of summarization evaluation. Notably, SummEQuAL demonstrates a 19.7% improvement over QuestEval in terms of sample-level Pearson correlation with human assessments of consistency on the SummEval dataset. Furthermore, it exceeds the performance of the BERTScore baseline by achieving a 17.3% increase in Spearman correlation on our MultiWOZ-based dataset. Our study illuminates the potential of LLMs for a unified evaluation framework, setting a new paradigm for future summarization evaluation.

## introduction summary evaluation remains a complex task, and to this day, it cannot be adequately accomplished by automatic metrics @xcite . while n-gram-based metrics like bleu @xcite and rouge @xcite are widely used, they often show a poor correlation with human judgment, particularly in content assessment @xcite . in addition, these methods rely on references as a "gold standard", which diminishes their effectiveness especially when assessing varied and abstract summaries due to the limited availability of reference texts. summary evaluation has higher requirements for diversity and accuracy. the rise of large language models (llms) offers a promising way to evaluate generative texts more effectively, due to their understanding and reasoning abilities. while previous studies have been conducted in various contexts, such as machine translation @xcite and summarization @xcite @xcite (without relying on reference summaries), the use of llms for evaluation has its limitations. specifically, there are large variation across different prompts and llms, which complicates the use of a unified evaluation framework. while the question-answering (qa) approach offers a structured method for evaluation to mitigate this issue, existing qa approaches still have several limitations @xcite @xcite : (1) previous works are more confined to direct answers, and answers based on reasoning or hidden information are often tricky; (2) models in generating questions lack focus so they may introduce irrelevant information; and (3) models need expensive pre-training and the performance will be influenced by the coverage and quality of training data. to address these issues, we propose a novel unified framework for summarization evaluation sum-mequal, which can effectively identify abstract information across a wide range of topics and perform complex inference. specifically, summe-qual streamlines the evaluation process by breaking it down into separate tasks and conducts qa to approximate human-like evaluation. it guides llms with a structured schema to identify key information and employs a qa mechanism to complete the content evaluation, producing more reliable results. our experimental results on the summeval dataset and multiwoz-based dataset demonstrate that the qa process benefits from llms. particularly remarkable is that summe-qual improves the spearman correlation with human evaluation by 19.7% on summeval consistency, compared with our baseline questeval. in summary, the main contributions of this paper are as follows: ## summequal framework we propose a qa-based framework utilizing llms to evaluate summarization systems without needing human references for each summary. sum-mequal incorporates both recall-oriented and precision-oriented approaches, featuring three submetrics, as illustrated in figure 1 . below, we provide detailed explanations for each metric in our framework. ## experiments and results evaluation strategy. we evaluate automatic metrics by comparing their alignment with reference human evaluations. three prevalent correlations, the spearman correlation, the person correlation and the kendall's tau correlation are employed. given n source texts and m summary models, the i-th text's summary generated by the j-th model is denoted as s i,j . the formula of correlation at a sample level is as follows: ## conclusion this work proposes a novel summarization evaluation framework, summequal, based on llms. the summequal framework provides a reliable and effective approach for the evaluation of summarization, opening up a new direction for future work on unified and reproducible summarization evaluation using llms. summequal sheds new light on developing reliable and consistent summarization evaluation methods, expected to help researchers more precisely understand and evaluate the performance of summarization models, thereby improving the quality of summarization content.