D oc A s R ef: An Empirical Study on Repurposing Reference-based Summary Quality Metrics as Reference-free Metrics (2023)

Abstract: Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of < 0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.

## introduction summarization is an important natural language generation (nlg) task. a problem that goes hand in hand with it is summary evaluation, which quantifies the quality of a summarizer or a system summary it generates. the traditional approach to automated † summary quality assessment is reference-based, such as rouge @xcite , bertscore @xcite and mover-score @xcite , which assesses a system summary against one or a plurality of humanwritten reference summaries. † the ground truth is still human evaluation. requiring highly educated human labor, reference summaries are very costly to obtain. therefore, many reference-free metrics have emerged recently @xcite @xcite , which directly compute a score between a system summary and its source document. however, the performance of referencefree metrics has historically lagged behind that of reference-based metrics because a human-written reference summary serves as a fluent and comprehensive representation of the key facts in the input document and thus gives reference-based metrics an advantage. recently, large language models (llms) have shown promise in building reference-free summary quality metrics. metrics based on llms like gpt-3.5/4 @xcite @xcite have outperformed both reference-free and reference-based baselines. however, llms are computationally expensive, and the closed nature of gpt-3+ restricts their usage with legal and reproducibility ‡ limitations. a more viable solution that uses much more cost-effective language models is highly expected. to build an accurate but efficient metric, we revisit the reference-based metrics and hypothesize that they can be repurposed into referencefree metrics by directly comparing a summary with its source document. after being repurposed, bertscore outperforms not only its original reference-based version, but also most existing reference-free metrics across the summeval, newsroom, and tac2010 datasets on both semantic and linguistic aspects. notably, the repurposed bertscore achieves superior or comparable per- ‡ https://hackingsemantics.xyz/2023/ closed-baselines/ 1 1226 formance to gpt-3.5-based summarization evaluators. it is worth noting that these results are achieved using foundation models with significantly fewer parameters (<0.5b) compared to gpt-3.5's extensive 175 billion parameters. we hope this paper can inspire more work into zero-shot summarization or nlg evaluation using cost-effective (e.g., <1b parameters) lms. our source code is at https://github.com/ sigmawe/docasref . in summary, the key findings of this paper include: ## conclusion in this paper, we explore repurposing summary evaluation metrics that were originally designed or trained for reference-based use as reference-free metrics. the motivation was to reuse their power in comparing texts. comprehensive experiments on multiple datasets show that four representative metrics generally perform better after the repurposing. the best among them, bertscore, is further studied with different configurations. the repurposed bertscore using 0.5b-parameter lms can outperform all non-gpt baselines significantly and even most of the times those based on gpt3.5. work. forrest bao also wants to dedicate this paper to the people of ukraine who have been courageously fighting for freedom since february 24, 2022.