Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation (2023)

Abstract: Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. In this work, we develop strong-performing automatic metrics for reference-based summarization evaluation, based on a two-stage evaluation pipeline that first extracts basic information units from one text sequence and then checks the extracted units in another sequence. The metrics we developed include two-stage metrics that can provide high interpretability at both the fine-grained unit level and summary level, and one-stage metrics that achieve a balance between efficiency and interpretability. We make the developed tools publicly available at https://github.com/Yale-LILY/AutoACU.

## introduction automatic evaluation is an integral part of scaling natural language generation (nlg) system development and evaluation. while neural models have seen great success in nlg systems, their adoption in automatic metric development has been much slower, and classic metrics such as rouge @xcite and bleu @xcite are still used more often than neural ones @xcite @xcite . compared to neural systems, we argue that there are unique requirements for neural metrics to be adopted: (1) interpretability -the metric scores should be interpretable and provide intuitive insights into system performance and system output quality. (2) evaluation efficiency -ideally, the automatic metrics should only introduce a small computation overhead, since it should be possible to use them on the fly for system development and fine-tuning. in this work, we aim to design neural metrics that are more aligned with these requirements, which we believe will facilitate adoption. our method focuses on a two-step decomposition of text sequence comparison following related work @xcite @xcite -first dissecting the information in one text sequence into multiple simple facts and then checking the presence of these facts in another text sequence. specifically, our automatic evaluation pipeline mirrors the human evaluation protocol of @xcite , which uses atomic content units, or acus, as the simple facts for comparing text sequences. we believe such a two-stage automatic evaluation can be more interpretable and transparent. specifically, at the acu level, the evaluation result indicates the presence or absence of an extracted information unit; at the summary level, the aggregated acu score represents the percentage of information overlap from one text sequence to another. in contrast, it can be difficult to interpret the results of certain neural metrics such as bertscore @xcite or bartscore @xcite . for example, bartscore assigns the normalized log-likelihood as the text similarity score, which is non-positive and non-linear, making it difficult to understand the system output quality based on its metric score. despite their advantages, such two-stage metrics @xcite @xcite can be much slower to run, even slower than the evaluated systems. therefore, apart from the two-stage evaluation method, we also propose a more efficient one-stage metric that directly predicts the aggregated summary-level scores by training on the recently proposed rose @xcite benchmark. the one-stage metric retains the summary-level interpretability as in the two-stage evaluation, striking a balance between efficiency and interpretability. we also explore using the twostage evaluation as pre-training for the one-stage metric, which further improves performance. our contributions can be summarised as: (1) a fine-grained two-stage automatic metric for reference-based summarization evaluation, which provides high interpretability; (2) an efficient one-stage automatic metric, which offers a balance between interpretability and evaluation efficiency; (3) both types of metrics we developed achieve stateof-the-art performance on summarization evaluation, and we make them publicly available and release them as an easy-to-use python package. foot_0 ## preliminaries information similarity for summarization evaluation the most common and important evaluation method of summarization systems is assessing the similarity between system-generated summaries and reference summaries. since there are no widely accepted definitions of such similarity among the related work @xcite @xcite @xcite , we refer it as the information similarity in this work -two completely similar text sequences should convey exactly the same information to the users, following the suggestion of @xcite . ## methods we first describe our approach of a two-stage decomposition of automatic information similarity evaluation -(1) extracting fine-grained content units from one text sequence; (2) checking the existence of the extracted units in another text sequence. then we introduce methods of training a one-stage automatic metric for information similarity and using the two-stage decomposition for pre-training. ## conclusions we develop high-performing reference-based summarization automatic metrics, including two-stage metrics providing fine-grained interpretability and one-stage metrics for a balance between efficiency and interpretability. furthermore, we show that the two-stage metric can be used to effectively pretrain the one-stage metric, helping to mitigate the data scarcity in developing automatic metrics.