UMSE : Unified Multi-scenario Summarization Evaluation (2023)

Abstract: Summarization quality evaluation is a non-trivial task in text summarization. Contemporary methods can be mainly categorized into two scenarios: (1) reference-based: evaluating with human-labeled reference summary; (2) reference-free: evaluating the summary consistency of the document. Recent studies mainly focus on one of these scenarios and explore training neural models built on PLMs to align with human criteria. However, the models from different scenarios are optimized individually, which may result in sub-optimal performance since they neglect the shared knowledge across different scenarios. Besides, designing individual models for each scenario caused inconvenience to the user. Inspired by this, we propose Unified Multi-scenario Summarization Evaluation Model (UMSE). More specifically, we propose a perturbed prefix tuning method to share cross-scenario knowledge between scenarios and use a self-supervised training paradigm to optimize the model without extra human labeling. Our UMSE is the first unified summarization evaluation framework engaged with the ability to be used in three evaluation scenarios. Experimental results across three typical scenarios on the benchmark dataset SummEval indicate that our UMSE can achieve comparable performance with several existing strong methods which are specifically designed for each scenario.

## introduction quantitatively evaluating the quality of generated summary is a non-trivial task that can measure the performance of the summarization system @xcite @xcite , and can also be used as a reward model to give an additional training signal * equal contribution. † corresponding author. 1 code is available at https://github.com/zt-yao/umse . for the summarization model @xcite @xcite @xcite . the dominant evaluation methods are traditional word-overlap-based metrics like rouge @xcite and bleu @xcite . although these metrics are very easy to use, they cannot evaluate semantic similarity. in recent years, many researchers focus on semantic-based evaluation tools @xcite @xcite . different to traditional metrics which only use one score to measure the quality of the summary, @xcite propose to evaluate the summary quality in several dimensions (e.g., coherence, consistency, and fluency) by calculating the similarity between the generated summary and the human-annotated summary. the summarization evaluation methods can be categorized into two scenarios based on the input data type: (1) reference-based methods require the human-annotated summary as input and (2) reference-free methods only use the corresponding document. the reference-based methods @xcite @xcite @xcite @xcite usually use the human written summary (a.k.a., reference summary) as the ground truth and calculate the similarity between generated and reference summary. with the help of the pre-train language model, these methods have a powerful ability to measure semantic similarity. however, not all realworld application scenarios have human-annotated summaries. using the reference-based evaluation method with the human-annotated ground truth summary is labor-consuming. thus, referencefree methods @xcite @xcite propose to evaluate the summary by modeling the semantic consistency between the generated summary and the document. when evaluating a summarization system, even though we can individually select a proper evaluator condition on whether we have a reference summary, it is not very convenient. moreover, since human annotation is costly, some summarization methods @xcite @xcite choose to use the automatic evaluator to provide an additional training signal, instead of relying entirely on human-labeled document-summary pair data. in this type of usage, the evaluator needs to measure the quality of the model-generated summary with partial humanlabeled document-summary data. besides, contemporary trainable evaluation models for different scenarios (with or without reference summary) are built on pre-train language models, which may transfer knowledge across different scenarios and provides a great opportunity to bridge these evaluation scenarios with a better combination of the best of both worlds. hence, it is valuable to build a unified multi-scenario summarization evaluator that can be used for processing both types of input data. intuitively, this naturally leads to two questions: (1) how to build a unified multi-scenario evaluation model regardless of whether we have a reference summary? (2) how to train the evaluator so that it can share knowledge between scenarios and maintain the exclusive knowledge in a specific task? in this paper, we propose a unified multiscenario summarization evaluation method unified multi-scenario summarization evaluation model (umse). umse unifies three typical summary quality evaluation scenarios in one model: (1) sum-ref: evaluate using reference summary. umse measures the similarity between the generated summary and the human-annotated reference summary. to sum up, our umse can bring the following benefits: to verify the effectiveness of the umse, we first compare with several baselines including the reference-based and reference-free methods. specifically, umse outperforms all the strong reference-free evaluation methods by a large margin and achieves comparable performance with the state-of-the-art in a unified model. ablation studies verify the effectiveness of our proposed perturbed prefix-tuning method. ## umse model problem formulation given a model-generated summary @xmath0, x 2 , . . . , x lx } with l x tokens, our goal is to use a unified evaluation model to produce a score @xmath1. for the sum-ref scenario, the model uses generated summary x and ground truth summary @xmath2, y 2 , . . . , y ly } as input. for the sum-doc scenario, we evaluate the summary quality by using generated summary x and document @xmath3, d 2 , . . . , d l d } with l d tokens as input, which does not require any human annotation (e.g., ground truth summary y ). for the sum-doc-ref scenario, the model uses generated summary x, ground truth summary y , and document d as input. to train the evaluation model, we do not use any human-annotated summary quality dataset and we construct the training dataset by using several self-supervised training strategies.