A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy (2021)

Abstract: In recent years, reference-based and supervised summarization evaluation metrics have been widely explored. However, collecting human-annotated references and ratings are costly and time-consuming. To avoid these limitations, we propose a training-free and reference-free summarization evaluation metric. Our metric consists of a centrality-weighted relevance score and a self-referenced redundancy score. The relevance score is computed between the pseudo reference built from the source document and the given summary, where the pseudo reference content is weighted by the sentence centrality to provide importance guidance. Besides an F_1 -based relevance score, we also design an F_\beta -based variant that pays more attention to the recall score. As for the redundancy score of the summary, we compute a self-masked similarity score with the summary itself to evaluate the redundant information in the summary. Finally, we combine the relevance and redundancy scores to produce the final evaluation score of the given summary. Extensive experiments show that our methods can significantly outperform existing methods on both multi-document and single-document summarization evaluation. The source code is released at https://github.com/Chen-Wang-CUHK/Training-Free-and-Ref-Free-Summ-Evaluation .

## preliminary notations. we denote vectors as bold lowercase characters and matrices as bold uppercase characters. the characters that are not bold are used to denote scalars. calligraphy uppercase characters are utilized to represent sets. problem definition. we formally define the reference-free summarization evaluation problem as follows. give a set of documents @xmath0. for multidocument summary (i.e., @xmath1, we compute relevance scores between the summary x and each document d k , and then average them as the final relevance score. ## our methodology the overall framework is illustrated in figure 1 . our final evaluation score of a summary consists of an averaged centrality-weighted relevance score and a self-referenced redundancy score. both scores are calculated on a semantic-level instead of utilizing n-gram overlapping. the averaged relevance score is computed from the relevance score between the summary and each document in the document set. the redundancy score is calculated based on the summary itself. ## conclusion in this paper, we propose a novel training-free and reference-free summarization evaluation metric consisting of a relevance score and a redundancy score. experiments on multi-document and single-document summarization settings show the effectiveness of our methods. one promising future direction is to solve the performance dropping issue after applying our key features into mover-score and the other is to tackle the problem that current metrics struggle to assign higher scores for good summaries.