Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning (2020)

Abstract: Evaluation of a document summarization system has been a critical factor to impact the success of the summarization task. Previous approaches, such as ROUGE, mainly consider the informativeness of the assessed summary and require human-generated references for each test summary. In this work, we propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. To learn the metric, for each summary, we construct different types of negative samples with respect to different aspects of the summary qualities, and train our model with a ranking loss. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries. Furthermore, we show that our method is general and transferable across datasets.

## introduction recently, there has been great success in automatic text summarization and generation @xcite @xcite . to better compare and improve the performance of models, evaluation for such systems has been a problem of interest. the selection of evaluation metrics will greatly affect the assessed quality of a generated summary and thus affect the evaluation of summarization models. the most ideal metric is definitely human judgement, which is often treated as the gold standard. but human evaluation is time-consuming and laborintensive, an automatic evaluation metric that cannot only save human resources but also simulate the ability of human judgement is of crucial importance. ## method as shown in the previous section, two of the most important factors that impact the summary qualities are linguistic quality and semantic quality. linguistic quality indicates how natural the generated summary is; it generally includes the fluency of each sentence, the coherence of entities/consecutive sentences, and the correctness of grammars. semantic quality indicates whether a summary expresses the most important information of the original documents; it generally includes informativeness, relevance, and redundancy, etc. we consider both aspects and design our method in the following sections. our model architecture is shown in figure 1 . the figure contains two parts, first we design our evaluator to assign scores to summaries based on a bert encoder. then we create negative samples and use a contrastive learning framework to train the evaluator. ## experiments we conduct our experiments to answer the following questions: ## conclusion in this paper, we propose a new evaluation method in the field of text summarization. we found that the quality of a summary can be evaluated in two separate dimensions: semantic quality and linguistic quality. since human-authored references used in most of the existing metrics are costly, we investigate automatic evaluation metrics in an unsupervised reference-free setting. leveraging powerful representations of bert, our methods achieve the highest performance on two datasets. although our experiments are only on single-document summarization datasets, our method can also be also extended to evaluation of multi-document summarization with slight changes, especially in the part of semantic quality evaluation.