Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio (2022)

Abstract: A document can be summarized in a number of ways. Reference-based evaluation of summarization has been criticized for its inflexibility. The more sufficient the number of abstracts, the more accurate the evaluation results. However, it is difficult to collect sufficient reference summaries. In this paper, we propose a new automatic reference-free evaluation metric that compares semantic distribution between source document and summary by pretrained language models and considers summary compression ratio. The experiments show that this metric is more consistent with human evaluation in terms of coherence, consistency, relevance and fluency.

## introduction summarization evaluation metrics that measure the quality of generated summaries are very important for the development of summarization systems @xcite @xcite @xcite @xcite . most previous summarization evaluation metrics need human-annotated summaries as reference and measure summary quality through the similarity between generated summaries and their reference summaries @xcite @xcite @xcite . such reference-based evaluation metrics cannot accurately evaluate the summary, because a document has many correct but different summaries. it is difficult and expensive to write many reference summaries by human for evaluation. thus, it is useful to develop reference-free evaluation metrics for this task. in this paper, we focus on reference-free evaluation metrics. as shown in figure 1 , a highquality summary should be concise and contain the most important information of its document. some reference-free evaluation metrics @xcite unsupervisedly construct a pseudo ## approach in this section, we introduce our proposed reference-free summarization evaluation metric which computes the semantic distribution correlation between generated documents with and without a summary and combines the correlation with compression ratio. ## experiment in this section, we first introduce the humanannotated datasets and baseline evaluation metrics. then we will show the results of our proposed sdc and sdc*, and analyze the effectiveness of semantic distribution correlation and compression ratio used in summarization evaluation.