P ref S core: Pairwise Preference Learning for Reference-free Summarization Quality Assessment (2022)

Abstract: Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. Inspired by preference labeling in existing work of summarization evaluation, we propose to judge summary quality by learning the preference rank of summaries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experiments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings.

## introduction summarization is a task in natural language processing in which automatic systems generate summaries from documents. to judge the quality of system-generated summaries, human evaluation is the best option, but it is non-trivial and laborious. hence, many automatic metrics have been developed. they can be categorized as referencebased ones and reference-free ones, depending on whether reference summaries are needed in the evaluation stage. reference-based metrics include rouge @xcite , bleu @xcite , cider @xcite , meteor @xcite , s 3 @xcite , moverscore @xcite , bertscore @xcite , etc. calculating the lexical overlap or the embedding similarity between a system-generated summary and its corresponding human-written reference summary, they reportedly have high correlations with human assessments. because creating human-written reference summaries is laborious and expensive, recent works are shifting to reference-free metrics. sum-maqa @xcite and blanc @xcite leverage pretrained language models to carry out text understanding tasks to evaluate the helpfulness of a summary for understanding its source document. supert @xcite measures the semantic similarity against a pseudo reference summary in a multi-document summarization setting. however, reference-free metrics may show a lower correlation @xcite with human evaluation scores than some of the reference-based metrics. to trade off between the human effort needed and the quality of the evaluation, some work pursues a pairwise preference approach which collects preference labels over sentences in documents or over summaries from a human assessor as it requires less cognitive effort than writing a reference summary or manually scoring a machine-generated summary. @xcite proposes a reference-free evaluation approach by estimating sentence-level preferences on source documents rather than directly on the generated summaries. @xcite train a linear model to estimate a summary preference utility function via active preference learning to guide a reinforcement learning based summarization system. but they do not examine the learned preference model as a metric for summarization evaluation. inspired by human-involved pairwise preference in summarization evaluation @xcite and simple nlp data augmentation methods like eda @xcite , in this work, we explore reference-free summary quality assessment via pairwise preference learning using negative sampling. a pre-trained text embedding model is used in a siamese network to learn the preference utility in an end-to-end, weakly supervised fashion. the closest work to ours is ls_score @xcite . we achieve improved performance by using a better-attended model, a loss function based on preference learning, and introducing a mixed transitive negative sampling strategy. in addition, we promote our work to cross-domain and multi-document settings. we show that the learned models are competitive compared to the state-of-the-art reference-free metrics. our code is at https://github.com/ nkwbtb/prefscore .