S ue N es: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling (2022)

Abstract: Canonical automatic summary evaluation metrics, such as ROUGE, focus on lexical similarity which cannot well capture semantics nor linguistic quality and require a reference summary which is costly to obtain. Recently, there have been a growing number of efforts to alleviate either or both of the two drawbacks. In this paper, we present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries. In cross-domain tests, our strategy outperforms baselines with promising improvements, and show a great advantage in gauging linguistic qualities over all metrics.

## introduction in natural language processing, the problem of summarization studies generating a summary from a source document which is longer than the summary. de facto metrics to judge a generated summary include rouge @xcite , bleu @xcite , and meteor @xcite . previous work @xcite @xcite agrees on two major drawbacks of them: 1) they favor lexical similarity, falling short on semantic similarity or linguistic quality, and 2) they require a reference summary which is often expensive to obtain @xcite . initially, the first drawback is partially alleviated by replacing lexicons with their word embeddings @xcite @xcite . after the birth of transformers @xcite , this effort has expanded to sentence or document level, including reference-based @xcite , and reference-free ones @xcite @xcite . the main difference between the two groups is whether a reference summary is needed when evaluating a machine-generated summary. the two groups have complementary pros and cons. reference-based metrics have a better performance, but they are impractical when summarization is used industrially, such as in customer support @xcite , team conversation @xcite , and bug reporting @xcite , where it is too costly to manually craft an equally massive amount of reference summaries. in contrast, without human written reference summaries, reference-free approaches generally perform poorer. modern transformer-based referencefree approaches often rely on non-summarization tasks, such as qa @xcite . such fact-focused approach makes them excel on content/fact aspects (still worse than reference-based ones) but not on linguistic ones. the non-summarization tasks also introduce noises. therefore, in this paper, as a proof of concept, we explore a hybrid or middle approach to combine the best of both worlds. using documentsummary pairs in existing summarization datasets, our weakly supervised approach mutates * reference summaries and pair them with documents to form training data and then use the trained model to evaluate unseen summaries in the presence of 1 2450 documents without corresponding reference summaries. in this way, we make use of human written summaries, which are very precious, in training, but we do not need them in summary evaluation. we call our approach suenes , which stands for "summary evaluation by negative sampling." the quality of a summary is usually evaluated on two facets: content/fact aspects and linguistic qualities. experiments later will show that a value of our approach is that we can use the same model architecture to build models that excel on different tasks by feeding training data from the same source but mutated in different strategies. for example, deleting words is the best for linguistic qualities while deleting sentences is the best for content/fact coverage. our approach is empirically compared against an array of existing metrics on three human summary evaluation datasets. despite being training-based, our approach exhibits consistent results across various training domains which are all different from the test domain. it outperforms reference-free baselines with promising improvements on content/fact aspects, and further outperforms all existing metrics in gauging linguistic qualities. in summary, our contributions or merits are: we hope our study can inspire more research into hybridizing reference-free and reference-based summary evaluation. our code is at https:// github.com/forrestbao/suenes/ 2 the approach 2.1 model architecture a reference-free single-document summary quality assessor can be formulated as a regression function f (d, s) ∈ [0, 1] of an input document @xmath0, t 2 , • • •], and a machine-generated summary @xmath1, t ′ 2 , • • •], where t i 's and t ′ i 's are text tokens. as a proof of concept, we explore an extremely lean † the definition of "reference-free" is that reference summaries are not needed in the evaluation stage. the function g can be implemented in the bert @xcite style with an input sequence [[cls], t 1 , t 2 , • • •, [sep], t ′ 1 , t ′ 2 , • • •, [sep]]. the output on the [cls] token is a joint representation of both the document d and the summary s. while the human evaluation to a summary may cover multiple aspects, such as content/fact coverage and linguistics, a model of us will only yield one number. but by using different data mutation strategies, we can get models (different f 's) adept at different aspects of a summary. ## conclusion in this paper, we propose a weakly supervised approach to summary quality evaluation. a few mutation methods are introduced to make use of the massive, precious human written summaries in summarization datasets. in cross-domain experiments, our approach achieves better performance than baselines, especially on linguistic aspects. we hope this proof-of-concept study can inspire more reference-free summary evaluation. we assign the label 0 to a mismatching documentsummary pair, and the label 1 to any original pair of a document and its reference summary.