SUPERT : Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020)

Abstract: We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the quality of a summary by measuring its semantic similarity with a pseudo reference summary, i.e. selected salient sentences from the source documents, using contextualized embeddings and soft token alignment techniques. Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18- 39%. Furthermore, we use SUPERT as rewards to guide a neural-based reinforcement learning summarizer, yielding favorable performance compared to the state-of-the-art unsupervised summarizers. All source code is available at https://github.com/yg211/acl20-ref-free-eval .

## introduction evaluating the quality of machine-generated summaries is a highly laborious and hence expensive task. most existing evaluation methods require certain forms of human involvement, thus are supervised: they either directly let humans rate the generated summaries (e.g. pyramid @xcite ), elicit human-written reference summaries and measure their overlap with the generated summaries (e.g. using rogue @xcite or moverscore @xcite ), or collect some human annotations (e.g. preferences over pairs of summaries @xcite ) to learn a summary evaluation function. evaluation in multidocument summarization is particularly expensive: @xcite reports that it requires 3,000 hours of human effort to evaluate the summaries from the document understanding conferences (duc) 1 . contributions. first, to better measure the semantic overlap between source documents and machine-generated summaries, we propose to use state-of-the-art contextualized text encoders, e.g. bert @xcite and its variant sentence-bert (sbert) @xcite , which is optimized for measuring semantic similarity between sentences, to develop unsupervised evaluation methods. we measure the relevance of a summary in two steps: (i) identifying the salient information in the input documents, to build a pseudo reference summary, and (ii) measuring the semantic overlap between the pseudo reference and the summary to be evaluated. the resulting evaluation method is called supert (summarization evaluation with pseudo references and bert). fig. 1 illustrates the major steps of supert. we show that compared to state-of-the-art unsupervised metrics, the best supert correlates better with the human ratings by 18-39% (in kendall's τ ). second, we use supert as reward functions to guide reinforcement learning (rl) based extractive summarizers. we show it outperforms the state-of-the-art unsupervised summarization methods (in multiple rouge metrics). ## related work reference-based evaluation. popular metrics like rouge @xcite , bleu @xcite and meteor @xcite fall into this category. they require (preferably, multiple) human written references and measure the relevance of a summary by comparing its overlapping word sequences with references. more recent work extends rouge with wordnet @xcite , word embeddings @xcite , or use contextualizedembedding-based methods @xcite to measure the semantic similarity between references and summaries. annotation-based evaluation. some methods directly ask human annotators to rate summaries following some guidelines, e.g. responsiveness, which measures the overall quality (relevance, fluency and readability) of summaries, and pyramid @xcite , which measures summaries' relevance. recently, systems have been developed to ease the construction of pyramid scores, e.g. @xcite @xcite , but they still require human-annotated summary content units (scus) to produce reliable scores. besides scus, recent work has explored eliciting preferences over summaries @xcite @xcite and annotations of important bi-grams (p.v.s and meyer, 2017) to derive summary ratings. some methods collect human ratings on a small number of summaries to train an evaluation function. @xcite ; @xcite propose to learn an evaluation function from pyramid and responsiveness scores, by using classic supervised learning methods with hand-crafted features. @xcite use the same idea but design corpus based and lexical re-source based word embeddings to build the features. @xcite train a bert-based evaluation function with 2,500 human ratings for 500 machinegenerated summaries from the cnn/dailymail dataset; their method correlates better with human ratings than rouge and bleu. however, as their method is designed for evaluating single-document summaries, it correlates poorly with the pyramid scores for multi-document summaries (see §3). unsupervised evaluation. @xcite measure the relevance of a summary using multiple heuristics, for example by computing the jensen-shannon (js) divergence between the word distributions in the summary and in the source documents. @xcite ; @xcite develop evaluation heuristics inspired by the maximal marginal relevance metrics @xcite . but these methods have low correlation with human ratings at summary level (see §3). @xcite propose to generate questions from source documents and evaluate the relevance of summaries by counting how many questions the summaries can answer. however, they do not detail how to generate questions from source documents; also, it remains unclear whether their method works for evaluating multi-document summaries. @xcite propose a single-document summary evaluation method, which measures the cosine similarity of the elmo embeddings @xcite of the source document and the summary. in §3, we show that their method performs poorly in evaluating multi-document summaries. supert extends their method by using more advanced contextualized embeddings and more effective text alignment/matching methods ( §4), and by introducing pseudo references ( §5). ## datasets, baselines and upper bounds datasets. we use two multi-document summarization datasets from the text analysis conference (tac) foot_0 shared tasks: tac'08 and tac'09. in line with @xcite , we only use the initial summaries (the a part) in these datasets. tac'08 includes 48 topics and tac'09 includes 44. each topic has ten news articles, four reference summaries and 57 (tac'08) and 55 (tac'09) machine-generated summaries. each news article on average has 611 words in 24 sentences. each summary has at most 100 words and receives a tac'08 tac'09 r ρ τ r ρ τ baselines (unsupervised evaluation) tf-idf .364 .330 .236 .388 .395 .288 js .381 .333 .238 .388 .386 .283 reaper .259 .247 .174 .332 .354 .252 celmo .139 .108 .076 .334 .255 .183 böhm19 .022 -.001 .001 .075 .043 .031 upper bounds (reference-based evaluation) rouge1 .747 .632 .501 .808 .692 .533 rouge2 .718 .635 .498 .803 .694 .531 mover .760 .672 .507 .831 .701 .550 pyramid score, which is used as the ground-truth human rating in our experiments. baselines & upper bounds. for baselines, we consider tf-idf, which computes the cosine similarity of the tf-idf vectors of source and summaries; js, which computes the js divergence between the words distributions in source documents and summaries; and the reaper heuristics proposed by @xcite . in addition, we use the learned metric from @xcite (böhm19) and the elmo-based metric by @xcite (c elmo , stands for cosine-elmo; see §2). in all these methods, we remove stop-words and use the stemmed words, as we find these operations improve the performance. for c elmo , we vectorize the documents/summaries by averaging their sentences' elmo embeddings. as for upper bounds, we consider three strong reference-based evaluation metrics: rouge-1/2 and moverscore @xcite ; note that references are not available for unsupervised evaluation metrics. we measure the performance of the baselines and upper bounds by their average summary-level correlation with pyramid, in terms of pearson's (r), spearman's (ρ) and kendall's (τ ) correlation coefficients. 3 tac'08 tac'09 r ρ τ r ρ τ cbert .035 .066 .048 .130 .099 .071 croberta .100 .126 .091 .262 .233 .165 calbert .152 .122 .086 .247 .219 .157 csbert .304 .269 .191 .371 .319 .229 mroberta .366 .326 .235 .357 .316 .229 msbert .466 .428 .311 .436 .435 .320 ## measuring similarity with contextualized embeddings in this section, we explore the use of more advanced contextualized embeddings and more sophisticated embedding alignment/matching methods (rather than cosine similarity) to measure summaries relevance. we first extend c elmo by considering more contextualized text encoders: bert, roberta @xcite , albert @xcite and sbert foot_1 . we use these encoders to produce embeddings for each sentence in the documents/summaries, and perform average pooling to obtain the vector representations for the documents/summaries. we measure the relevance of a summary by computing the cosine similarity between its embedding and the embedding of the source documents. the upper part in tac'08 tac'09 r ρ τ r ρ τ random3 .139 .194 .189 .123 .172 .175 random5 .144 .203 .199 .147 .204 .206 random10 .163 .228 .229 .201 .279 .284 random15 .206 .287 .320 .185 .258 .268 top3 .449 .408 .295 .378 .390 .291 top5 .477 .437 .316 .413 .421 .314 top10 .492 .455 .332 .444 .450 .333 top15 .489 .450 .327 .454 .459 .340 the wmd-based scores substantially outperform their cosine-embedding counterparts; in particular, m sbert outperforms all lexical-based baselines in ## guiding reinforcement learning we explore the use of different rewards to guide neural temporal difference (ntd), a rl-based multi-document summarizer @xcite . we consider three unsupervised reward functions: two baseline methods reaper and js (see §3 and tac'08 tac'09 r1 r2 rl r1 r2 rl ntdrp .348 .087 .276 .360 .090 .187 ntdjs .353 .090 .281 .368 .095 .192 ntdsp .376 * .102 * .296 * .380 * .103 * .194 yls15 .375 * .096 n/a .344 .088 n/a in addition, we consider a non-rl-based stateof-the-art unsupervised summarizer proposed by @xcite . we use rouge to measure the quality of the generated summaries and leave human evaluations for future work. ## conclusion we explored unsupervised multi-document summary evaluation methods, which require neither reference summaries nor human annotations. we find that vectorizing the summary and the top sentences in the source documents using contextualized embeddings, and measuring their semantic overlap with soft token alignment techniques is a simple yet effective method to rate the summary's quality. the resulting method, supert, correlates with human ratings substantially better than the state-of-the-art unsupervised metrics. furthermore, we use supert as rewards to train a neural-rl-based summarizer, which leads to up to 17% quality improvement (in rouge-2) compared to the state-of-the-art unsupervised summarizers. this result not only shows the effectiveness of supert in a downstream task, but also promises a new way to train rl-based summarizers: an infinite number of summary-reward pairs can be created from infintely many documents, and their supert scores can be used as rewards to train rl-based summarizers, fundamentally relieving the data-hungriness problem faced by existing rl-based summarization systems.