Facet-Aware Evaluation for Extractive Summarization (2020)

Abstract: Commonly adopted metrics for extractive summarization focus on lexical overlap at the token level. In this paper, we present a facet-aware evaluation setup for better assessment of the information coverage in extracted summaries. Specifically, we treat each sentence in the reference summary as a facet , identify the sentences in the document that express the semantics of each facet as support sentences of the facet, and automatically evaluate extractive summarization methods by comparing the indices of extracted sentences and support sentences of all the facets in the reference summary. To facilitate this new evaluation setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evaluation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods. Data can be found at https://github.com/morningmoni/FAR .

## introduction text summarization has enjoyed increasing popularity due to its wide applications, whereas the evaluation of text summarization remains challenging and controversial. the most commonly used evaluation metric of summarization is lexical overlap, i.e., rouge @xcite , which regards the system and reference summaries as sequences of tokens and measures their n-gram overlap. however, recent studies @xcite @xcite reveal the limitations of rouge and find that in many cases, it fails to reach consensus with human judgment. since lexical overlap only captures information in this paper, we argue that the information coverage in summarization can be better evaluated by facet overlap, i.e., whether the system summary covers the facets in the reference summary. specifically, we treat each reference sentence as a facet, identify document sentences that express the semantics of each facet as support sentences of the facet, and measure information coverage by facet-aware recall (far), i.e., how many facets are covered. we focus on extractive summarization for the following two reasons. theoretically, since extractive methods cannot paraphrase or compress the document sentences as abstractive methods, it is somewhat unfair to penalize them for extracting long sentences that cover the facets. pragmatically, we can evaluate extractive methods automatically by comparing the indices of extracted sentences and support sentences. we denote the mappings from each facet (sentence) in the reference summary to its support sentences in the document as facet-aware mappings (fams). fams can be used as labels indicating which sentences should be extracted but they are grouped with respect to each facet, while conventional extractive labels correspond to the entire reference summary rather than individual facets (detailed explanations in sec. 2.1). compared to treating one summary as a sequence of n-grams, facet-aware evaluation considers information coverage at a semantically richer granularity, and thus can contribute to a more accurate assessment on the summary quality. to verify the effectiveness of facet-aware evaluation, we construct an extractive version of the cnn/daily mail dataset @xcite by annotating its fams (sec. 2). we revisit state-ofthe-art extractive methods using this new extractive dataset (sec. 3.2), the results of which show that far correlates better with human evaluation than rouge. we also demonstrate that fams are beneficial for fine-grained evaluation of both abstractive and extractive methods (sec. 3.3). we then illustrate how facet-aware evaluation can be useful for comparing different extractive methods in terms of their capability of extracting salient and non-redundant sentences (sec. 3.4). finally, we explore the feasibility of automatic fam creation by evaluating sentence regression approaches against the ground-truth annotations (i.e., fams), and generalize facet-aware evaluation to the entire cnn/daily mail dataset without any human annotation (sec. 4). we believe that the summarization community will benefit from the proposed setup for better assessment of information coverage and gain deeper understandings of the current benchmark dataset and state-of-the-art methods through our analysis. acteristics of the cnn/daily mail dataset. ## dataset creation in this section, we describe the process of creating an extractive summarization dataset to facilitate facet-aware evaluation, which involves annotating fams between the documents and abstractive reference summaries. we first formalize the fams and then describe the fam annotation on the cnn/daily mail dataset @xcite . ## facet-aware evaluation in this section, we introduce the facet-aware evaluation setup (sec. 3.1) and demonstrate its effectiveness by revisiting state-of-the-art summarization methods under this new setup (sec. 3.2). we then illustrate the additional benefits of facet-aware evaluation, including fine-grained evaluation (sec. 3.3) and comparative analysis (sec. 3.4). ## evaluation without human annotation in the previous sections, we have demonstrated the effectiveness and benefits of facet-aware evaluation. one remaining issue that might prevent facet-aware evaluation from scaling is the need of human-annotated fams. we thus study the feasibility of automatic fam creation with sentence regression and present a pilot study of conducting facet-aware evaluation without any human annotation in this section. ## conclusion and future work we propose a facet-aware evaluation setup for better assessment of information coverage in extractive summarization. we construct an extractive summarization dataset and demonstrate the effectiveness of facet-aware evaluation on this newly constructed dataset, including better human correlation on the assessment of information coverage, and the support for fine-grained evaluation as well as comparative analysis. we also evaluate sentence regression approaches and explore the feasibility of fully-automatic evaluation without any human annotation. in the future, we will investigate multi-document summarization datasets such as duc @xcite and tac @xcite to see whether our findings coincide when multiple references are provided. we will also explore better sentence regression approaches for the use of both extractive summarization methods and automatic fam creation. wei zhao, maxime peyrard, fei liu, yang gao, christian m. meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 563-578, hong kong, china. association for computational linguistics. qingyu zhou, nan yang, furu wei, shaohan huang, ming zhou, and tiejun zhao. 2018. neural document summarization by jointly learning to score and select sentences. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 654-663, melbourne, australia. association for computational linguistics. markus zopf, eneldo loza mencía, and johannes fürnkranz. 2018. which scores to predict in sentence regression for text summarization? in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1782-1791, new orleans, louisiana. association for computational linguistics. reference summary document extracted summary scu1.1, scu2.1, scu2.3, scu3.1 scu1.1 scu1.2 scu1.3 scu1.4 scu2.1 scu2.2 scu2.3 scu3.1 rouge evaluation (token match) reference sentence 2 (facet 2) reference sentence 3 (facet 3) document sentence 2 document sentence 3 document sentence 6 document sentence 1 document sentence 2 document sentence 3 document sentence 4 document sentence 5 document sentence 6 document sentence 7 facet-aware evaluation (facet match) fam annotation (colored) pyramid scu annotation pyramid scu annotation (during evaluation) pyramid evaluation (scu match) reference sentence 1 (facet 1) human effort annotation evaluation figure 4: comparison of summarization metrics. support sentences are marked in the same color as their corresponding facets. scus have to be annotated for each extracted summary during evaluation, while facetaware evaluation can be conducted automatically by comparing sentence indices.