Fact-based Content Weighting for Evaluating Abstractive Summarisation (2020)

Abstract: Abstractive summarisation is notoriously hard to evaluate since standard word-overlap-based metrics are insufficient. We introduce a new evaluation metric which is based on fact-level content weighting, i.e. relating the facts of the document to the facts of the summary. We fol- low the assumption that a good summary will reflect all relevant facts, i.e. the ones present in the ground truth (human-generated refer- ence summary). We confirm this hypothe- sis by showing that our weightings are highly correlated to human perception and compare favourably to the recent manual highlight- based metric of Hardy et al. (2019).

## introduction text summarisation compresses long textual documents into short summaries while retaining the most important information from the source. in contrast to extractive summarisation, which directly copies the most relevant fragments, abstractive summarization retains the most important facts and expresses them via paraphrasing, aggregating and even inferring new facts. recent advances in neural decoders led to a number of single-document summarisation systems that exhibit some level of abstraction in their outputs, usually in the simplest form of paraphrasing (see et al. (2017); narayan et al. (2018); liu and lapata (2019), inter alia). evaluating abstractive summarisation remains an open challenge (schluter, 2017; kryściński et al., 2019): first, decoders are amenable to pathogeniessuch as hallucination and/or omission of important information, which are hard to capture using existing evaluation metrics (cao et al., 2018; rohrbach et al., 2018; dušek et al., 2020). second, most datasets used for abstractive summarisation only contain a single reference summary, e.g. (narayan et al., 2018; völske et al., 2017), which most existing automatic metrics evaluate against, e.g. rouge using exact n-gram overlap (lin, 2004), and thus tend to downvote paraphrases. we propose a new evaluation metric based on content weighting, where we abstract away from the particular surface form of the target summary, but represent it as facts using semantic role labelling (srl). in this way, we aim to better capture the semantic correctness of a summary, i.e. be more sensitive to hallucinations and omissions. @xcite in particular, we weight the facts present in the source document according to the facts selected by a human-written summary. this alignment is conducted using contextual, rather than token-level, embeddings, e.g., bert (devlin et al., 2019). for evaluation, we measure whether an automatically generated summary is able to capture the same facts as the target. we also show that the computed weights correlate well with human perception. our code is available at https://github. com/xinnuoxu/corrfa_for_summarizaion . ## related work the problem of reference bias has been addressed in several ways. first, metrics based on tokenlevel or wider context embedding similarities which aim to better capture paraphrases but remain largely word-oriented, e.g. (sun and nenkova, 2019; zhang et al., 2019; zhao et al., 2019; clark et al., 2019). goodrich et al. (2019) come close to our approach by using entity and relation extraction, but their approach is limited to texts that lend themselves to be represented by rdf triples. an alternative is manual evaluation against the source document. this entails selecting content either using domain experts, e.g., the pyramid method (nenkova and passonneau, 2004), factoids fact1-tweet: [arg0: the queen] has [v: tweeted] [arg1: her thanks] [arg2: to people who sent her 90th birthday messages on social media] fact2-send: the queen has tweeted her thanks to [arg0: people] [r-arg0: who] [v: sent] [arg1: her 90th birthday messages] [argm-loc on social media] fact1-tweet arg0 v arg1 arg2 the queen had tweeted her thanks srl propositions tree mr arg0 v arg1 argm-loc people r-arg0 who sent her 90th birthday messages on social media fact2-send figure 1: list of srl propositions and corresponding tree mr with two facts for the sentence "the queen has tweeted her thanks to people who sent her 90th birthday messages on social media". (teufel and van halteren, 2004), or via crowdsourcing (shapira et al., 2019; hardy et al., 2019). however, evaluation based on a small human-labelled test set is noisy, time consuming, and costly. xenouleas et al. (2019) propose a referenceless metric, which only checks properties of the summary, not its relation to the original document. sun and nenkova (2019) compare average token and sentence elmo embeddings against the document and claim good (system-level) correlations. another option to avoid reference bias is question-based evaluation, either elicited manually (clarke and lapata, 2010; narayan et al., 2018) or automatically (scialom et al., 2019). however, it requires reference summaries as base for generating questions, thus only checking the summary contents indirectly. ## content-weighting-based metrics we now use these weights to introduce two metrics: corr-f (fact-level) and corr-a (argumentlevel). let w f gold and w f cand denote the fact-level content weights calculated using the procedure from section 3 based on human-reference and system-generated summaries, respectively. similarly, w a gold , and w a cand denote the argument-level weights. corr-f is then the pearson correlation coefficient (pcc) between w f gold and w f cand . corr-a is pcc between w a gold and w a cand . in other words, corr-f and corr-a indicate whether the generated summary focuses on the informative main points in the document (i.e. the same points as the reference summary), on two different levels of granularity. ## metrics evaluation we validate our corr-f and corr-a metrics by collecting human judgements. in the following, we (1) collect content highlights from human judges using the amazon mechanical turk platform foot_8 and calculate manual content weighting based on them, (2) calculate correlations of the manual content weights with our automatic content weights, (3) compare our metrics against existing referencebased rouge (lin, 2004) and bertscore (zhang et al., 2019), as well as the referenceless manual hrouge score (hardy et al., 2019). @xcite we use the extreme summarisation dataset (xsum; narayan et al., 2018), which consists of bbc articles and accompanying single-sentence summaries, i.e. sub-headlines of the original articles, professionally written by the authors of the articles. due to the abstractive nature of the summaries, factoid content selection on phrase level is required beyond sentence-level extraction or tokenlevel matching, making this dataset a popular test bed for abstractive summarisation. we use the outputs of three recent abstractive summarization systems as evaluation targets for our metrics: (i) the pointer-generator model (ptgen; see et al., 2017); (ii) the topic-aware convolutional sequence-to-sequence model (tconvs2s; narayan et al., 2018) and (iii) the abstractive summarization model using pretrained bert encoders (bertsumabs; liu and lapata, 2019). @xcite ## conclusions and future work we present an automatic evaluation framework for abstractive summarisation, which is low-cost and robust, as it does not rely on expert annotators nor is susceptible to crowdsourcing noise. using fact representations, we are able to capture semantically similar, but at the same time distant in surface form, content in the summary that aligns with arbitrarily far-apart parts of the input document, casting our metric to be directly interpretable. our metric is more sensitive to perturbations of the facts in the target summary, which resemble common hallucination phenomena of neural decoders (see figure 2 -3 in appendix a for examples). in the future, we intend to investigate different meaning representation formalisms, such as amr (banarescu et al., 2013) and dynamic syntax (kempson et al., 2001) and extend to other datasets (e.g. multiplereference summarization) and tasks (e.g. response generation in dialogue). unite! unsupervised metrics for reinforced summarization models. in 2019 conference on empirical methods in natural language processing (emnlp) and 9th international joint conference on natural language processing (ijcnlp), hong kong. arxiv: 1909.01610. abigail see, peter j. liu, and christopher d. manning. 2017. get to the point: summarization with pointer-generator networks. in proceedings of the 55th annual meeting of the association for computational linguistics, pages 1073-1083, vancouver, canada. ori shapira, david gabay, yang gao, hadar ronen, ramakanth pasunuru, mohit bansal, yael amsterdamer, and ido dagan. 2019. crowdsourcing lightweight pyramids for manual summary evaluation. in naacl, minneapolis, mn, usa. arxiv: 1904.05929. simeng sun and ani nenkova. 2019. the feasibility of embedding based automatic evaluation for single document summarization. in 2019 conference on empirical methods in natural language processing (emnlp) and 9th international joint conference on natural language processing (ijcnlp), hong kong. simone teufel and hans van halteren. 2004. evaluating information content by factoid analysis: human annotation and stability. in proceedings of the 2004 conference on empirical methods in natural language processing, pages 419-426, barcelona, spain. association for computational linguistics. michael völske, martin potthast, shahbaz syed, and benno stein. 2017. tl;dr: mining reddit to learn automatic summarization. in proceedings of the workshop on new frontiers in summarization, pages 59-63, copenhagen, denmark. association for computational linguistics. stratos xenouleas, prodromos malakasiotis, marianna apidianaki, and ion androutsopoulos. 2019. sum-qe: a bert-based summary quality estimation model. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6007-6013, hong kong, china. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2019. bertscore: evaluating text generation with bert. arxiv preprint arxiv:1904.09675. wei zhao, maxime peyrard, fei liu, yang gao, christian m. meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in 2019 conference on empirical methods in natural language processing (emnlp) and 9th international joint conference on natural language processing (ijc-nlp), hong kong. arxiv: 1909.02622.