Finding a Balanced Degree of Automation for Summary Evaluation (2021)

Abstract: Human evaluation for summarization tasks is reliable but brings in issues of reproducibility and high costs. Automatic metrics are cheap and reproducible but sometimes poorly correlated with human judgment. In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but replaces the manual work of judging SCUs’ presence in system summaries with a natural language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role labeling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and retain SCUs that are more difficult to simulate, which provides a smooth transition and balance between automation and manual evaluation. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 existing meta-evaluation datasets and our newly collected PyrXSum (with 100/10 XSum examples/systems). It shows that Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection.

## introduction evaluating the quality of summaries is a challenging task. human evaluation is usually regarded as the gold standard. out of different human evaluation methods, pyramid (nenkova and passonneau, 1 our code and data are publicly available at: https: //github.com/zhangshiyue/lite2-3pyramid 2004) has been perceived as an objective and reliable protocol and used by early summarization benchmarks, e.g., tac @xcite . given one or several reference summaries of an example, human assessors first exhaustively extract summary content units (scus), each scu contains a single fact, from the reference(s), and then check whether they are present in a system summary. figure 1 shows an example of human-labeled scus. despite the reliability, manual evaluation is usually: (1) not reproducible, results may change when different evaluators are involved, making it hard to compare the results across papers; (2) expensive, in terms of time and cost. thus, it is unlikely to apply human evaluation extensively in model selection (e.g., to choose the best checkpoint); instead, people usually treat it as an additional quality verification step. aiming to work as a proxy of humans, many automatic metrics have been proposed @xcite @xcite @xcite . however, most of them cannot reliably substitute human evaluation due to the unstable performance across datasets @xcite , weak to moderate correlations with human judgment @xcite , or more indication of topic similarity than information overlap @xcite . in this work, we want to combine human and automatic evaluations and find a balance between reliability and reproducibility (plus expense). recall the pyramid method @xcite , where these scus for reference summaries only need to be annotated once, then they can be fixed. it means scus can come with the datasets and are reusable for evaluating different systems. hence, what hinders this method from being reproducible is its second step of asking humans to judge the presence of scus in system summaries. whenever we have a new summarizer, we need to collect human labels for this step. therefore, we propose to retain the reusable scus but replace human effort in the second step with a neural model. basically, people are answering whether a scu is entailed by the summary, which is closely related to the natural language inference (nli) task, i.e., judging whether a hypothesis is entailed by the premise. a lot of nli datasets are available @xcite @xcite and recent nli models have achieved close-to-human-level performance. hence, we use a pretrained nli model and finetune it on some in-domain gold labels of scus' presence. then, we replace humans with the finetuned model, so that the evaluation results are reproducible as long as the same model is used. meanwhile, it can run automatically during development to guide model selection and the evaluation cost will be dramatically reduced. @xcite propose litepyramid to simplify the standard pyramid method via crowdsourcing. following but different from their work, we additionally automate the presence annotation, and hence we call our method lite 2 pyramid. lite 2 pyramid still requires human efforts to extract scus from reference summaries, and this step is usually considered to be more difficult. early benchmarks, e.g., tac @xcite , are small-sized with fewer than 100 examples in the evaluation set, for which it is already expensive to manually collect scus. however, current popular summarization datasets, e.g., cnn/dm @xcite , contain more than 10k evaluation examples, and hence we want to simulate scus via an automatic method for such large-scale datasets. for this, we make use of semantic role labeling (srl) that can automatically decompose a sentence to semantic triplets, e.g., subject-verb-object, and we take each triplet as a pseudo-scu, which we call semantic triplet unit (stu). figure 1 illustrates the difference between scus and stus. although stus do not always contain a single fact and some information might also be misrepresented, we find that it can reasonably simulate scus and lead to a fully automatic metric, lite 3 pyramid. lastly, instead of using either all human-labeled scus or all automated stus, we investigate balanced trade-offs in between, e.g., using half scus and half stus. a naive way is to randomly sample some reference sentences and substitute their scus with stus. however, we find it is unstable and sometimes even works worse than using all stus. more reasonably, we design an active learning @xcite inspired selection method to help decide which sub-parts of the dataset are more worthy of obtaining expensive scus for. for this, we develop a regressor to predict the "simulation easiness" of each reference sentence: if a sentence is too complex to be well represented by stus, we will ask humans to annotate scus for it; otherwise, we can apply automatic srl. we call this method as lite 2.x pyramid, since it provides a smooth, flexible transition from lite 2 pyramid to lite 3 pyramid and balances reliability with cost. to comprehensively evaluate the quality of metrics, we not only use 3 existing meta-evaluation datasets (tac2008 @xcite , tac2009 @xcite , realsumm @xcite ) but also newly collect pyrxsum with 100 xsum @xcite test examples plus summaries produced by 10 systems. next, we compare our new metrics to 15 existing automatic metrics on these 4 meta-evaluation setups for both system-level and summary-level correlations with human pyramid scores. we find that lite 2 pyramid consistently has the best summary-level correlations and is reliable as an out-of-the-box metric. lite 3 pyramid also mostly performs better or competitively. lastly, the regressor-based lite 2.x pyramid can help substantially reduce annotation efforts for only small correlation drops, e.g., on tac2008, tac2009, it trades off only 0.01 absolute summary-level pearson correlation and 0 system-level correlation for 50% scu reduction. ## related works & background each example in a summarization dataset contains one or several source document(s) and one or several human-written reference(s). system-generated summaries are evaluated by comparing them to the references (i.e., reference-based) or directly scored (i.e., reference-free). this evaluation process is critical and directly affects our development choices. human (or manual) evaluation has been considered as the gold standard. early benchmarks @xcite conducted three human evaluations: responsiveness, linguistic quality, and pyramid. the first two ask humans to directly rate the overall responsiveness or linguistic quality on a likert scale. following this, some works collect ratings for different aspects, e.g., relevance, readability @xcite @xcite . however, these ratings may suffer from raters' subjectivity. pyramid @xcite has been perceived as a more objective method, and it is reference-based. it has two steps: pyramid creation and system evaluation. in the first step, humans exhaustively find the summary content unit (scu) contributors from references, each contributor describes a single fact; contributors with the same meaning will be merged into one single scu; then each scu is weighted by how many contributors it has, equal to the number of references in which it is found. in the second step, each scu has been manually checked its presence in the system summary; and the pyramid score is the normalized sum of present scus' weights (essentially, a recall score). @xcite normalize it by the total weight of the best possible summary. recently, @xcite propose litepyramid. it removes scu merging and weighting, allowing scus of the same meaning to co-exist, and they show that the evaluation can be reliably conducted by crowdsourcing workers. automatic metrics trade off the reliability of human evaluation for reproducibility, low cost, and fast speed. many automatic metrics have been introduced, the majority of which are referencebased. some metrics measure the n-gram overlap @xcite , out of which rouge @xcite is the most widely adopted metric till today. some other works compute the similarity over n-gram graphs @xcite or distributions @xcite . since exact n-gram matching is too rigid, meteor (banerjee and lavie, 2005; denkowski and lavie, 2014) provides flexibility by stemming, synonyms, etc., and recently, a few metrics enable "soft" matching through contextualized word embeddings @xcite @xcite . however, @xcite point out that the n-gram based metrics indicate more topic similarity than information overlap. structural evaluation metrics have also been proposed beyond n-grams. bewt-e @xcite decomposes the system summary and the reference(s) into syntactic units and compute their similarities, and decomposed-rouge @xcite computes rouge for each syntactic category. apes @xcite and qaeval @xcite are qa-based metrics that assume similar answers will be obtained from similar system summaries and reference(s). automatic pyramid methods have also been proposed @xcite @xcite . they usually decompose both the system summary and the references into smaller units (e.g., elementary discourse units) and compare the two list of units. differently, our lite 3 pyramid only decomposes the reference summaries to semantic triplet units (stus), and we use nli to judge the presence of each stu in the system summary, which is closer to the original pyramid's procedure and leads to better correlations with human scores (refer to section 5). @xcite propose a learned metric, s3, that is trained to directly predict human pyramid or responsiveness scores based on rouge, framenet features, etc., which is similar to how we finetune the nli model with human labels of scus' presence. @xcite is distantly related to us in the way of representing texts by srl, but it is used to weigh the content in the source document(s). besides, some reference-free metrics are introduced for summary quality estimation @xcite @xcite or faithfulness evaluation @xcite . semi-automatic evaluation is introduced by @xcite . they automatically decompose both system summary and reference(s) into semantic units and then ask humans to match/align the two lists of units. in contrast, our semi-automatic lite 2 pyramid retains the reusable scus while automatically judges the scus' presence in the system summary (via nli). ## evaluation correlation with human scores. following the standard meta-evaluation strategies used in previous works @xcite @xcite , we evaluate metrics by two types of correlation with gold human scores. system-level correlation aims to evaluate how well can the metric compare different summarization systems? we denote the correlation measure as k, human scores as h, the metric as m, and generated summaries as s. we assume there are n examples and s systems in the mete-evaluation dataset. then, the system-level correlation is defined as: summary-level correlation answers if the metric can reliably compare summaries generated by different systems for the same document(s). using the same notations, this correlation is written by: we use pearson r or spearman ρ as the correlation measure k. pearson measures linear correlation while spearman measures ranking correlation. ## conclusion we propose to combine manual effort and automation for summary evaluation. we introduce a semiautomatic lite 2 pyramid that gains reproducibility by replacing part of human effort with an nli model. following it, an automatic lite then, we collect the scus' presence labels for each system summary on amazon mechanical turk. figure 3 illustrates the data annotation instructions and interfaces shown to crowdsourcing workers. the summaries usually only contain one sentence. we estimate it will take around 30-45 seconds for a native english speaker to finish one hit. following bhandari et al. (2020), we pay $0.15 per hit, which is respectably higher than the u.s. federal minimum wage requirement. meanwhile, we select annotators that are located in the u.s., have an approval rate greater than 98%, and have at least 10,000 approved hits. we collect 4 responses per summary (100 * 10 * 4 hits) and finally, 104 workers were involved. after annotation, we filter the annotations from a noisy worker who did 210 hits but disagreed with the majority in 72% of the time. after this filter-