How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation (2021)

Abstract: Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent summarization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experiments on two aspects of summaries’ linguistic quality (coherence and repetitiveness) to compare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our survey, we also find that study parameters such as the overall number of annotators and distribution of annotators to annotation items are often not fully reported and that subsequent statistical analysis ignores grouping factors arising from one annotator judging multiple summaries. Using our evaluation experiments, we show that the total number of annotators can have a strong impact on study power and that current statistical analysis methods can inflate type I error rates up to eight-fold. In addition, we highlight that for the purpose of system comparison the current practice of eliciting multiple judgements per summary leads to less powerful and reliable annotations given a fixed study budget.

## introduction current automatic metrics for summary evaluation have low correlation with human judgements on summary quality, especially for linguistic quality evaluation @xcite . as a consequence, manual evaluation is still vital to properly compare the linguistic quality of summarization systems. while the document understanding conferences (duc) established a standard manual evaluation procedure @xcite , we conduct a comprehensive survey of recent works in text summarization that reveals a wide array of different evaluation questions and methods in current use. furthermore, duc procedures were designed for a small set of expert judges, while current evaluation campaigns are often conducted by untrained crowd-workers. the design of the manual annotation, specifically the overall number of annotators as well as the distribution of annotators to annotation items, has substantial impact on power, reliability and type i errors of subsequent statistical analysis. however, most current papers (see section 2) do not consider the interaction of annotation design and statistical analysis. we investigate the optimal annotation methods, design and statistical analysis of summary evaluation studies, making the following contributions: in summarization evaluation, grouping factors arise whenever one annotator rates multiple summaries. ## literature survey we survey all summarization papers in acl, eacl, naacl, conll, emnlp, tacl and the computational linguistics journal in the years 2017-2019. we choose this timeframe as we are interested in current practices in summarization evaluation: 2017 marks the publication of the pointer generator network @xcite , which has been highly influential for neural summarization. we focus our analysis on papers that present a novel system for single-or multi-document summarization and take a single or multiple full texts as input and also output text (sds/mds). this allows us to concentrate on recommendations for human evaluation of newly developed summarization systems. 1 out of the resulting 105 sds/mds system papers, we identify all papers that conduct at least one new comparative system evaluation with human annotators for further analysis, leading to 58 papers in the survey. the fact that this is only about half of all papers is troubling given that it has been recently demonstrated that current automatic evaluation measures such as rouge @xcite ) are 1 excluded from the analysis are sentence summarization or headline generation papers, although most of the points we make hold for their evaluation campaigns as well. summarization evaluation papers that do not present a new system but concentrate on sometimes large-scale system comparisons are discussed in the related work section instead. lists of all included and excluded papers are given in supplementary material, which also contains exact evaluation parameters per paper in a spreadsheet. ## coherence and repetition annotation to elicit summary judgements for analysis, we conduct studies on two linguistic quality tasks. in the first, we ask annotators to judge the coherence of the summaries, while in the second we ask for the repetitiveness of the summary. we select these two tasks over the more frequent fluency task as we found in preliminary investigations that many recent summarization systems already produce highly fluent text, making them hard to differentiate. we do not evaluate overall and content as both require access to the input document, which differentiates these questions from the linguistic quality evaluation of the summaries. for both tasks, we conduct one study using a seven-point likert-scale (likert) and another using a ranking-based annotation method (rank), where annotators rank summaries for the same document from best to worst. screenshots of the interfaces for both approaches and full annotator instructions are given in appendix a. corpus and systems. mirroring a common setup (see section 2), we select four abstractive summarization systems and the reference summaries (ref) for analysis. we randomly sample 100 documents from the popular cnn/dm corpus @xcite with corresponding summaries from all systems to form the item set for all our studies. study design. we ensure a sufficient total number of annotators by using a block design. we separated our corpus into 20 blocks of 5 documents and included all 5 summaries for each document in the same block, which results in 5 × 5 = 25 summaries per block. all items in a block were judged by the same set of three annotators. no annotator was allowed to judge more than one block. this results in a total of 3 × 20 = 60 annotators and 1500 judgements per task. figure 1 shows a schematic overview of our design, which balances the need for a large enough annotator pool with a sufficient task size to be worthwhile to annotators. we recruited native english speakers from the crowdsourcing platform prolific 2 and carefully adjusted the reward to be no lower than £7.50 per hour based on pilot studies. summaries (or sets 2 prolific.com of summaries for rank) within a block were presented in random order. ## ranking vs. likert in the remainder of this section, we focus on the reliability of the two methods as well as their cost-effectiveness. ## statistical analysis and type i errors the two most common significance tests in summarization studies, anova and t-test (see method. the ordered logit model we employ can be described as follows: we fit all models using the ordinal r-package @xcite and compute pairwise contrasts between the parameters estimated for each system using the emmeans-package @xcite with tukey-adjustment. to demonstrate the problem of ignoring the grouping factors, we can now sample artificial data from the model distribution and try to analyse it with inappropriate tests. this monte-carlo simulation is similar to the more general analysis of @xcite . we set β to 0 so all systems perform equally well on the population level and only keep the (zero-mean) document and annotator effects in the model. the false-positive rate of statistical tests on this artificial data should thus be no higher than the significance level. we then repeatedly apply both the t-test and the approximate randomization test (art) @xcite , a non-parametric test, to samples drawn from the model and determine the type i error rate at @xmath0.05. we set the number of documents to 100 and demand 3 judgements per summary to mirror a common setup in manual evaluation. we then vary the total number of annotators between 3 and 300 by changing how many summaries a single annotator judges. results. we report results given the model estimated for likert in figure 4 . ignoring the dependencies between samples leads to inflated type i error rates, whether using the t-test or the art. this is especially severe when only few annotators judge the whole corpus. in the extreme case with only three annotators in total, the null-hypothesis is rejected in about 40% of trials at a significance level of 0.05 in both tasks. even our original design with 60 annotators still sees an increase of the type i error rate by about 3%. only if every annotator judges a single document and annotations are averaged per document, samples are independent and thus the real error is at the nominal 0.05 level. this design, however, is unrealistic given that annotators must be recruited and instructed. we suggest two solutions to this problem: either use mixed effect models or aggregate the judgements so samples become independent. this allows the assumptions of simpler tools such as art to be met. in our study, we could average judgements in every block to receive independent samples. this is only possible, however, if the design of the study considers this problem in advance: a crowd-sourcing study that allows annotators to judge as many samples as they like is unlikely to result in such a design. ## study design and study power when conducting studies for system comparison, we are interested in maximizing their power to detect differences between systems. for traditional analysis, the power is ultimately determined by the number of documents (or judgements, when no aggregation takes place) in the study. however, when analysis takes into account individual annotators, power becomes additionally dependent on the total number of annotators and how evenly they participated in the study. this gives additional importance to the design of evaluation studies. in this section, we thus focus on how to optimize studies for power and reliability. we first show that for well-powered experiments, we need to ensure that a sufficient total number of annotators participates in a study. in the second part of this section, we will then demonstrate studies can improve their power by not eliciting multiple judgements per summary. ## related work human evaluation has a long history in summarization research. this includes work on the correlation of automatic metrics with human judgements @xcite @xcite @xcite @xcite @xcite and improving the efficiency of the annotation process @xcite @xcite . the impact of annotator inconsistency on system ranking has been studied both by @xcite and @xcite . to the best of our knowledge, we are the first to investigate the implications of annotator variance on the statistical analysis and the design in summarization system comparison studies. for general nlg evaluation, van der lee et al. (2019) establish best practices for evaluation studies. we extend on their advice by conducting experimental studies specifically for summary evaluation. in addition, we show the importance of study design and consideration of annotator-effects in analysis on real world data. the advice of @xcite regarding annotation sequence effects should be taken into account in addition to our suggestions. ## conclusion we surveyed the current state of the art in manual summary quality evaluation and investigated methods, statistical analysis and design of these studies. we distill our findings into the following guidelines for manual summary quality evaluation: method. both ranking and likert-type annotations are valid choices for quality judgements. however, we present preliminary evidence that the optimal choice of method is dependent on task characteristics: if many summaries are similar for a given aspect, likert may be the better option. analysis. analysis of elicited data should take into account variance in annotator preferences to avoid inflated type i error rates. we suggest the use of mixed effect models for analysis that can explicitly take into account grouping factors in studies. alternatively, traditional tests can be used with proper study design and aggregation.