Best Practices for Crowd-based Evaluation of G erman Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020)

Abstract: One of the main challenges in the development of summarization tools is summarization quality evaluation. On the one hand, the human assessment of summarization quality conducted by linguistic experts is slow, expensive, and still not a standardized procedure. On the other hand, the automatic assessment metrics are reported not to correlate high enough with human quality ratings. As a solution, we propose crowdsourcing as a fast, scalable, and cost-effective alternative to expert evaluations to assess the intrinsic and extrinsic quality of summarization by comparing crowd ratings with expert ratings and automatic metrics such as ROUGE, BLEU, or BertScore on a German summarization data set. Our results provide a basis for best practices for crowd-based summarization evaluation regarding major influential factors such as the best annotation aggregation method, the influence of readability and reading effort on summarization evaluation, and the optimal number of crowd workers to achieve comparable results to experts, especially when determining factors such as overall quality, grammaticality, referential clarity, focus, structure & coherence, summary usefulness, and summary informativeness.

## introduction even though there has been an enormous increase in automatic summarization research, human evaluation of summarization is still an understudied aspect. one the one hand, there is no standard procedure for conducting human evaluation, which is leading to a high degree of variation and different results @xcite ; on the other hand, human evaluation is usually carried out in a traditional laboratory environment by linguistic experts, which is costly and time-consuming to run and prone to subjective biases @xcite . therefore, automatic evaluation metrics such as bleu and rouge have been used as substitutes for human evaluation @xcite . however, they require expert summaries as references to be calculated and are often reported not to correlate with human evaluations regarding the readability, grammaticality, and content-related factors @xcite . in the other nlp domains, crowdsourcing has been proposed as an alternative to overcome these challenges, showing that crowd workers' aggregated responses could produce quality approaching those produced by experts @xcite @xcite . in the summarization evaluation, very few researchers have investigated crowdsourcing as an alternative, eventually concluding that the chosen crowd-based evaluation methods are not reliable enough to produce consistent scores @xcite . however, the authors did not apply any pre-qualification test, did not provide information about the number of crowd workers, did not apply annotation aggregation methods, or did not analyze the effect of reading effort and readability of source texts caused by the text's structural, and formal composure. additionally, they used the tac and cnn/daily mail data set derived from high-quality english texts. so, there is a research gap regarding the best practices for crowd-based evaluation of summarization, especially for languages other than english and noisy internet data. we address this gap in the following ways: 1) we use a german summarization data set derived from an online question-answering forum; 2) we apply pre-qualification tests and set a threshold for minimum task completion duration in crowdsourcing; 3) we collect intrinsic and extrinsic quality ratings from 24 different crowd workers per summary in order to analyze consistency; 4) we use different annotation aggregation methods on crowdsourced data; 5) we analyze the effect of annotation aggregation methods, reading effort, and the number of crowd workers per item on robustness, comparing results from a) expert assessment; b) crowd assessment; c) state of the art automatic assessment metrics. especially, languages other than english can benefit from our results, since they lack easyto-use automatic evaluation metrics in the form of simplified toolkits, and a well-executed evaluation can accelerate the research on automatic summarization @xcite . ## results results are presented for the scores overall quality (oq), the five intrinsic quality scores (including grammaticality (gr), non-redundancy (nr), referential clarity (rc), focus (fo), structure & coherence (sc)) and the three extrinsic quality scores (summary usefulness (su), post usefulness (pu) and summary informativeness (si)). we will refer to these labels by their abbreviations in this section. for our human-based evaluation, we analyzed 10,800 ratings from the crowdsourcing study and 900 ratings from the expert evaluation. for automatic evaluation, we analyzed the bleu, rouge-1, rouge-2, rouge-l, bertscore (we use fscores for these metrics), bleurt by taking the mean of scores calculated using two expert summaries and the blanc scores resulting in 350 scores (50 summaries x 7 automatic metrics). ## conclusion and future work in this paper, we provide a basis for best practices for crowd-based summarization evaluation by comparing different annotation aggregation methods, analyzing the effect of reading effort and readability, and approaching an estimate of an optimal number of required crowd workers per item in order to as closely as possible resemble experts' assessment quality through crowdsourcing. when determining structure & coherence, we suggest that crowdsourcing can be used as a direct substitute for experts proven by the very strong correlation coefficient. for determining overall quality, grammaticality, referential clarity, focus, summary usefulness, and summary informativeness, crowdsourcing can be preferred as the overall correlation still results strong, but the results should be interpreted carefully. however, when evaluating nonredundancy and post usefulness, experts should be used for more robust results, as correlations result moderate only. our experiments further recommend following best-practices when using crowdsourcing instead of experts: 1) in general 5-8 crowd workers should annotate a given summary, 2) mos should be used as an aggregation method to achieve optimally comparable results to experts, 3) crowdsourcing may be used at best when readability of the source and reading effort of the task is of rather low and straightforward nature. we also confirm the findings of dumitrache et al. (2018b) that crowdtruth performs better than the mace. further, we confirm that the automatic evaluation metrics bleu, rouge, and bertscore can not be used to evaluate the linguistic quality, and we show that automatic evaluation metrics correlate poorly with any content-related absolute human rating, from crowd or expert, verifying the findings of @xcite for our domain. therefore, crowdsourcing should generally be the preferred evaluation method over automated scores in the summarization evaluation. since the vast majority of research on summarization bases on the tac or cnn/dailymail data sets, there is a lack of works from other languages or domains. we address this gap by using a german forum summarization data set derived from an online forum in the telecommunication domain. contrary to the findings of @xcite and fabbri et al. (2020), we achieve significant correlations between the crowd and expert ratings ranging from moderate to very strong magnitude, as well as no significant difference in absolute mean rating in between the crowd and expert assessment for overall quality, structure & coherence, summary usefulness, and summary informativeness. other scales show a slight but still significant bias towards lower ratings of about less than 0.3pt absolute. these are important findings in the development of nlg tools for summarization. in particular, summarization tools developed for languages other than english for which it is harder to conduct expert evaluations and find easy-to-use automatic metrics could benefit highly from our findings. however, this study has some limitations since we conduct our analysis using only a single data set derived from an online forum in the telecommunication domain. the level of domain knowledge of crowd workers and experts about the telecommunication service might play a role when determining content-related quality measures such as post usefulness. so, the effect of the domain knowledge should be investigated in detail in future work. another shortcoming of this paper is that our summarization data set is derived from noisy internet data, and the summary length does not differ much. as shown in section 4.1.2, the readability of the source document and varying lengths of summaries might affect the results; therefore, the same anal-ysis should be conducted on one more data set. additionally, our data set was monolingual, so exploring the language-based effects will also be part of future work. further, this study is that we did not investigate the effect of the crowdsourcing task design and learning effect on the correlation coefficient between the crowd and expert ratings. questions regarding the limitation to the number of assignments taken on by an evaluator (both for crowd and expert) and evaluators' behavior (becoming more lenient or strict over time) should also be analyzed in future work. also, we did not use the pairwise comparison in our task design and only focused on absolute quality rating. for that reason, investigating the pairwise comparison using crowdsourcing and its comparison to absolute rating should be considered as an essential aspect of the crowdsourcing task design in future work. despite the limitations of our study, this paper is the first paper in the summarization evaluation literature that provides evidence for clear support for using crowdsourcing to evaluate summarization quality and adds to a growing corpus of research on the summarization evaluation.