Reliability of Human Evaluation for Text Summarization: Lessons Learned and Challenges Ahead (2021)

Abstract: Only a small portion of research papers with human evaluation for text summarization provide information about the participant demographics, task design, and experiment protocol. Additionally, many researchers use human evaluation as gold standard without questioning the reliability or investigating the factors that might affect the reliability of the human evaluation. As a result, there is a lack of best practices for reliable human summarization evaluation grounded by empirical evidence. To investigate human evaluation reliability, we conduct a series of human evaluation experiments, provide an overview of participant demographics, task design, experimental set-up and compare the results from different experiments. Based on our empirical analysis, we provide guidelines to ensure the reliability of expert and non-expert evaluations, and we determine the factors that might affect the reliability of the human evaluation.

## introduction evaluation of summarization quality plays a crucial role in the development of summarization tools since a well-executed evaluation can help to determine whether the system has adequately outperformed the existing tools in terms of quality and speed or whether the designed properties work as intended @xcite . the human evaluation has been the most trusted evaluation method and used as gold standard for summarization evaluation @xcite . however, in recent years, some researchers have provided an extensive overview of papers with human evaluation and pointed out that there is a lack of standardized procedures leading to mostly non-comparable and non-reproducible results (van der @xcite @xcite . @xcite have reported based on the analysis 165 papers with human evaluation published in inlg and enlg that more than 200 different terms have been used for human evaluation, which results in lack of clarity in reports and extreme diversity in approaches. @xcite have analyzed 304 research papers published in inlg and acl conferences and reported that only 3% of 304 analyzed papers described the demographics, 6% provided the details about task design, 19% reported any inter-rater agreement score, 23% conducted a statistical analysis for human evaluation, and 32% reported the number of different evaluators per item, where 92% of the reported cases only one rating is used. in this paper, we aim to contribute the human evaluation research as follows: 1) we conduct series of human evaluation with experts, crowd, and laboratory participants on two different data sets, 2) we report on the participant demographics, task design, and evaluation criteria 3) we demonstrate a comprehensive statistical analysis of human experiments, and 4) we provide guidelines to ensure the reliability of experts and non-experts and determine the factors affecting the human reliability grounded by the empirical evidence from our experiments. data associated with this work is available at https://github.com/nesliskender/ reliability_humeval_summarization . ## related work human evaluation of text summarization can be conducted either by linguistic experts or nonexperts such as laboratory participants or crowd workers. however, expert evaluation has been established as the gold standard in the summarization evaluation and the reliability of non-experts has been repeatedly questioned @xcite . @xcite have conducted a crowdsourcing experiment for summarization evaluation for the first time and concluded that crowd workers can not evaluate summary quality because of the non-correlation with experts. however, they did not report the number of crowd workers per summary. @xcite have compared the crowd ratings with expert ratings using five crowd workers per item. they have also reported that crowd ratings do not correlate with experts and emphasized the need for protocols for improving the human evaluation of summarization. further, @xcite al. (2018); falke et al. (2017); fan et al. (2018) have used crowd workers to evaluate the quality of their automatic summarization systems without questioning the reliability of crowd workers. when we look at the approaches used for human summarization evaluation, they can be broadly classified into two categories: intrinsic and extrinsic evaluation @xcite @xcite . in intrinsic evaluation, the summarization output's quality is measured based on the summary itself without considering the source text. generally, it has been carried out as a pair comparison (compared to expert summaries) or using absolute scales without showing a reference summary (jones and @xcite . however, the extrinsic evaluation, called also task-based evaluation, aims to measure the summary's impact on the completion of some task based on the source document @xcite . reiter and @xcite have argued that the extrinsic evaluation is more useful than intrinsic because the summarization systems are developed to satisfy the information need from the source text in a condensed way, but van der @xcite have reported that only 3% of the papers presented an extrinsic evaluation. further, the quality criteria used in the human evaluation and the terminology used for describing these criteria had a high degree of variation, 200+ variations in terminology @xcite . researchers have used either the same terminology but evaluated something different or used different terminology but measured the same thing @xcite . in most cases, they did not define the quality criteria they investigated or cite a reference for it, making it difficult to compare the results and draw conclusions across the papers. the scales for evaluation have also varied often, such as likert @xcite , categorical choice (yes or no), or rank-based scale (van der @xcite . so, human evaluation lacks structured, reliable evaluation practices, and the current way of reporting human evaluation in research papers generates non-comparable and non-reproducible results. we aim to contribute to human evaluation research for text summarization by determining the intrinsic and extrinsic quality in a reliable and reproducible way with our experiments in section 3. ## experiments as our source documents, we used the 67 unique post-query pairs from a telecommunication company's customer service forum in german, where customers ask questions about the company's products and services such as "where can i find my customer number" or "my internet is not working". each query had 6-10 corresponding forum posts, including the answers from other customers to provide a solution or at least some help to the customer problem. the average word count of the posts was 571.2, the shortest one with 150 words, and the longest one with 1006 words, where the average word count of the corresponding queries was 9.1, the shortest query with three, and the longest with 23 words. we conducted series of human experiments on this data set shown in so, we determined intrinsic quality using six different quality criteria: overall quality, defined as "responsiveness evaluation" in @xcite , and the five readability (linguistic) measures (grammaticality, non-redundancy, referential clarity, focus, and structure & coherence) defined as in @xcite . we evaluated the extrinsic quality using following three measures: summary usefulness defined as "content responsiveness" in @xcite , source usefulness (in our case post usefulness, because our source documents are forum posts) defined as "relevance assessment" in @xcite , and summary informativeness defined as "informativeness" in @xcite . we conducted all our evaluations using a continuous scale, 5-point mean opinion score (mos) with the labels very good, good, moderate, bad, very bad, which is one of the most applied scales in subjective quality assessment @xcite . ## results results are presented for the mean opinion scores (mos) of overall quality (oq), grammaticality (gr), non-redundancy (nr), referential clarity (rc), focus (fo), structure & coherence (sc), summary usefulness (su), post usefulness (pu) and summary informativeness (si) collected in experiments 2, 3, 4, 5, 7, and 8 (see ## conclusion and future work in this paper, we report a comparative analysis of series of human evaluation experiments with crowd workers, laboratory participants, and experts on two different data sets to determine the reliability of human evaluation for text summarization. however, the research papers with expert evaluations for summarization have not reported any mediation meetings, let alone only 19 % reported the inter-rater agreement scores in the range of 0.3-0.5 @xcite . this raises the question of expert reliability, and to avoid that, we recommend having mediation meetings with experts for reliable expert evaluation based on our results in section 4.1.1. with our analysis, we showed that mediation meetings are elementary to assure the reliability of expert evaluations for all quality measures. further, we found out that the number of crowd workers per item determines the crowd evaluation's reliability. van der @xcite showed only 57 % of papers specified number of evaluators and the median was 3 among the papers which have reported the evaluator number. but our analysis in section 4.1.2 showed that when using crowdsourcing, three crowd workers per item can only deliver moderately reliable results and around ten or more different crowd workers should evaluate each summary. this result is also inline with our previous findings in @xcite . while the environment (crowd vs. lab) does not affect the human evaluations, the level of expertise might have affected the human evaluation. although there are mostly strong correlations between the experts and non-experts, their evaluations do not match 100%. depending on the evaluation aim or the end-user group of the summarization system, the evaluator's expertise should be determined, e.g., summarization systems developed for naive end-users should be evaluated by the naive end-users rather than the experts, and expert systems should be evaluated by linguistic experts. additionally, the summary generation method (human vs. machine) might cause a bias in crowd assessments. because of machine summaries' unnaturalness, the crowd workers tended to rate machine summaries lower than the experts. the feedback that the summaries were very "unnatural" and "robotic" from the crowd workers in experiment 8 also confirms this finding. but still, crowd workers can be used as a direct substitute for experts to determine the goodness of automatic evaluation metrics developed for machine summaries. however, this paper has some limitations regarding the data set and task design. we used one task design with a single rating scale (5-point mos scale) and the same set of definitions and explanations for our evaluation criteria in all our experiments, which were conducted on small sized data sets. in future work, we plan to include different human evaluation criteria, compare different rating scales with each other, conduct a/b testing with a second task design, which includes improved definitions of evaluation criteria based on the expert mediation sessions, and expand the data set size to increase the statistical power of our analysis. additionally, we plan to conduct virtual mediation sessions between two or three crowd workers to find out if we can reach similar results to experts with a small number of crowd workers. despite the limitations of our paper, we believe that this paper makes a significant contribution to human evaluation research of text summarization. as