Challenges in Reproducing Human Evaluation Results for Role-Oriented Dialogue Summarization (2023)

Abstract: There is a growing concern regarding the reproducibility of human evaluation studies in NLP. As part of the ReproHum campaign, we conducted a study to assess the reproducibility of a recent human evaluation study in NLP. Specifically, we attempted to reproduce a human evaluation of a novel approach to enhance Role-Oriented Dialogue Summarization by considering the influence of role interactions. Despite our best efforts to adhere to the reported setup, we were unable to reproduce the statistical results as presented in the original paper. While no contradictory evidence was found, our study raises questions about the validity of the reported statistical significance results, and/or the comprehensiveness with which the original study was reported. In this paper, we provide a comprehensive account of our reproduction study, detailing the methodologies employed, data collection, and analysis procedures. We discuss the implications of our findings for the broader issue of reproducibility in NLP research. Our findings serve as a cautionary reminder of the challenges in conducting reproducible human evaluations and prompt further discussions within the NLP community.

## introduction natural language processing (nlp) has witnessed remarkable advances in recent years. human evaluation plays a pivotal role in assessing the effectiveness of nlp systems and their performance in meeting specific task requirements. however, concerns have arisen regarding the reproducibility of human evaluation studies in the nlp community @xcite . reproducibility is defined as the ability of other researchers to repeat the experiments under identical conditions and obtain consistent results. as part of the reprohum campaign @xcite , which strives to systematically assess the reproducibility of human evaluation studies in nlp, we conducted a rigorous reproduction study of @xcite , with the title other roles matter! enhancing role-oriented dialogue summarization via role interactions. dialogue summarization aims to distil relevant information from conversations while preserving their context, presenting a concise and informative summary. the quality of such summarization systems is critical in real-world applications, and a careful evaluation of said quality is a prerequisite to the application of summarization systems. @xcite start from the idea that, when a system summarises a dialogue between a user (e.g., a customer) and an agent (e.g., someone who answers the customer's questions), it can be helpful to attend to each of these two roles @xcite separately. when a user's utterance is summarised, some information from the agent should be taken into account, and the other way around. the authors hypothesise that cross-attention and selfattention can help create an optimal combination of both roles, and they investigate various neural mechanisms for doing so, in particular bert @xcite and pgn @xcite . after an extensive metric-based evaluation, their human evaluation -on which our paper focuses -tests the hypothesis that, for both bert and png, better summaries are generated when both cross-attention and self-attention are used (in @xcite , the systems with these mechanisms are referred to as bert-both and png-both), compared to settings where the dialogue is summarised as one whole without separating the two roles (settings referred to as bert-multi and png-multi). they conclude that adding both interactions increases performance with respect to the baseline case. the objective of our reproduction study was to validate the reported statistical results from the original paper and to investigate the reproducibility of the human evaluation process as outlined by the authors. to achieve this, we meticulously replicated the experimental setup provided in the original work, while also seeking clarification from the authors regarding details of their experimental procedure. in this paper, we present our findings from the reproduction study, shedding light on the challenges and implications of conducting reproducible human evaluations in the nlp domain. our study uncovers significant discrepancies between the statistical results reported in the original paper and those obtained in our reproduction attempt. while we did not find any contradicting evidence, our results raise questions about the validity of the original statistical-significance findings. we emphasise that our aim is not to undermine a valuable piece of work, but to contribute to the ongoing discussion on reproducibility, fostering a more transparent and reliable foundation for future advancements. ## data for our evaluation, we used 100 sample dialogues from the same chinese sales dialogue summarization (csds) dataset used in @xcite . the samples were provided to us by the reprohum organizers. for each of the 100 dialogues, there are two kinds of summaries (user and agent) generated by each of the following four systems: pgn-multi, pgn-both, bert-multi, bert-both. thus, there are 800 summaries in total. a sub-summary refers to a complete sentence in the role-oriented summary. ## experimental setup we closely follow the guidelines provided in the original paper by @xcite . we asked participants to assess the summary quality of role-oriented dialogue summarization models on three aspects: informativeness, non-redundancy, and fluency. we sought to replicate the evaluation process as faithfully as possible, while also addressing certain details that were obtained from the original authors but were not explicitly mentioned in the original paper. we treated each "sub-summary" (i.e. sentence) in the role-oriented summary as an individual unit to be scored by the annotators. the evaluation was carried out by three trained volunteers who were familiarised with the evaluation rules provided by the original authors. in the original study, annotators were graduate school-level students and spoke native mandarin. they were recruited from among the members of the lab conducting the study. in a similar spirit, we recruited three phd candidates from the department of information and computing sciences at utrecht university, all of whom self-reported mandarin as their native language. contrary to the original study where the annotators were not paid for their participation, we will pay each of our annotators 120 euros for 12 hours of work. 1 the annotators assessed each sub-summary according to three pre-defined aspects: informativeness, non-redundancy, and fluency. each subsummary received a score for each aspect based on the perceived quality of the summary with respect to that particular aspect. as was done in the original paper, we first gave the three annotators the same ten summaries, and asked them to rate those summaries. to ensure the reliability of the obtained scores, we conducted an inter-annotator agreement analysis. this process involved comparing the scores given by the three volunteers for each sub-summary. we used cohen's kappa coefficient as a measure of agreement. this was calculated by concatenating all values for each participant together. we then gave each participant a different set of 30 summaries to rate. in total, there were 100 summaries: 10 were annotated by all three participants, while the remaining 90 were annotated by one participant each. to represent the summary quality in general, we aggregated the scores for all three aspects (informativeness, non-redundancy, and fluency) into an "overall" metric for each sub-summary. the overall score for a sub-summary was obtained by averaging the individual scores assigned by the annotators for that specific aspect. the obtained scores were then normalised to a range between 0 and 1 to facilitate comparison and presentation. the normalised scores were compiled into a table, which is analogous to there is some ambiguity regarding how scores should be computed for the summaries that were evaluated by the three participants. in particular, every sub-summary evaluated by a single participant has a single score; but for the summaries evaluated by all participants (which was done for the purpose of computing the inter-annotator agreement), there are three scores per sub-summary. the original paper does not specify how the scores were computed for these multi-annotated summaries. we performed our analysis under four "cases". these are defined by the way we compute the score for each multi-annotated summary: to ensure transparency and to facilitate reproducibility of our study, we have made our code and datasets publicly available on our github repository foot_1 . the repository contains the necessary scripts and documentation to replicate our experimental procedures and results accurately. ## conclusions in this paper, we conducted a reproduction study of the human evaluation in @xcite , as part of the reprohum campaign to assess the reproducibility of human evaluation in nlp @xcite . our objective was to assess the reproducibility of the results reported in the original paper and thoroughly investigate the difference between our results and the original paper's, if any. throughout our study, we sought to adhere closely to the original experimental setup. however, our findings reveal notable discrepancies in the statistical results obtained, particularly in comparing the improvements of the "both" method with respect to the "multi" method. in the original paper, "multi" is a baseline method, while "both" adds cross-attention and self-attention interactions to the models (see section 1). unlike the original work, our experiments did not demonstrate clear improvements in summary quality when considering role interactions. despite the differences in our obtained results, we acknowledge the high pearson correlation coefficient between the original paper's scores and our own, indicating a strong consistency in the relative ranking of models across the evaluation aspects. furthermore, while our findings were different from the original results in terms of statistical significance, we acknowledge that they are not contradictory, i.e., there is no model for which the authors of the original paper claim statistically significantly better results for "both" or "multi", while we find the opposite to be true (i.e., statistically significantly worse results). we believe that the statistical significance analysis employed in the original paper may have certain flaws. firstly, we maintain that a correction procedure for inflated type-1 error should have been applied, considering that multiple statistical significance tests were conducted on the same dataset. failure to account for this potential bias might have resulted in too many false positives (i.e. results which appear to be statistically significant but are not). secondly, the authors computed statistical significance tests, but then also drew conclusions from results that were not statistically significantly different. this should be avoided. one significant observation from our study was the relatively low level of agreement among the annotators. this raises concerns about the consistency of the evaluation process and the potential for different interpretations of the instructions. it would have been valuable to closely scrutinise the reasons for such disagreement. if the disagreements stemmed from differing interpretations of the guidelines, an update to the instructions and a restart of the annotation process might have been necessary. alternatively, if the disagreements were legitimate, the study could have been improved by having multiple annotators assess all the summaries, allowing for a better understanding of the inherent variability. this is along the lines of recent work that tries to account for inherent variability when training nlp models @xcite . regarding the reproducibility experiment itself, the description provided in the original paper was insufficient for us to fully attempt a replication. nonetheless, thanks to the cooperation of the authors, we were able to clarify the necessary procedures. even so, we had to perform four studies under different "cases", which refer to the various ways we pooled together the results of the first 10 summaries, annotated by all participants. moreover, the data collection process posed significant challenges, largely due to the participants making multiple errors that needed to be corrected before statistical analysis became feasible. namely, we observed several mismatches between the number of sentences and the number of annotations provided by participants. this was probably caused by the annotation being done in a spreadsheet. in those cases, we had to ask participants to correct their work. ensuring data quality and accuracy is crucial in human evaluation studies, and these difficulties further underscore the importance of transparent reporting and careful handling of data. finally, we wish to clarify that our focus was on the parts of the original paper that dealt with human evaluations, particularly in terms of reproducibility. we do not make any general claims about the strength of the entire original paper, which included metric-based evaluations as well. the results of the metric-based evaluation in the original work may indeed be more convincing. in conclusion, our reproduction study highlights the importance of carefully reporting the conditions under which a human evaluation was conducted to enhance reproducibility, and the need for thorough reporting of experimental details necessary for reproduction studies, as well as scrutiny of statistical significance analyses in nlp research. we also provide suggestions for future studies to enhance the reproducibility and transparency of human evaluation experiments. despite the challenges we encountered, we commend the authors for their cooperation, which allowed us to perform a comprehensive reproduction of their work. we believe that open dialogue and collaborative efforts within the research community are essential for advancing the field of nlp and achieving meaningful progress in dialogue summarization and other language generation tasks. anya belz, craig thomson, and ehud reiter. 2023. missing information, unresponsive authors, experimental flaws: the impossibility of assessing the reproducibility of previous human evaluations in nlp. in the fourth workshop on insights from negative results in nlp, pages 1-10, dubrovnik, croatia. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. rudali huidrom, ondřej dušek, zdeněk kasner, thiago castro ferreira, and anya belz. 2022. two reproductions of a human-assessed comparative evaluation of a semantic error detection system. in proceedings of the 15th international conference on natural language generation: generation challenges, pages 52-61, waterville, maine, usa and virtual meeting. association for computational linguistics. a heds sheet { "heds-criteria-criterion-response_elicitation-scale_presented_as-5": { "data": { "informativeness": true, "fluency": true, "non-redundancy": true }, "control": {}, "text": { "informativeness": "5. other (please describe)", "fluency": "5. other (please describe)", "non-redundancy": "5. other (please describe)" } }, "heds-criteria-criterion-response_elicitation-response_aggregation": { "data": { "informativeness": "average the set of per-sentence values.", "non-redundancy": "average the set of per-sentence values.", "fluency": "average the set of per-sentence values." }, "control": {} }, "heds-criteria-criterion-evaluation_mode-objective_or_subjective-1": { "data": { "informativeness": true, "non-redundancy": true, "fluency": true }, "control": {}, "text": { "informativeness": "1. objective", "non-redundancy": "1. objective", "fluency": "1. objective" } }, "heds-paper_and_resources-names_and_affiliations-person_completing_this_sheet-affiliation": { "data": { "": "utrecht university / tohoku university" }, "control": {} }, "heds-paper_and_resources-names_and_affiliations-person_completing_this_sheet-name": { "data": { "": "takumi ito" }, "control": {} }, "heds-criteria-criterion-criteria-output_aspect-1": { "data": { "informativeness": false, "non-redundancy": false, "fluency": true }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "1. form of output" } }, "heds-sample_evaluators_design-evaluators-evaluators-expertise-other_text": { "data": { "": "" }, "control": {} }, "heds-sample_evaluators_design-evaluators-evaluators-expertise-1": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-criteria-criterion-response_elicitation-form_of_response-10": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-criteria-criterion-response_elicitation-scale_presented_as-2": { "data": { "informativeness": false, "fluency": false, "non-redundancy": false }, "control": {}, "text": { "informativeness": "", "fluency": "", "non-redundancy": "" } }, "heds-system-input_types-8": { "data": { "": true }, "control": {}, "text": { "": "8. text: dialogue" } }, "heds-paper_and_resources-names_and_affiliations-contact_author-affiliation": { "data": { "": "utrecht university" }, "control": {} }, "heds-criteria-criterion-response_elicitation-form_of_response-7": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-experimental_design-experimental_conditions-8": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-system-input_languages-29": { "data": { "": true }, "control": {}, "text": { "": "29. chinese" } }, "heds-criteria-criterion-response_elicitation-form_of_response-6": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-evaluators-evaluators-payment-3": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-criteria-criterion-response_elicitation-form_of_response-5": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-sample-system_output_selection-1": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-sample_evaluators_design-experimental_design-quality_assurance-description": { "data": { "": "n/a" }, "control": {} }, "heds-sample_evaluators_design-evaluators-evaluators-expertise-2": { "data": { "": true }, "control": {}, "text": { "": "2. non-experts" } }, "heds-sample_evaluators_design-experimental_design-experimental_conditions-1": { "data": { "": true }, "control": {}, "text": { "": "1. evaluation carried out by evaluators at a place of their own choosing, e.g. online, using a paper form, etc." } }, "heds-ethics-review_body": { "data": { "": "no" }, "control": {} }, "heds-criteria-criterion-response_elicitation-effect_size_method": { "data": { "non-redundancy": "" }, "control": {} }, "heds-criteria-criterion-response_elicitation-verbatim_question": { "data": { "informativeness": "n/a ", "non-redundancy": "n/a", "fluency": "n/a" }, "control": {} }, "heds-sample_evaluators_design-evaluators-evaluators-known_to_authors-2": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-criteria-criterion-evaluation_mode-intrinsic_or_extrinsic-1": { "data": { "informativeness": false, "non-redundancy": true, "fluency": true }, "control": {}, "text": { "informativeness": "", "non-redundancy": "1. intrinsic", "fluency": "1. intrinsic" } }, "heds-criteria-criterion-evaluation_mode-objective_or_subjective-other_text": { "data": { "informativeness": "", "non-redundancy": "", "fluency": "" }, "control": {} }, "heds-criteria-criterion-response_elicitation-scale_presented_as-other_text": { "data": { "informativeness": "fill in the cells on the spreadsheet", "non-redundancy": "fill in the cells on the spreadsheet", "fluency": "fill in the cells on the spreadsheet" }, "control": {} }, "heds-sample_evaluators_design-evaluators-evaluators-payment-1": { "data": { "": true }, "control": {}, "text": { "": "1. paid (monetary compensation)" } }, "heds-sample_evaluators_design-evaluators-evaluators-description": { "data": { "": "chinese phd candidates in the same department as the authors." }, "control": {} }, "heds-sample_evaluators_design-sample-number_of_system_outputs": { "data": { "": "100" }, "control": {} }, "heds-system-output_types-4": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-criteria-criterion-evaluation_mode-objective_or_subjective-2": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-criteria-criterion-response_elicitation-form_of_response-4": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-criteria-criterion-response_elicitation-size_of_scale-2": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-evaluators-evaluators-known_to_authors-other_text": { "data": { "": "" }, "control": {} }, "heds-criteria-criterion-criteria-output_aspect-other_text": { "data": { "informativeness": "", "non-redundancy": "", "fluency": "" }, "control": {} }, "heds-criteria-criterion-evaluation_mode-absolute_or_relative-1": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-experimental_design-quality_assurance-method-1": { "data": { "": true }, "control": {}, "text": { "": "1. evaluators are required to be native speakers of the language they evaluate." } }, "heds-sample_evaluators_design-evaluators-evaluators-payment-other_text": { "data": { "": "" }, "control": {} }, "heds-sample_evaluators_design-experimental_design-experimental_conditions-other_text": { "data": { "": "" }, "control": {} }, "heds-criteria-criterion-criteria-quality_type-2": { "data": { "informativeness": false, "non-redundancy": false, "fluency": true }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "2. goodness" } }, "heds-criteria-criterion-response_elicitation-form_of_response-9": { "data": { "informativeness": false, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "", "non-redundancy": "", "fluency": "" } }, "heds-criteria-criterion-evaluation_mode-intrinsic_or_extrinsic-2": { "data": { "informativeness": true, "non-redundancy": false, "fluency": false }, "control": {}, "text": { "informativeness": "2. extrinsic", "non-redundancy": "", "fluency": "" } }, "heds-sample_evaluators_design-sample-system_output_selection-other_text": { "data": { "": "the same samples used in the original paper." }, "control": {} }, "heds-sample_evaluators_design-sample-system_output_selection-5": { "data": { "": true }, "control": {}, "text": { "": "5. other (please describe)" } }, "heds-sample_evaluators_design-experimental_design-evaluator_freedom-other_text": { "data": { "": "no restrictions." }, "control": {} }, "heds-criteria-criterion-criteria-quality_type-1": { "data": { "informativeness": true, "non-redundancy": true, "fluency": false }, "control": {}, "text": { "informativeness": "1. correctness", "non-redundancy": "1. correctness", "fluency": "" } }, "heds-criteria-criterion-evaluation_mode-absolute_or_relative-2": { "data": { "informativeness": true, "non-redundancy": true, "fluency": true }, "control": {}, "text": { "informativeness": "2. relative", "non-redundancy": "2. relative", "fluency": "2. relative" } }, "heds-paper_and_resources-names_and_affiliations-contact_author-name": { "control": {} }, "heds-sample_evaluators_design-evaluators-recruitment_method": { "data": { "": "sent an email to those who met the requirements" }, "control": {} }, "heds-paper_and_resources-names_and_affiliations-contact_author-email": { "data": { "": "c.j.vandeemter@uu.nl" }, "control": {} }, "heds-sample_evaluators_design-evaluators-evaluators-known_to_authors-1": { "data": { "": true }, "control": {}, "text": { "": "1. previously known to authors" } }, "heds-criteria-criterion-response_elicitation-form_of_response-2": { "data": { "informativeness": true, "non-redundancy": true, "fluency": true }, "control": {}, "text": { "informativeness": "2. direct quality estimation", "non-redundancy": "2. direct quality estimation", "fluency": "2. direct quality estimation" } }, "heds-sample_evaluators_design-experimental_design-experimental_conditions-7": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-sample_evaluators_design-evaluators-evaluators-are_authors-2": { "data": { "": true }, "control": {}, "text": { "": "2. evaluators do not include any of the authors" } }, "heds-sample_evaluators_design-experimental_design-evaluators_place_of_choosing": { "data": { "": "n/a" }, "control": {} }, "heds-sample_evaluators_design-sample-system_output_selection-3": { "data": { "": false }, "control": {}, "text": { "": "" } }, "heds-criteria-criterion": { "data": {}, "control": { "informativeness": false, "non-redundancy": false, "fluency": true }, "text": {} }, "heds-sample_evaluators_design-evaluators-training_practice": { "data": { "": "ask the participants to read the task description provided by the original authors before starting the annotation."