A Reproduction Study of an Annotation-based Human Evaluation of MT Outputs (2021)

Abstract: In this paper we report our reproduction study of the Croatian part of an annotation-based human evaluation of machine-translated user reviews (Popovic, 2020). The work was carried out as part of the ReproGen Shared Task on Reproducibility of Human Evaluation in NLG. Our aim was to repeat the original study exactly, except for using a different set of evaluators. We describe the experimental design, characterise differences between original and reproduction study, and present the results from each study, along with analysis of the similarity between them. For the six main evaluation results of Major/Minor/All Comprehension error rates and Major/Minor/All Adequacy error rates, we find that (i) 4/6 system rankings are the same in both studies, (ii) the relative differences between systems are replicated well for Major Comprehension and Adequacy (Pearson’s > 0.9), but not for the corresponding Minor error rates (Pearson’s 0.36 for Adequacy, 0.67 for Comprehension), and (iii) the individual system scores for both types of Minor error rates had a higher degree of reproducibility than the corresponding Major error rates. We also examine inter-annotator agreement and compare the annotations obtained in the original and reproduction studies.

## introduction interest in, and concern about, reproducibility is growing in natural language processing (nlp). reproducibility of human evaluations, however, has received next to no attention, and the reprogen shared task 1 on reproducibility of human evaluations in natural language generation (nlg) addresses this lack. we participated in reprogen with a contribution in track b, the reproduce your own track. more specifically, we repeated the human evaluation of a mixed set of movie and product 1 https://reprogen.github.io/ review translations produced by three leading machine translation (mt) systems, as reported by @xcite . in this paper, we summarise the original study in terms of the overall evaluation method (section 2.1), the quality criteria underlying the annotations from which evaluation scores were derived (section 2.2), the annotation process and instructions (section 2.3), and the process by which reviews were selected and translated for the evaluation (section 2.4). we then present comprehension and adequacy error rate results from the original and reproduction studies side by side, and look at how similar system rankings and individual scores are in the two studies (section 3). next we compare the inter-annotator agreement in the two studies (section 4) using diverse metrics. we discuss and interpret results obtained in our reproduction study (section 5), and draw some conclusions (section 6). ## evaluation scores columns 2-7 in the last three columns in since there are only three systems to calculate correlation on, we also calculated pearson's r between sentence-level error counts and the results are presented in ## inter-annotator agreement the original study reported inter-annotator agreement (iaa) in terms of f-score and normalised edit distance (definitions below). in this paper we also report krippendorff's α for both original and reproduction study, following @xcite who used it in a similar error marking study. 8 8 cohen's kappa was not considered appropriate for either of the studies for the reasons explained in detail in the original krippendorff's α: in order to quantify agreement by this method, error annotations were converted to a sentence-level quality score, namely the number of words marked up for error in a given sentence. for a perfect sentence, no words would be marked so this score would be zero. using the standard definition, 9 we computed three separate α scores: (i) from just the major error annotations, (ii) from just the minor error annotations, and (iii) from both (corresponding to the all subcategory from previous sections). ## discussion in previous sections, we presented results and similarities/differences observable in them in objective terms. in this section, we discuss and interpret re-sults, aiming to draw conclusions and to identify reasons for similarities and differences. ## conclusion in this paper, we reported results from a reproduction study of an annotation-based human evaluation of mt outputs where errors related to comprehensibility and meaning correctness were annotated in texts by marking up word involved in an error. we compared the corresponding comprehension and adequacy system-level error rates for the three mt systems assessed in the two studies, distinguishing subcategories all, major and minor for each. we found that 4 out of 6 system rankings were the same in both studies, but that the relative differences between systems are not well replicated for both types of minor error rates (pearson's 0.36 for adequacy-minor, 0.67 for comprehension-minor). however, the individual system scores for both types of minor error rate had a higher degree of reproducibility (as measured by the coefficient of correlation, cv), than the corresponding major error rates. results also showed that minor adequacy and major comprehension annotations and system rankings differed more than other error categories. the reproduction study reported here was a contribution to the reprogen shared task in the 'reproduce your own' track, and as such we had the benefit of having full access to all resources and information from the original evaluation, a luxury not normally available when conducting a reproduction study of someone else's work. the main difference between properties of the original study and our reproduction was the characteristics of the cohort of evaluators who had slightly different backgrounds. there were pronounced similarities between the two studies, but also very clear differences, notably including in system rankings. all in all, while repeating the study was simply a matter of recruiting a new cohort of evaluators, obtaining the same results proved somewhat less simple.