With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector (2023)

Abstract: This work presents our efforts to reproduce the results of the human evaluation experiment presented in the paper of Vamvas and Sennrich (2022), which evaluated an automatic system detecting over- and undertranslations (translations containing more or less information than the original) in machine translation (MT) outputs. Despite the high quality of the documentation and code provided by the authors, we discuss some problems we found in reproducing the exact experimental setup and offer recommendations for improving reproducibility. Our replicated results generally confirm the conclusions of the original study, but in some cases statistically significant differences were observed, suggesting a high variability of human annotation.

## introduction reproducibility of experimental results is a fundamental principle of scientific research that ensures the validity, credibility, and reliability of scientific findings. the nlp research community is increasingly interested in reproducibility, which leads to the organization of shared tasks @xcite @xcite , the formulation of reproducibility guidelines @xcite , and so on. however, most previous efforts are limited to the reproducibility of automatic measures, and reproducibility of human evaluation has received less attention @xcite . the reprohum project, 1 which this paper is a part of, aims to improve this situation. in this paper, we describe our efforts to reproduce the results of the human evaluation experiment conducted in @xcite to evaluate the performance of their over-and undertranslation detection method for machine translation (mt). more specifically, the method detects 1 https://reprohum.github.io/ phrases in the source texts whose meaning is not reflected in an mt output, or phrases in the mt output that are not supported by the source (see details in sec. 2). the human annotators evaluated the detection accuracy and provided additional reasons for their evaluation by choosing from a list (see figure 1 ). the original experiment was run for english-chinese and english-german translation pairs, but our reproducibility study is limited to the english-german pair due to the availability of skilled human annotators. despite the precise description of the experiment in the original paper, we still encountered difficulties in running the study (see section 4), and were only able to finish it successfully thanks to the strong support of the original authors. our results overall support the main claims of @xcite 's original paper, but we still found some discrepancies, despite using the same data, interface and guidelines for the annotation (see section 5). our collected annotation outputs, reproduction code, and the filled heds sheet @xcite for the reproduction study are available on github. foot_0 ## original experiment the original paper @xcite proposed an automatic method for detecting coverage errors in the output of mt systems. coverage errors include undertranslations, i.e. the omission of important source content in the mt system output, and overtranslations, i.e. the addition of superfluous words to the translation that may not be supported by the source. the method uses contrastive conditioning (vamvas and sennrich, 2021) and finds coverage errors by iteratively computing the probability of the generated translation with an mt system conditioned on an incomplete text source. if the probability of the generated translation increases when a particular phrase is deleted from the source, the method takes this as an indicator that the deleted phrase is not adequately reflected in the translation and treats it as an omission (undertranslation). similarly, by reversing the source with the target, the method also detects overtranslations. to summarize, for a given input-translation pair, the output of the method is the type of problem detected (overor undertranslation) and a phrase that has been omitted from the translation (in the case of undertranslation), or that is superfluous (in the case of overtranslation). the corresponding human evaluation aimed to analyze in detail the predicted problematic text spans and assess their correctness. human annotators were presented with a source sentence, the generated mt translation, and a highlighted passage. the annotator's task was to decide whether the highlighted passage was correctly translated and later to select additional feedback for fine-grained analysis from a given single-choice list. if the annotator confirmed that the highlighted span was incorrectly translated, they were asked to specify the type of error (e.g., fluency error, accuracy error, addition/omission of non-trivial information, etc). on the other hand, if the annotator considered the span to be correctly translated, they were asked to give a possible reason why one could think that it was translated incorrectly (e.g., syntactic differences, adding/removing trivial/obvious information). the full list of possible reasons can be seen in figure 1 . the manual evaluation was carried out by two linguists who were provided with a two-page document containing annotation guidelines. the guidelines included the task description, instructions on using the annotation interface, and examples of three incorrect and three correct translations. each annotator responded to approximately 700 randomly selected examples. ## differences in our reproduction study we aimed to conduct the reproduction as close as possible to the original study. we worked on the same set of system outputs, with the identical annotation interface and instructions. 3 however, there were some differences with respect to annotator hiring and to splitting the annotation between them. ## implementation issues we found two implementation problems while running the study: one with setting up the annotation interface and the other with the script computing final statistics. ## results a side-by-side presentation of all the results can be found in in both experiments, each annotator provided feedback on approximately 700 examples, which were similarly divided into examples of over-and under-translation. the inter-annotator agreement, measured with cohen's κ, was higher in our reproduction for the simple evaluation of the correctness of highlighted spans, but the results were within the constructed 95% confidence interval for the original value. according to @xcite , these κ values should be interpreted as weak/moderate agreement. on the other hand, the inter-annotator agreement for fine-grained responses was statistically significantly lower in our repeated study, but again, both values from the original and the repeated study should be interpreted as "minimal agreement" @xcite . in the original paper, the results of the experiments are reported in terms of word-level precision of the highlighted spans. the comparison of the original precision values with those obtained in our reproduction can be observed in a 95% confidence interval using wilson's score method @xcite . the precision values obtained in our reproduction are generally lower than those reported in the original study. however, the differences in precision for the overtranslation spans are still within the confidence intervals. in contrast, the differences for under-translation are substantial, as the reproduced precision values are about 44-46% lower. this difference is also statistically significant at the significance level α = 5%. to compare the results of the fine-grained analysis, we performed χ 2 goodness-of-fit tests between the answers provided by the original annotators and those provided by us, as well as calculating cramér's v for goodness-of-fit. the results are presented in the differences are visible even to the naked eye, as our annotators selected unknown reasons for highlighting a correctly translated text span about four times more often than in the original study. in the case of undertranslations, the annotators in the original study chose much more often that the translation is incorrect, but the missing information can be inferred or is trivial (107 vs. 7 counts). on the other hand, our annotators were much more likely to consider that the translation was correct but could be considered inaccurate because some trivial or easily inferable information was missing (25 vs. 103 counts). our annotators also found more spans highlighted as under-translations as reasons for accuracy or fluency errors in the trans-lation. these larger differences for examples of undertranslation may also indicate that this variant of the evaluation task is more difficult. note that the annotators have a highlighted text span in the source text, but still have to answer questions about the final translation without any word/phrase alignment information. ## quantifying reproducibility following the guidelines of the reprohum shared task @xcite , sect. a5), we identify reproduction targets in the following categories: the precision of text spans labeled as over-/undertranslations to truly contain over-/undertranslation errors the precision of text spans labeled as over-/undertranslations to contain some translation errors the set of precision results for examples marked as overtranslations the set of precision results for examples marked as undertranslations type ii results are usually evaluated with pearson's correlation @xcite , but there is little point in calculating it here. it is equal to 1 for both over-and under-translations, while the standard statistical test for correlation fails to reject the null hypothesis that the true correlation is 0. type iii finally, the reproducibility of categorical labels was assessed with krippendorff's alpha. since the aim of this analysis is not to measure the agreement between all four annotators (2 from the original study and 2 from the replication) but rather to measure the reproducibility, for the purpose of computing krippendorff's alpha the annotations obtained from each pair of annotators were combined into one set. for the overlapping examples i.e. examples annotated by both annotators, the response of a randomly chosen annotator was retained. the values of krippendorff's alpha together with the percentage of identical evaluations in both the original and reproduced study for coarse-grained annotations (i.e. correct/incorrect translation label) are given in tab. 4. the value for overtranslation is significantly higher than for undertranslations and is above the 2 /3 threshold suggested by @xcite as the lowest limit to consider a good agreement between the raters. the high agreement between the original annotation and the reproduced one can also be observed in terms of the proportion of identical answers -almost 85% for the whole dataset. similar results for fine-grained annotations are provided in tab. 5. all reported values of krippendorff's alpha are relatively low, and the proportion of examples that are evaluated identically in both studies is below 51%. however, the ratio for overtranslation spans marked as correct translations is significantly higher than the same ratio for undertranslations (roughly 16 percentage points). ## findings based on the manual evaluation, the authors of the original paper present several findings/conclusions: all of the above findings roughly correspond to the results of our reproduced experiment. the precision for undertranslations is also higher than for overtranslations, but the difference between the two was considerably smaller in our experiment. for example, the difference in precision for true coverage errors is 34.0 percentage points in the original study but only 12.3 in ours. similarly, our fine-grained analysis confirms that syntactic differences contribute to false positives, but they are reported about 40% less frequently than in the original study. however, as mentioned above, this is partly due to the fact that our annotators were much more likely to select the "unknown reason" response. ## conclusion we carefully repeated a human evaluation study of paper @xcite . despite the high-quality documentation and well-organized code provided by the authors, we encountered several problems that were difficult to overcome. in particular, we would not have been able to run the annotation interface and repeat the study without the authors' help. we also noticed a minor error in their evaluation script and suggested a modification. the reproduction process made us realise that it is almost impossible to publish a fully reproducible paper without actually trying to reproduce it endto-end. we also advocate distributing annotation interfaces in the form of a docker image containing all the dependencies. our overall results agree with the high-level conclusions made in the original paper. the reproduction of results of the coarse-grained analysis was successful for overtranslations, but the results for undertranslations were significantly lower than the reported in the original study. the results of the fine-grained analysis were even more difficult to reproduce -we observed significant differences in all analyzed groups of answers. this may suggest that when designing experiments with human judgments, setups with a very limited number of possible answers (especially binary questions) are easier to replicate and should be prioritized over more complex setups whenever possible.