H um E val’23 Reproduction Report for Paper 0040: Human Evaluation of Automatically Detected Over- and Undertranslations (2023)

Abstract: This report describes a reproduction of a human evaluation study evaluating automatically detected over- and undertranslations obtained using neural machine translation approaches. While the scope of the original study is much broader, a human evaluation is included as part of its system evaluation. We attempt an exact reproduction of this human evaluation, pertaining to translations on the the English-German language pair. While encountering minor logistical challenges, with all the source material being publicly available and some additional instructions provided by the original authors, we were able to reproduce the original experiment with only minor differences in the results.

## introduction this report presents a reproduction of a human evaluation originally conducted and presented in the paper as little as possible, as much as necessary: detecting over-and undertranslations with contrastive conditioning @xcite . the paper proposes an approach for detecting overand under-translations using contrastive conditioning @xcite , a method that relies on hypothetical reasoning over the likelihood of partial sequences and thus has the advantage of not requiring access to the original translation system or to a quality estimation model. the authors evaluate their system based on real machine translations and show that the approach outperforms a supervised baseline in the detection of omissions. while the scope of their original study is much broader, a human evaluation is included as part of the system evaluation and is described in section 5.2 of their paper. in this evaluation step, the original authors employ expert annotators to determine whether the spans of text that their system predicts as mistranslated are indeed under-or overtranslations, and do this on the english-german and english-chinese language pairs. in our reproduction study, we attempt to reproduce the evaluations of the english-german data, by employing expert annotators to evaluate the same data samples. this reproduction study was conducted as part of the reprohum project foot_0 @xcite , the aim of which is to build on existing work on recording properties of human evaluations datasheet-style @xcite , and assessing how close results from a reproduction study are to the original study @xcite , to systematically investigate what factors make human evaluations more-or less-reproducible. our choice to reproduce this particular paper is motivated by our previous experience in related fields: both authors have previously worked in the space of machine translation @xcite @xcite @xcite @xcite , have a track record of interest in human evaluation @xcite @xcite and reproducibility @xcite , and are thus well-positioned to conduct this reproduction experiment. ## original study design for the english-german language pair, the original study employed two linguistic experts as evaluators. as their annotation interface, the authors opted for doccano 2 (nakayama et al., 2018), an open-source text annotation tool which provides annotation features for text classification, sequence labeling, and sequence to sequence tasks. each expert evaluator was shown 80+720 (dev+test set) randomly sampled positive predictions across both types of coverage errors. evaluators were shown the source sequence, the machine translation, and the predicted error span. they were asked whether the highlighted span was indeed translated badly, and were asked to perform a fine-grained analysis based on a list of predefined answer options (see appendix a). a subset of the samples (100 sentences) was annotated by both raters in order to calculate inter-annotator agreement. the authors made all predictions, annotations and notebooks used for calculating the precision values available in the github repository 3 . ## reproduction study details we used the exact same dataset as provided by @xcite and had each annotator annotate the same set of instances as provided by the original authors. once we obtained the evaluations, we used the original authors' evaluation script, as provided on their github page. it is worth noting that during the reproduction phase, another team reproducing the same experiment noticed a possible bug in the authors' results processing script. after communication via the reprohum team, the issues were clarified and corrected, and the authors uploaded a revised script to fix one of the errors that arose. the updated script is also included in their github page and is the one we used for result processing 4 . ## reproduction results for the human evaluation aspect, the original paper reports three sets of results: (a) a table containing word-level precision scores of the spans that were highlighted by their automatic approach, based on the human evaluations ( above results (a) and (b) fall under type i results as defined in the reprohum reproduction guidelines, given that they are numerical error counts or precision calculations. results (c) fall under type iii, as they are multi-rater categorical labels attached to text spans. it should be noted that regarding (a), the original paper does not seem to mention how these precision values are calculated, nor does such a calculation seem to be included in the authors' annotation processing script or reproducibility guidelines, making these results difficult to reproduce without relying on guesswork. regarding (b), while the plots presented in the paper are indicative of general trends, precise error counts are difficult to infer from the graphics alone. fortunately the authors do provide the full annotated data and the exact output of the calculations as part of the notebook on their github page. the same notebook also includes a calculation for (c), making both (b) and (c) straightforward to reproduce. one could argue that the error counts and the cohen's kappa are the core reproduction values, as they constitute the raw outputs of the human evaluation. tables 1 and 2 show the original values provided by @xcite and our reproduced values side by side. it is worth noting that the original values were not provided in the paper itself, but rather in supplementary material, specifically the notebook on the original author's github page (which is still publicly accessible, but requires some digging to acquire the data). ## conclusion while we were not able to reproduce the core findings on model precision due to lack of information, we did manage to achieve similar cohen's kappa scores for our annotator agreement on one question, and a somewhat higher score on the more difficult question. we also reproduced the distribution of labels on question 1 and on most categories in question 2. at the adapt sfi research centre at technological university dublin and maynooth university. adapt, the sfi research centre for ai-driven digital content technology, is funded by science foundation ireland through the sfi research centres programme, and is co-funded under the european regional development fund. references petra bago, sheila castilho, edoardo celeste, jane dunne, federico gaspari, niels runar gislason, andre kasen, filip klubička, gauti kristmannsson, helen mchugh, roisin moran, orla ni loinsigh, jon arild olsen, carla parra escartin, akshai ramesh, natalia resende, paraic sheridan, and andy way. 2022. sharing high-quality language resources in the legal domain to develop neural machine translation for under-resourced languages. revista de llengua i dret, (78):9-34. anya belz, maja popovic, and simon mille. 2022. quantified reproducibility assessment of nlp results. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 16-28, dublin, ireland. association for computational linguistics. anya belz, craig thomson, and ehud reiter. 2023. missing information, unresponsive authors, experimental flaws: the impossibility of assessing the reproducibility of previous human evaluations in nlp. in the fourth workshop on insights from negative results in nlp, pages 1-10, dubrovnik, croatia. association for computational linguistics. filip klubička, lorena kasunić, danijel blazsetin, and petra bago. 2022. challenges of building domainspecific parallel corpora from public administration documents. in proceedings of the bucc workshop within lrec 2022, pages 50-55, marseille, france. european language resources association. filip klubička, giancarlo d. salton, and john d. kelleher. 2018a. is it worth it? budget-related evaluation metrics for model selection. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). filip klubička, antonio toral, and víctor m sánchez-cartagena. 2018b. quantitative fine-grained human evaluation of machine translation systems: a case study on english to croatian. machine translation, 32(3):195-215. filip klubička, antonio toral ruiz, and m. víctor sánchez-cartagena. 2017. fine-grained human evaluation of neural versus phrase-based machine translation. the prague bulletin of mathematical linguistics, 108(1):121-132. filip klubička and raquel fernández. 2018. examining a hate speech corpus for hate speech detection and popularity prediction. in proceedings of 4real: 1st workshop on replicability and reproducibility of research results in science and technology of language. yasmin moslem, rejwanul haque, john kelleher, and andy way. 2022. domain-specific text generation for machine translation. in proceedings of the 15th biennial conference of the association for machine translation in the americas (volume 1: research track), pages 14-30, orlando, usa. association for machine translation in the americas. yasmin moslem, rejwanul haque, john d. kelleher, and andy way. 2023. adaptive machine translation with large language models. arxiv preprint arxiv:2301.13294. hiroki nakayama, takahiro kubo, junya kamura, yasufumi taniguchi, and xu liang. 2018. doccano: text annotation tool for human. software available from https://github.com/doccano/doccano . maja popović, mihael arčan, and filip klubička. 2016. language related issues for machine translation between closely related south slavic languages. in proceedings of the third workshop on nlp for similar languages, varieties and dialects (vardial3), pages 43-52, osaka, japan. the coling 2016 organizing committee. maja popovic, vasudevan nedumpozhimana, meegan gower, sneha rautmare, nishtha jain, and john kelleher. 2023. using mt for multilingual covid-19 case load prediction from social media texts. european association for machine translation. giancarlo salton, robert ross, and john kelleher. 2014a. an empirical study of the impact of idioms on phrase based statistical machine translation of english to brazilian-portuguese. in proceedings of the 3rd workshop on hybrid approaches to machine translation (hytra), pages 36-41, gothenburg, sweden. association for computational linguistics. giancarlo salton, robert ross, and john kelleher. 2014b. evaluation of a substitution method for idiom transformation in statistical machine translation. in proceedings of the 10th workshop on multiword expressions (mwe), pages 38-42, gothenburg, sweden. association for computational linguistics. anastasia shimorina and anya belz. 2022. the human evaluation datasheet: a template for recording details of human evaluation experiments in nlp. in proceedings of the 2nd workshop on human evaluation of nlp systems (humeval), pages 54-75, dublin, ireland. association for computational linguistics. antonio toral, miquel esplá-gomis, filip klubička, nikola ljubešić, vassilis papavassiliou, prokopis prokopidis, raphael rubino, and andy way. 2017. crawl and crowd to bring machine translation to under-resourced languages. language resources and evaluation, 51:1019-1051. jannis vamvas and rico sennrich. 2021. contrastive conditioning for assessing disambiguation in mt: a case study of distilled bias. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 10246-10265, online and punta cana, dominican republic. association for computational linguistics. jannis vamvas and rico sennrich. 2022. as little as possible, as much as necessary: detecting over-and undertranslations with contrastive conditioning. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 2: short papers), pages 490-500, dublin, ireland. association for computational linguistics.