Test Set Sampling Affects System Rankings: Expanded Human Evaluation of WMT 20 E nglish- I nuktitut Systems (2022)

Abstract: We present a collection of expanded human annotations of the WMT20 English-Inuktitut machine translation shared task, covering the Nunavut Hansard portion of the dataset. Additionally, we recompute News rankings to take into account the completed set of human annotations and certain irregularities in the annotation task construction. We show the effect of these changes on the downstream task of the evaluation of automatic metrics. Finally, we demonstrate that character-level metrics correlate well with human judgments for the task of automatically evaluating translation into this polysynthetic language.

## introduction translation between inuktitut 1 and english was featured as part of the 2020 news translation task at wmt @xcite . the english-inuktitut machine translation system rankings published in @xcite were incomplete, due to the delay in the 2020 annotation campaign and because they only cover the out-of-domain portion of the test set. in this work, we present: our aim with this work is to release a broader set of human annotations, for continued research on translation between english and inuktitut, as well as to demonstrate the downstream impacts of irregularities in wmt data collection and publication. in particular, we draw attention to how errors in human annotation setup (not errors in the work of the annotators, but errors in the way that organizers constructed the annotation tasks) and the failure to account for their effects impact both the validity of the rankings themselves and the shared task on automatic metrics which relies on the rankings. in this way, the 2020 wmt task on inuktitut serves as a case study of a broader issue in the field. we provide suggestions for how to account for irregularities in existing annotation task construction and release the code and data to replicate our results. ## data the test set for the wmt 2020 english-inuktitut shared task consisted of data from the nunavut hansard as well as from nunatsiaq news, collected and shared with permission. the news test data consisted of documents collected from nunatsiaq news between september and november, 2020, as was standard for the shared task. the hansard data in the test set, however, was collected from earlier dates. the main training data available for building constrained systems was the nunavut hansard 3.0 h-a h-b n-1 n-2 n-c total a --2800 --2800 b ---3200 -3200 c -3801 -200 -4001 d 5600 4000 4600 2000 -16200 e --600 6602 -7202 f 2403 ----2403 g 3401 ----3401 un. ----9728 9728 the annotators who did the work of human evaluation discussed here were fluent language experts at the pirurvik centre, 3 paid at professional rates. all the human annotations were collected in the segment rating with document context (sr+dc) style of direct assessment (da; @xcite @xcite using the appraise interface (see @xcite for additional interface details). for each segment, annotators viewed the source sentence and a candidate translation, and scored the translation on a pseudo-continuous sliding scale from 0-100. these segments were displayed in document context, and annotators also provided document scores (we omit those from this work). each news story had a unique document id, but the hansard data was treated as a single document containing 1566 lines. this had two main consequences with respect to human annotation of system outputs. the first and most obvious is that the annotations collected at wmt (on which the findings paper's rankings @xcite were at least partially based) only included annotations of the news data (out-of-domain). the reason for this is that the code used to generate sessions of annotations in the sr+dc da human annotation task structures pooled all documents and then sampled documents "at random (without replacement) and assigned @xcite to the current hit [human intelligence task] until the current @xcite no more than 70 segments in total"; since the hansard data was treated as one document with more than 3 https://www.pirurvik.ca/ 70 segments, it was never sampled. 4 the second is that the news documents were longer on average than news documents for other language pairs. english-inuktitut news documents ranged from 12 to 137 lines in length, with a mean of 39.0 (standard deviation 20.5) and a median of 36. documents for other language pairs that were evaluated in the sr+dc format ranged from 2 to 32 lines in length, with a mean of 12.0 lines (standard deviation 6.2) and a median of 11. as a consequence, each annotation session of english-inuktitut news data is less likely to contain documents translated by the full set of submitted systems (12 submitted systems and human reference), which has the potential to cause problems when scores are normalized per-annotation session @xcite . 5 additionally, a portion of the source and reference data news segments contained spurious quotation marks (see appendix b). we now discuss the news and hansard annotation processes. ## approaches to rankings in this work, we will generate two sets of rankings: system rankings over the hansard data and system rankings over the news data. while there would be reason to desire a single ranking that covers both in-domain (hansard) and out-of-domain (news), that would raise the question of how to balance the two, and would also be a challenge to produce given the differences in the data collection processes. having two rankings also highlights differences in performance across those domains. the hansard rankings come from the annotations that will be released alongside this paper, while the news rankings are a reranking based on the data collected at the wmt shared task. here we discuss how the rankings computed for this paper differ from those produced at wmt. a partial description of the wmt rankings can be found in @xcite . the main issues we try to address in our new rankings are those raised in @xcite around the instability of rankings, particularly when the annotation sessions contain distributional issues that make the usual z-score computation inappropriate. we attempt to handle these issues both in proactive ways (through modifications to the "document" lengths and system distributions in the setup of the annotation of hansard data) and in reparative ways (when we make use of the existing wmt news annotations). ## rankings we observe several similarities and differences between the hansard ( the system that shows the smallest gap in raw scores between hansard and news and the greatest improvement in clustering, moving from the fifth cluster for hansard to the second for news was ubiqus, which saw a difference of just 1.8. in comparison, the systems with the greatest drops in rankings (uedin from third cluster to sixth, and groningen from one to four) saw raw score average drops between 14.7 and 20.6. the oppo and nrc systems also saw large drops in raw average scores (18.8 and 17.2, respectively) but with smaller or no corresponding ranking drops (in the case of oppo, it was ranked last in hansard so no drop was possible). ## automated mt evaluation another important area of research on english-inuktitut machine translation is accurate automated mt evaluation metrics for a polysynthetic language. language model based metrics usually correlate better with human judgments when evaluating translation in non-polysynthetic languages but they suffer from a training resource scarcity problem when evaluating polysynthetic languages. character based metrics are more commonly used for evaluating translation in low resource and polysynthetic languages @xcite but there is not enough study on their correlation with human judgments. a complete collection of human annotations on both domains of the english-inuktitut test set with translation output from diverse mt systems enables further studies on automated mt evaluation metrics, with the caveat that caution should be taken with news, due to the issues in the data collection described above and in appendix b. ## conclusion in this work we present additional human annotations for the hansard portion of the wmt 2020 english-inuktitut machine translation shared task test set. we provide new system rankings on this portion of the data and present revised rankings on the news portion. we demonstrate that these changes in rankings have downstream effects on the evaluation of automatic metrics for the shared task, and examine the difficulty of performing automatic evaluation on out-of-domain text in a polysynthetic language. when it comes to automatic metrics, we find that the top-performing system incorporated training data in the low-resource target language. however, character-level automatic metrics (which did not require training) also performed amongst the top systems, demonstrating their appropriateness for evaluating translation into inuktitut. while additional research will be required to confirm that this finding generalizes to other polysynthetic languages, we release this expanded dataset to enable more study of automatic metrics for low-resource and polysynthetic languages.