Findings of the WMT 2022 Shared Task on Automatic Post-Editing (2022)

Abstract: We present the results from the 8th round of the WMT shared task on MT Automatic PostEditing, which consists in automatically correcting the output of a “black-box” machine translation system by learning from human corrections. This year, the task focused on a new language pair (English→Marathi) and on data coming from multiple domains (healthcare, tourism, and general/news). Although according to several indicators this round was of medium-high difficulty compared to the past,the best submission from the three participating teams managed to significantly improve (with an error reduction of 3.49 TER points) the original translations produced by a generic neural MT system.

## introduction this paper presents the results of the 8 th round of the wmt task on mt automatic post-editing (ape). the task consists in automatically correcting the output of a "black-box" machine translation system by learning from human-revised machinetranslated output supplied as training material. the overall task formulation (see section 2) remained the same as in all previous rounds, where the challenge consisted in fixing the errors present in english documents automatically translated by stateof-the-art, not domain-adapted neural mt (nmt) systems unknown to participants. however, two main factors of novelty characterized the ape 2022 evaluation setting: this year, we had three teams submitting a total of five systems for final evaluation (see section 5). while the difficulty (section 4) of this round falls in a medium-high range attested by relatively high baseline results on the test data (20.28 ter / 67.55 bleu), final results indicate the overall good quality of the submitted runs. two teams were indeed able to significantly improve over the baseline in terms of the official automatic evaluation metrics (section 6). in particular, according to the primary metric (i.e., the ter score computed between automatic and human post-edits), the top-ranked system (16.79 ter / 72.92 bleu) achieved an error reduction of 3.49 ter points. also, this year, the standard automatic evaluation was complemented by a human evaluation based on direct assessment. however, some problems in the procedure foot_0 were later discovered, which make it unreliable to draw insights except for the confirmation that two of the three submitted systems were able to improve over the baseline significantly. specifically, both of them achieved a mean direct assessment score that drastically reduces the gap between the baseline and human post-editing quality. however, due to the mentioned problems in the human evaluation procedure, further details about it will not be included in the discussion below. although the different language/domain testing conditions prevent from drawing precise conclusions about the progress of ape technology with respect to last year, the overall positive results confirm its viability for downstream improvements of "black-box" mt systems whose inner workings are not accessible. ## task description mt automatic post-editing (ape) is the task of automatically correcting errors in a machinetranslated text. as pointed out by @xcite , from the application point of view, the task is motivated by its possible uses to: this 8 th round of the wmt ape shared task kept the same overall evaluation setting of the previous seven rounds. specifically, the participating systems had to automatically correct the output of an unknown "black-box" mt system (a generic nmt system not adapted to the target domain) by learning from training data containing human revisions of translations produced by the same system. the selected language pair and the data domain, however, were totally new to the task. different from previous rounds covering more language pairs (or directions), this year focused only on english-marathi, presenting participants with the traditional source language and, for the third time in a row, an eastern language as the target. moreover, while the training, development and test data released in previous rounds were always drawn from a single domain, this year, they covered three domains: healthcare, tourism, and general/news. ## complexity indicators to get an idea of the difficulty of the task, in previous rounds, we focused on three aspects of the released data, which provided us with information about the possibility of learning useful correction patterns during training and successfully applying them at test time. these are: i) repetition rate, ii) mt quality, and iii) ter distribution in the test set. for the sake of comparison across the eight rounds of the ape task (2015-2022), ## submissions as shown in ## conclusion the 8 th round of the shared task on automatic post-editing at wmt was characterized by two main factors of novelty: the language pair (english-marathi) and the domain of the released data (a mix covering healthcare, tourism, and general/news). apart from this, the overall setting was the same as in previous recent rounds, in which participating systems had to automatically correct the output of a generic neural mt system, being evaluated with the ter (primary) and bleu (secondary) automatic metrics. in continuity with the past, also human evaluation via source-based direct assessment was carried out, but it is not discussed in this report due to its unreliable outcomes. in terms of the three complexity indicators discussed in section 4 (repetition rate, original mt quality and ter distribution), the difficulty of this round falls in a medium-high range. this is reflected by the performance of the systems submitted by the three participating teams: two of them were indeed able to improve over the do-nothing baseline with (statistically significant) error reductions up to -3.49 ter points (+5.37 bleu). although these results are not comparable with those from previous years due to the different language/domain testing conditions, the observed improvements in the new language direction confirm the viability of ape for downstream improvements of "black-box" mt systems whose inner workings are not accessible.