Findings of the WMT 2020 Shared Task on Automatic Post-Editing (2020)

Abstract: We present the results of the 6th round of the WMT task on MT Automatic Post-Editing. The task consists in automatically correcting the output of a “black-box” machine translation system by learning from existing human corrections of different sentences. This year, the challenge consisted of fixing the errors present in English Wikipedia pages translated into German and Chinese by state-ofthe-art, not domain-adapted neural MT (NMT) systems unknown to participants. Six teams participated in the English-German task, submitting a total of 11 runs. Two teams participated in the English-Chinese task submitting 2 runs each. Due to i) the different source/domain of data compared to the past (Wikipedia vs Information Technology), ii) the different quality of the initial translations to be corrected and iii) the introduction of a new language pair (English-Chinese), this year’s results are not directly comparable with last year’s round. However, on both language directions, participants’ submissions show considerable improvements over the baseline results. On English-German, the top ranked system improves over the baseline by -11.35 TER and +16.68 BLEU points, while on EnglishChinese the improvements are respectively up to -12.13 TER and +14.57 BLEU points. Overall, coherent gains are also highlighted by the outcomes of human evaluation, which confirms the effectiveness of APE to improve MT quality, especially in the new generic domain selected for this year’s round.

## introduction mt automatic post-editing (ape) is the task of automatically correcting errors in a machinetranslated text. as pointed out by @xcite , from the application point of view, the task is motivated by its possible uses to: in its 6 th round, the ape shared task organized within the wmt conference on machine translation kept the same overall evaluation setting of the previous five rounds. specifically, the participating systems had to automatically correct the output of an unknown "black box" (neural) mt system by learning from training data containing human revisions of translations produced by the same system. this year, the task focused on two language pairs: english-german and english-chinese. the former has been part of the ape evaluation campaigns since 2016 @xcite , while the latter represents a new entry. a second difference with respect to previous rounds is that, for both language pairs, the source/domain of the data changed from information technology (it) to wikipedia articles. the third major novelty factor consists in the type of mt systems used to generate the translations to be corrected. although for the third year in a row the task focused on translations produced by neural mt (nmt) systems, this year these models were not adapted to the target domain. these radical changes have advantages and disadvantages. on one side, moving away from the "narrow" it domain allowed to test ape technology on the challenging scenario represented by the generic domain of wikipedia articles. indeed, as shown in the previous rounds of the task @xcite , the high level of repetitiveness of it data makes this domain easier to model compared to a generic and less repetitive domain, both for mt and ape technology. moreover, fixing the output of generic nmt models that are not domain-adapted allowed to test ape on lower-quality initial data and verify its potential as a downstream domain adaptation component. on the other side, the disadvantage of changing domain is the reduced possibility to compare results and measure progress across years. specifically, the lower quality of the original sentences to be corrected (and, in turn, the larger room for improvement left to ape) make the participants' results and the overall technology advancements difficult to analyze in the light of previous rounds. six teams participated in the english-german task, submitting eleven runs in total. two teams participated in the english-chinese task, submitting two runs each. similar to last year, all teams developed their systems based on neural technology, which confirms to be the state-of-the-art approach to ape. in most of the cases (see section 3), participants experimented with the transformer architecture @xcite , either directly or by adapting it to the task. as in previous rounds, their systems exploit information both from the mt output to be corrected and from the corresponding source sentence. this was done either by concatenating the two, as in last year's winning system @xcite , or by means of multi-source solutions @xcite successfully explored in the past @xcite . following the recent trends in other nlp areas, the integration of pre-trained bert-like language models was also considered. model ensembling and the integration of word/sentence-level quality estimation techniques geared to ape (similar to @xcite ) were also explored. finally, also this year participants took advantage of data augmentation techniques, either by creating their own escape-like corpora @xcite , or by generating synthetic data by adding artificial noise to simulate post-editing errors, or by exploiting external mt candidates as a source of auxiliary information to be concatenated to the input. the overall evaluation results show significant improvements over the baseline on both the language directions. on english-german, where the "do-nothing" baseline (see section 2.3) was 31.56 ter @xcite and 50.21 bleu @xcite , the top-ranked system (20.21 ter, 66.89 bleu) shows an impressive -11.35 ter reduction, which corresponds to a +16.68 gain in terms of bleu score. considering all the submissions, the average gain is -4.89 ter and +6.5 bleu points, with only one system performing slightly worse than the baseline. different from last year, where the differences between the top four submissions were not statistically significant, this year we have a clear winner, whose best submission is 6.78 ter points (and 11.12 bleu points) above the second ranked team. nevertheless, though possibly favoured by the relatively low baseline results (+14.72 ter and -24.52 bleu compared to last year), the globally good performance of the participants is a good indicator of overall progress. the newly proposed english-chinese task is no exception. here, both participating teams were able to outperform the baseline (59.49 ter and 23.12 bleu) by a significant margin. the largest gains are up to -12.13 ter and +14.57 bleu points and, on average for the four submitted runs, they are -8.15 ter and +10.1 bleu points. the good results observed with automatic metrics on both the language pairs are confirmed by the human evaluation outcomes. on english-german, for the first time, the top-ranked primary submission is not significantly worse compared to the human post-edited output (suggesting that automatic corrections are indistinguishable from the human ones foot_0 ). all the other systems except one, moreover, are significantly better than the baseline. this also happens for the two primary submissions to the english-chinese subtask which, however, are both significantly worse than human post-edits. all in all, the improvements observed on both the language pairs can be most likely ascribed to the lower quality of the initial translations to be corrected. on english-german, the baseline (31.56 ter, 50.21 bleu) was indeed much lower than in the past, when the mt systems used were always domain-adapted and hence more competitive. last year, for instance, the baseline was 16.84 ter (74.73 bleu), while in none of the previous rounds focusing on this language pair participants had to confront with ter above 25.0 and bleu below 62.0. on english-chinese, the baseline was even lower (59.49 ter, 23.12 bleu), with the lowest scores across all the past six editions of the ape task. on one side, the large gains observed are in line with (and indirectly confirm) previous observations @xcite @xcite about the difficulty to improve highquality mt output. conversely, as we can observe this year, translations of lower quality (like those coming from generic, not domain-adapted models) leave to ape technology a large margin for improvement. on the other side, the observed global gains in both settings motivate further research on ape as a tool for downstream mt adaptation in black-box conditions. ## task description in continuity with all the previous rounds of the ape task, participants were provided with training and development data consisting of (source, target, human post-edit) triplets, and were asked to return automatic post-edits for a test set of unseen (source, target) pairs. ## participants six teams submitted a total of 11 runs for the english-german subtask. two of them participated also in the english-chinese subtask by submitting 2 runs each. participants are listed in alibaba group (mind). alibaba participated only in the english-german subtask. their submission introduces a cross-lingual bert-like conditional model with a "memory-encoder", which can capture the semantic information of machine translations conditional on the source sentences @xcite . the system consists of three parts, namely: i) a general transformer encoder to encode the source sentences, ii) a transformer decoder without future mask adapted to a memoryencoder to encode machine translations with cross attention to the source encoder, and iii) a multisource transformer decoder to generate the automatic post-editing results with cross attentions to both the encoders. in addition, data augmentation, corpus filtering and imitation learning strategies are exploited to overcome the scarcity of real ape data and to further improve model's performance, together with model ensembling and conservativeness penalty strategies inspired by @xcite . 7 in addition to the do-nothing baseline, in the first three rounds of the task we also compared systems' performance with a re-implementation of the phrase-based approach firstly proposed by @xcite , which represented the common backbone of ape systems before the spread of neural solutions. as shown in @xcite , the steady progress of neural ape technology has made the phrasebased solution not competitive with current methods reducing the importance of having it as an additional term of comparison. since 2018, we hence opted for considering only one baseline. bering lab (berlinglab). bering lab participated only in the english-german subtask. their system relies on a transformer architecture, in which the encoder takes in input a concatenation of the source and mt sentences to generate a cross-lingual representation to be passed to the decoder. additionally, they explored methods to improve ape performance through word-level and sentence-level quality estimation (qe). based on word-level qe, they mask incorrect or missing words in the pe output. then, the most probable word for each masked token is predicted using xlm-roberta @xcite , which is fine-tuned based on the translation language modeling (tlm) objective @xcite . finally, they propose an output selection mechanism based on sentence-level qe to prevent over-correction. to this aim, they select the sentence with the lowest predicted hter among the pe outputs and the original mt sentence as the final output. for data augmentation, they use a parallel corpus to train an nmt model and generate artificial triplets, following the ideas from @xcite . huawei (hw-tsc). huawei participated both in the english-german and english-chinese subtasks. their system basically follows the architecture of last year's winning system @xcite , where src and mt sentences are concatenated as input to the encoder, and the decoder is used for decoding the pe sentence. however, there are several differences with respect to @xcite . first, instead of using a pretrained bert model, the system relies on a transformer nmt model (implemented with fairseq @xcite ), pre-trained on the wmt19 news translation corpora. second, the model integrates bottleneck adapter layers to prevent from over-fitting. third, external mt candidates (from google translate) are exploited as a source of auxiliary information. this results in a longer input sequence composed of (src, mt, auxiliary mt) triplets. due to the domain change introduced this year, system's training does not exploit the supplied additional corpora for data augmentation. finally, the system does not include methods to prevent over-correction, such as the penalty mentioned in @xcite . ## results participants' results are shown in tables 3 (english-german) and 4 (english-chinese). the submitted runs are ranked based on the average ter (case-sensitive) computed using human postedits of the mt segments as reference, which is the ape task primary evaluation metric. the two tables also report the bleu score computed using human post-edits, which represents our secondary evaluation metric. similar to last year, also in this round the primary and secondary evaluation metric produce rankings that are only slightly different from each other. 9 in spite of these minor difference, for 9 for english-german, the third and fourth-ranked submissions in terms of ter are switched in terms of bleu, both both languages we have a clear separation between the two top-ranked submissions (by the same team) and the other submitted runs. on english-german, the best results (20.21 ter, 66.89 bleu) respectively outperform the baseline by -11.35 ter and +16.68 bleu points, the second-best scores being lower by less than 1 point for both the metrics. all the other runs but the last are quite close to each other, being concentrated respectively in a 1.42 ter and 1.55 bleu points interval. on english-chinese, the best results (47.36 ter, 37.69 bleu) respectively outperform the baseline by -12.13 ter and +14.57 bleu points. also in this case, the second-best run is below the top-ranked one by less than 1 point for both the metrics, while the third and fourth submissions are close to each other (the difference is less than 0.2 points for both metrics). all in all, these results indicate that: in ## system/performance analysis as a complement to global ter/bleu scores, also this year we performed a more fine-grained analysis of the changes made by each system to the test instances. ## human evaluation in order to complement the automatic evaluation of ape submissions, manual evaluation of the primary systems submitted (seven for english-german, three for english-chinese) was conducted. in this section, we present the evaluation procedure, as well as the results obtained. ## conclusion we presented the results from the 6 th shared task on automatic post-editing at wmt. this year, we proposed two subtasks in which the mt output to be corrected was respectively generated by english-german and english-chinese neural systems unknown to the participants. the latter language pair represents a new entry for the task, which previously focused on spanish (in 2015), german (since 2016) and russian (in 2019) as target languages. the other major novelty factors are that: i) both the subtasks dealt with data drawn from the "generic" domain of wikipedia articles, and ii) the nmt systems used to generate the translations were not domain-adapted. as a consequence, participants had to confront with lower quality translations that left to ape large room for improvement. six teams participated in the english-german task, with a total of 11 submitted runs, while two teams participated in the english-chinese task submitting two runs each. their results computed with automatic metrics (ter and bleu) revealed significant gains over the "do-nothing" baseline. on english-german, the top-ranked system improved over the baseline by -11.35 ter and +16.68 bleu points, and the average improvements were the largest ones observed over the years (-4.89 ter, +6.5 bleu). on english-chinese the improvements of the top-ranked system are respectively -12.13 ter and +14.57 bleu points, with average gains of (-8.15 ter and +10.1 bleu). our human evaluation confirmed that on both the language pairs, almost all the primary submissions are significantly better than the baseline. on english-german, the improvement is up to the point that the quality of the automatic corrections produced by the top-ranked primary submissions is substantially on par with human corrections. all in all, these results confirm the effectiveness of ape to improve mt output in black-box conditions, especially when the adaptation of generic systems to a new "domain" is required.