Findings of the WMT 2023 Shared Task on Automatic Post-Editing (2023)

Abstract: We present the results from the 9th round of the WMT shared task on MT Automatic Post-Editing, which consists of automatically correcting the output of a “black-box” machine translation system by learning from human corrections. Like last year, the task focused on English→Marathi, with data coming from multiple domains (healthcare, tourism, and general/news). Despite the consistent task framework, this year’s data proved to be extremely challenging. As a matter of fact, none of the official submissions from the participating teams succeeded in improving the quality of the already high-level initial translations (with baseline TER and BLEU scores of 26.6 and 70.66, respectively). Only one run, accepted as a “late” submission, achieved automatic evaluation scores that exceeded the baseline.

## introduction this paper presents the results of the 9 th round of the wmt task on mt automatic post-editing (ape). the task involves the automatic correction of the output generated by a "black-box" machine translation system by learning from humanrevised machine-translated output supplied as training material. the overall task formulation (see section 2) remained consistent with that of all previous rounds. in this formulation, the challenge revolves around fixing errors in english documents that have been automatically translated by a stateof-the-art, non-domain-adapted neural mt (nmt) system unknown to the participants. in continuity with last year's round, the evaluation focused on @xmath0, 1 with training/dev/test data selected from a mix of domains, namely-healthcare, tourism, and general/news (see section 3). three teams participated in the task by submitting a total of four runs for the final evaluation (see 1 marathi is an indo-aryan language predominantly spoken by marathi people in the indian state of maharashtra. section 4). 2 however, while only two out of the three participants were able to submit their runs on time, the one remaining submission arrived with a two-month delay. this led us to categorize it as a late (therefore, unofficial) submission for the sake of fairness to the other participants. for all the teams, the task posed significant challenges primarily due to the high average quality of the initial translations slated for post-editing (26.6 ter / 70.66 bleu / 79.78 chrf). this challenge was compounded by the substantial imbalance in distribution between near-perfect translations (approximately 40% of the total) and those necessitating extensive revisions (approximately 20%). as a consequence, none of the official runs was able to improve over the baseline in terms of the task's automatic evaluation metrics (section 5.1), with the best run achieving results (27.73 ter / 69.03 bleu / 78.64 chrf) that highlight a slight quality degradation compared to the original, untouched nmt outputs that represent our baseline. for the sake of completeness, we report that the late submission achieved a slight improvement over the baseline, attested by ter, bleu, and chrf scores of 25.74, 71.27, and 80.41, respectively. the results computed by means of automatic evaluation metrics were also confirmed by our human evaluation based on direct assessment (section 5.2). this 9 th round of the wmt ape shared task kept the same overall evaluation setting of the previous eight rounds. specifically, the participating systems had to automatically correct the output of an unknown "black-box" mt system (a generic nmt system not adapted to the target domain) by learning from training data containing human revisions of translations produced by the same system. for the second year in a row, the selected language pair was english-marathi (with marathi as the target language for post-editing). training, development and test data were drawn from the following three domains: healthcare, tourism, and general/news. ## submissions as shown in korea advanced institute of science and technology (kaistai). this team participated with a system inspired by the recent surge of large language models (llms) that have been successfully applied to a variety of language generation tasks. their goal was to verify whether llms could perform the ape task through prompting. to this aim, they used gpt-3.5-turbo with specific prompts designed to generate either (a) post-edits or (b) post-edits along with the rationales behind them. while the results of preliminary evaluations based on comet suggested the viability of the approach for medium-/high-resource language pairs, they also highlighted that the often radical changes produced by llms can potentially be penalized by more strict reference-based evaluations based on bleu, ter, or chrf. ## conclusion we presented the results from the 9 th shared task on automatic post-editing at wmt. in continuity with the 2022 round, the task focused on the automatic correction of nmt outputs generated by a black-box english-marathi system. the three participating systems were evaluated both automatically (with ter as the primary metric, bleu, and chrf) and manually. according to automatic evaluation results, only one (late) submission succeeded in outperforming the do-nothing baseline. the analysis of this year's data suggests that one of the main causes of difficulty might be the bimodal, u-shaped ter distribution of the test instances, which substantially differs from the test set distributions observed in all previous rounds (skewed but a pattern closer to normal). our manual evaluation confirms the automatic evaluation outcomes and affirms the challenge posed by ape for the current approaches. we observe that one of the systems performs quite close to the mt baseline while the other performs well below the same. additionally, the lack of multilingual datasets in llm training/benchmarking raises a question on the viability of performing challenging multilingual tasks like ape. all in all, these findings advocate for further research on this challenging problem, which, far from being solved, this year revealed new nuances in terms of difficulty. next year, we plan to introduce two new low-resource language pair datasets for the ape task. future developments will also likely include a re-definition of some aspects of the evaluation settings, which have remained relatively stable over the years. for instance, the set of automatic evaluation metrics will likely be reconsidered and expanded so as to include more semantics-oriented measures, with an eye on the advent of large language models increasingly adopted also for ape.