Findings of the WMT 2023 Shared Task on Quality Estimation (2023)

Abstract: We report the results of the WMT 2023 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the provided data to new language pairs: we specifically target low-resource languages and provide training, development and test data for English-Hindi, English-Tamil, English-Telegu and English-Gujarati as well as a zero-shot test-set for English-Farsi. Further, we introduce a novel fine-grained error prediction task aspiring to motivate research towards more detailed quality predictions.

## introduction this edition of the shared task on quality estimation (qe) aims to build on previous editions and findings to further benchmark methods for estimating the quality of neural machine translation (mt) output at run-time, without the use of reference translations. it includes (sub)tasks that consider the quality of machine translations at word-and sentence-level. over the past years, the qe field has been moving towards explainable, large, multilingual models that have been shown to achieve high performance, especially at sentence-level @xcite . the recent proliferation of large language model (llm) technology and the consequential performance improvements in mt elevate the significance of advancing methodologies for quality estimation. in light of this, emphasis should be placed on multilingual quality estima-tion, in particular for low-and medium-resource languages, necessitating the development of more precise and interpretable quality assessment techniques. additionally, it is important to address the challenge of robustness to hallucinations, prioritise sustainability, and optimise computational efficiency. these considerations collectively contribute to progress toward trustworthy and dependable qe systems that could facilitate real-time, reliable assessments of translation quality. in this edition of the shared task, we further expand the provided resources, introducing new low-resource language pairs for indian languages, namely marathi, tamil, telugu, gujarati and hindi, as well as farsi and hebrew. following the previous editions, we provide both annotations for direct assessments (da), post-edits (pe) and multidimensional quality metrics (mqm) @xcite . we describe in detail the annotation process and provide statistics for the different language pairs in section 2. overall, in addition to advancing the state-of-theart at all prediction levels, our main goals are: we thus designed two tasks this year: task 1 the core qe task, which consists of separate sentence-level and word-level sub-tasks. for the sentence-level sub-tasks, the goal is to predict a quality score for each segment in a given test set, which can be a variant of da ( §2.1) or mqm ( §2.2). for the word-level subtask, participants had to predict translation errors in the form of binary quality tags (see §3.1.3). ## quality estimation tasks in what follows, we briefly describe each sub-task, including the datasets provided for them. ## participants bering lab (t1-sl; en-de, zh-en): for each language pair, the team used an ensemble of the best three models from a pool of 10 models jointly trained for the word and sentence level tasks using a novel relative ranking loss function and adversarial weight perturbation (awp) to improve the robustness of the model. using no additional pseudo-generated data, they pre-train the models using publicly available data from the previous wmt conferences that were augmented using the ter tool to generate binary word tags. they then finetuned 10 separate models on the labelled data from the wmt 2022 qe task randomly split into 10 folds. the models are fine-tuned in two steps. in the first step, the models are finetuned without awp using the same objective as the pre-training step. then, the best checkpoint from the first step is selected and tuned on the same objective, but with awp. for the final submission they ensemble (average) the z-normalized scores from the top three models to get the final predictions. ## results in this section, we present and discuss the results of our shared task. please note that for all the three sub-tasks we used statistical significance testing with @xmath0.05. ## discussion in what follows, we discuss the main findings of this year's shared task based on the goals we had previously identified for it. ## conclusions this year's edition of the qe shared task introduced a number of new elements: new lowresource language pairs (including two zero-shot ones), new test sets, and new fine-grained error detection task that we aspire to continue in future editions. it also introduced a mix of hallucinated data together with the original translations, allowing us to assess the robustness of submissions and detect failure patterns that will hopefully help develop more robust qe systems in the future. the tasks attracted a steady number of participating teams and we believe the overall results are a great reflection of the evolution of the qe field. we note that the gold labels and best submissions to all tasks are made available for those interested in further analysing the results. we aspire for the future editions to continue the efforts set in this and previous years and expand the resources and coverage of qe, while further exploring recent and more challenging subtasks such as fine-grained qe and explainable qe.