Findings of the WMT 2022 Shared Task on Quality Estimation (2022)

Abstract: We report the results of the WMT 2022 shared task on Quality Estimation, in which the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels, without access to reference translations. This edition introduces a few novel aspects and extensions that aim to enable more fine-grained, and explainable quality estimation approaches. We introduce an updated quality annotation scheme using Multidimensional Quality Metrics to obtain sentence- and word-level quality scores for three language pairs. We also extend the Direct Assessments and post-edit data (MLQE-PE) to new language pairs: we present a novel and large dataset on English-Marathi, as well as a zero-shot test set on English-Yoruba. Further, we include an explainability sub-task for all language pairs and present a new format of a critical error detection task for two new language pairs. Participants from 11 different teams submitted altogether 991 systems to different task variants and language pairs.

## introduction the 11th edition of the shared task on quality estimation (qe) builds on its previous editions and findings to further benchmark methods for estimating the quality of neural machine translation (mt) output at run-time, without the use of reference translations. it includes (sub)tasks that consider quality of machine translations at the word and sentence levels. over the past years, the qe field has been moving towards trainable, large, multilingual models that have been shown to achieve high performance, especially at sentence-level @xcite . in this edition, we further expand the provided resources, introducing new low-resource language pairs: a large dataset of english-marathi, suitable for training, development and testing and a smaller test-set on english-yoruba for zero-shot approaches. these, as well as previously published datasets for qe, rely mainly on direct assessments (da) foot_0 and post-edited translations, which provide estimates of quality either by using the human quality score(s) for each segment or by estimating the distance of a translation from a human-provided correction. as these annotations can sometimes obscure the exact location and/or significance of a translation error, we wanted to investigate the feasibility and efficiency of using a more finegrained annotation schema to obtain quality estimations at word-and sentence-level, namely multidimensional quality metrics (mqm) @xcite . mqm annotations have shown to be more trustworthy for the metrics task @xcite , motivating us to evaluate their suitability for the qe task. we make available new development and test data on three language pairs using mqm annotations. the aforementioned boost in performance of qe systems frequently comes at the cost of efficiency and interpretability, since they heavily rely on large models with many parameters. as a result, the predicted quality estimates are hard to interpret. at the same time, such high-performance, "black-box" models are frequently susceptible to systematic errors, such as negation omission @xcite and mistranslated entities @xcite . both phenomena are major concerns for mt quality estimation since they can undermine users' trust in new technologies and hamper the adoption of such models on a wide scale. to motivate approaches that address these cases we include an explainability subtask following its first edition at eval4nlp 2021 @xcite . in this subtask we ask participants to predict the erroneous words as rationale extraction for a sentence-level quality estimate, without any wordlevel supervision. by framing error identification as rationale extraction for sentence-level quality estimation systems, this subtask offers an opportunity to study whether such systems behave in the same way as humans would do. we also reshape the critical error detection task of last year and we build a new corpus to test the ability of qe systems to detect critical errors that simulate hallucinated content with additions, deletions, named entities, polarity changes and numbers. the corpus is created using smaug @xcite and we allow participation in constrained and unconstrained settings. for the constrained setting, participants have to build qe systems without having access to data from smaug, whereas participants from the unconstrained task can train their systems using additional data from smaug. in addition to advancing the state-of-the-art at all prediction levels, our main goals are: we thus have three tasks: task 1 the core qe task, consisting of separate sentence-level and word-level subtasks. for the sentence-level sub-tasks, the goal is to predict a quality score for each segment in the test set, which can be a variant of da ( §2.1.1) or mqm ( §2.1.1). for the word-level subtasks, participants have to predict translation errors at word-level, via binary quality tags (see §2.1.2). task 2 explainable qe task, aiming to obtain wordlevel rationales for sentence-level quality scores ( §2.2). task 3 the critical error detection task, aiming to predict sentence-level binary scores indicating whether or not a translation contains a critical error ( §2.3). the tasks make use of large datasets annotated by professional translators with either 0-100 da scoring, post-editing or mqm annotations. we update the training and development datasets of previous editions and provide new test sets for tasks 1 and 2. additionally, we provide a novel setup for task 3, with novel train, development and test data. the datasets and models released are publicly available foot_1 . participants are also allowed to explore any additional data and resources deemed relevant, across tasks. the shared task uses codalab as submission platform, where participants (section 4) could submit up to 2 submissions a day for each task and language pair (lp), up to a total of 10 submissions. results for all tasks evaluated according to standard metrics are given in section 5. baseline systems were trained by the task organisers and entered in the platform to provide a basis for comparison (section 3). a discussion on the main goals and findings from this year's task is presented in section 6. ## quality estimation tasks in what follows, we briefly describe each subtask, including the datasets provided for them. ## baseline systems task 1: quality estimation baseline systems: for task 1, both for word and sentence-level, we used a multilingual transformer-based predictor-estimator approach @xcite , which is described in detail in @xcite . for the implementation and training we use the openkiwi @xcite framework. we trained the baseline model using a multilingual and multitask setting and training jointly on the sentence-level scores and word-level tags. for the word-level loss, l word , the weight of bad tags is multiplied by a factor of λ @xmath0.0, but the sentence-and wordlevel loss have equal weight in the overall joint loss estimation: @xmath1. we trained different baselines for the da/post-edit originated language pairs and the mqm originated languagepairs. for the da/post-edit baseline, the model was trained using the da scores as sentence targets and the ok/bad tags as word targets. for training we used the concatenated data for all language pairs available under training data and used the concatenation of the additional language pairs that were made available in the development set as validation. we trained two baselines with this setup, using different encoders for the encoding (predictor) part of the architecture: (a) xlm-r transformer with the xlm-roberta-large model and (b) rembert model which has been pre-trained on additional languages that include yoruba and can hence account for the zero-shot language. for the mqm baseline, the model was trained using the normalised and inverted mqm scores as sentence targets and the ok/bad tags as word targets. the baseline model was trained using the concatenated training data for all three language pairs and used the concatenated development data for the same pairs as the validation set. the xlm-r transformer with the xlm-roberta-large model was used as an encoder. task 2: explainability baseline systems: we provide two baseline systems for task 2. one is a random baseline where we sampled scores uniformly at random from a continuous [0..1) range for each target token and for a sentence-level score. the other one is a combination of a supervised quality annotation model, openkiwi @xcite and @xcite where openkiwi is used to predict sentence-level quality scores while lime is used to compute, for every token in the target sentence, its importance for the sentence-level quality score returned by openkiwi. for the openkiwi implementation we used a similar setup described for the baselines of task 1, but we trained the openkiwi model using only sentence-level supervision, to align with the task requirements. we trained two multilingual instances, one on da-and one on mqm-derived data, using xlm-r large encoder in both cases. lime is a model-agnostic post-hoc explanation method which trains a linear model to estimate the behavior of a target model (i.e., openkiwi in our case) around an input example to be explained so the weights of the linear model correspond to the importance of individual input tokens. because higher sentence-level scores in our gold standard mean better translation quality, we invert tokenlevel scores generated by lime so that higher values correspond to errors as required by the task description. task 3: critical error detection baseline systems: for task 3, we consider a baseline system for each setting. in the constrained setting, we considered comet-qe @xcite foot_5 , which was a topperforming qe-as-a-metric system in last years metrics shared task @xcite . regarding the unconstrained setting, we finetune an xlm-roberta-large model using the comet framework @xcite . both the source and translation are jointly encoded into a vector representation which is the input of a final estimator that predicts the probability of the translation containing a critical error. here, the estimator weights are randomly initialised. we fine-tune the model on the provided training data for a maximum of 5 epochs. at the end of each epoch, we perform a validation step by measuring the mcc on the validation set considering a classification threshold of 0.5. we select the model with the highest mcc on the validation data. ## participants alibaba-translate (t1-da): for the da subtask, the team participated in all language pairs except the zero-shot lp. the implemented system @xcite , uses glass-box qe features to estimate the uncertainty of machine translation segments and incorporates the features into the transfer learning from the large-scale pre-trained model, xlm-r. the participants used exclusively the da data provided for this edition of the qe shared task. of the provided data, the 7 language pairs except for english-marathi, were combined to train a multilingual model. for english-marathi, a separate bilingual model was trained. for the final submission the participants ensembled multiple checkpoints. the applied framework unite @xcite considers three input formats -sourceonly (qe or reference-free metric), referenceonly and source-reference-combined. the participants used synthetic datasets with pseudo labels during continuous pre-training phase, and fine-tuned with da and mqm training datasets from the year 2017 to 2021. to obtain the final model predictions they use the source-only evaluation. for multilingual phase, they ensembled predictions using two different backbones -one using xlm-r encoder and the other using infoxlm. for the ensembling, they picked the best 2 checkpoints on the development dataset. bjtu-toshiba (t1-mqm): bjtu-toshiba participation focused on ensembling different models and using external data. they ensemble multiple pre-trained models, both monolingual and bilingual. the monolingual models are trained only on the text of the target language. specifically, they use monolingual bert, roberta, and electra-discriminator as the monolingual extractor, and xlm-r as the bilingual extractor. they also use in-domain parallel data to fine-tune and adapt the pretrained models to the target language and domain. the in-domain data is selected by a bert-classifier from the parallel data provided by the news translation task, and for each direction, they end up using roughly 1 million sentence pairs for fine-tuning. they explore two styles of fine-tuning, namely translation language model and replaced token detection. for replaced token detection, they use the first 1/3 layers of the model as generator, and after the training they drop the generator and only use the discriminator as the feature extractor. ## results in this section, we present and discuss the results of our shared task. please note that for all the three subtasks we used statistical significance testing with @xmath2.05. ## discussion in what follows, we discuss the main findings of this year's shared task based on the goals we had previously identified for it. ## conclusions this year's edition of the qe shared task introduced a number of new elements: new lowresource language pairs (marathi and yoruba), new annotation conventions for sentence and word level quality (mqm), new test sets, and new versions of explainability and critical error detection subtasks. the tasks attracted a steady number of participating teams and we believe the overall results are a great reflection of the state-of-the-art in qe. we have made the gold labels and all submissions to all tasks available for those interested in further analysing the results, while newly interested participants can still access the competition instances on codalab and directly compare their performance to other models. we aspire for the future editions to continue the efforts set in this and previous years and expand the resources and coverage of qe, while further exploring recent and more challenging subtasks such as fine-grained qe, explainable qe and critical error detection.