Findings of the WMT 2021 Shared Task on Quality Estimation (2021)

Abstract: We report the results of the WMT 2021 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word and sentence levels. This edition focused on two main novel additions: (i) prediction for unseen languages, i.e. zero-shot settings, and (ii) prediction of sentences with catastrophic errors. In addition, new data was released for a number of languages, especially post-edited data. Participating teams from 19 institutions submitted altogether 1263 systems to different task variants and language pairs.

## introduction the 10th edition of the shared task on quality estimation (qe) builds on its previous editions to further benchmark methods for estimating the quality of neural machine translation (mt) output at runtime, without the use of reference translations. it includes the (sub)tasks of word-level and sentencelevel estimation. the document-level task was removed from this edition, since it has not attracted many participants in previous editions. important elements introduced this year are: a new sentencelevel task where sentences are annotated with a binary label reflecting whether or not they contain a critical error that could lead to catastrophic consequences; new test data for languages that are not covered by any training set for zero-shot prediction (both direct assessment and post-editing based labels, at sentence and word levels. scores instead of labels based on post-editing; a new multilingual sentence-level dataset mainly from wikipedia articles, where the source articles can be retrieved for document-wide context; the availability of nmt models to explore system-internal information for the task. in addition to advancing the state-of-the-art at all prediction levels, our main goals are: we have three subtasks: task 1 aims at predicting a variant of da scores at sentence level (section 2.1); task 2 aims at predicting post-editing effort scores at both sentence and word levels, i.e. words that need editing, as well as missing words and incorrect source words (section 2.2); task 3 aims at predicting a binary label at sentence level to indicate whether the sentence contains one or more critical errors (section 2.3). tasks make use of large datasets annotated by professional translators with either 0-100 da scoring, post-edition, or critical error flagging. the text domains and languages vary for each subtask. neural mt systems were built on freely available data using an open-source toolkits to produce translations. we provide new training and test datasets for tasks 2 and 3, new test sets for task 1, as well as new zero-shot test sets for tasks 1 and 2. the datasets and models released are publicly available. participants are also allowed to explore any additional data and resources deemed relevant. baseline systems were entered in the platform by the task organisers (section 3). the shared task uses codalab as submission platform, where participants (section 4) could submit up to 30 systems for each task and language pair, except for the multilingual track of tasks 1 and 2 (up to 10 systems). results for all tasks evaluated according to standard metrics are given in section 5, which this year also included model size. a discussion on the main goals and findings from this year's task is presented in section 6. ## subtasks in what follows, we give a brief description for each subtask, including the datasets provided for them. ## baseline systems sentence-level baseline systems: for tasks 1 and 2, both word and sentence-level, we used a multilingual transformer-based predictor-estimator approach @xcite , which is described in detail in @xcite . both baselines are implemented in openkiwi @xcite and trained using the concatenated train portions of the data for training (combining all 7 language pairs) and the concatenated development portions of the data for validation/early-stopping. in all cases, the xlm-roberta transformer was used for the encoding (predictor) part of the architecture, using xlm-roberta-base for all experiments. the xlm-roberta encoder is initially trained on the concatenated train and development segments using ulm fine-tuning @xcite and then this fine-tuned encoder is used in the full predictor-estimator model which is fine-tuned separately for each task scores (da or hter). word-level baseline systems: for task 2, we used the same architecture and encoder as above, but it was trained to predict jointly word-level ok/bad tags and sentence-level hter scores. ## participants bergamot (t1): bergamot explores the use of a teacher-student knowledge distillation framework to transfer knowledge from a strong qe teacher model to a much smaller student model with a different, shallower architecture. namely, the system distill a large and powerful qe model transquest [1] based on xlm-roberta into a small birnn-based deepquest model [2]. the predictions from a teacher qe model trained on mlqe data [3] is used to train the lightweight student. additionally, the system employs data augmentation through teacher predictions on monolingual data sampled from wikipedia following id participating team bergamot university of sheffield & imperial college london & university of wolverhampton, uk (gajbhiye et al., 2021) bergamot-utartu university of tartu, estonia (yankovskaya and fishel, 2021) ensbrt university of illinois at chicago & iqvia, usa (chowdhury et al., 2021) hw-tsc huawei translation services center & nanjing university, china (chen et al., 2021) ist-unbabel instituto de telecomunicações lisbon & instituto superior técnico lisbon & unbabel, portugal (zerva et al., 2021) jhu-microsoft johns hopkins university & microsoft (ding et al., 2021) lama-icl lama -imperial college london, uk (jiang et al., 2021) nict kyoto national institute of ict, japan (rubino et al., 2021) papago naver, republic of korea (lim et al., 2021) postech pohang university of science and technology, republic of korea (heo et al., 2021) qemind alibaba, china (wang et al., 2021) rtm bogaziçi university, turkey (biçici, 2021) smob-eceiit technion -israel institute of technology, israel -tuda technische universität darmstadt, germany (geigle et al., 2021) ensbrt (t2): ensbrt propose a system that is an ensemble of multilingual bert (mbert)based regression models, which are generated by fine-tuning on different input settings. they adapted their system for the zero-shot setting by exploiting target language-relevant language pairs and pseudo-reference translations. hw-tsc (t1, t2, t3): hw-tsc's submissions in the three sub tasks follow the framework of predictor-estimator @xcite , with a pre-trained xlm-roberta as predictor and task-specific classifier or regressor as estimator. they further explore to incorporate additional high-quality translation sentences in the way of multitask learning or encoding it with the predictor directly. for task1, they enable the model to jointly learn to score translations with a regression task and to distinguish between translations and additional better translations (i.e. post-edits from task2 dataset) with a classification task. they also exploit a data augmentation strategy based on mc dropout to improve zero-shot performance. they ensemble multi results with mc dropout to keep a relatively small number of parameters and model size. for task 2, they leverage additional translation sentence generated by a nmt system trained for wmt21 news shared task in the way of directly concatenating it with source and original translation. a unified model is trained under multi-task learning framework where losses of source word, translation token and gap, additional translation token and gap, hter scores are all added up to train the model. for task 3, they translate source sentences with google and baidu translation api. each new translation is then concatenated to the corresponding source and translation pair, to get a sentence feature. they ensemble the results of three different models and take their majority voting as final result. ist-unbabel (t1, t2): for task 1, ist-unbabel's system is an ensemble of an xlm-r with stacked adapter layers and an mbart that incorporates different types of uncertainty (annotation uncertainty and mt uncertainty). for task 2, the submitted system is an ensemble of two xlm-r with adapters (the difference being the xlm-r checkpoint, while one uses the xlm-roberta-large normal checkpoint the other uses an xlm-r checkpoint pertained on data from the metrics shared task). the ensembled checkpoints learn to predict both word level tags and sentence level hter scores in a multi-task setting. jhu-microsoft (t2): the jhu-microsoft submission focuses on the word-level subtask of task 2, for which they adopt levenshtein transformer (gu et al. 2019) as their model architecture. the training procedure starts with training a non-autoregressive translation model using a levenshtein transformer, with its encoder and decoder initialized with those from the m2m multilingual translation model @xcite . they then fine-tune the model to perform the word-level qe task on the human-annotated training set, or optionally also on automatically generated pseudopost-editing translation triplets. the final submission is an ensemble of 4-8 best models on the 2020 test set for each language pair, and the ensemble is performed by linear interpolation of scores from each model, with the interpolation weights tuned by the nelder-meade method @xcite . lama-icl (t3): lama-icl's approach builds on cross-lingual pre-trained representations in a sequence classification model. we further improve the base classifier by (i) adding a weighted sampler to deal with unbalanced data and (ii) introducing feature engineering, where features related to toxicity, namedentities and sentiment, which are potentially indicative of critical errors, are extracted using existing tools and integrated to the model in different ways. we train models with one type of feature at a time and ensemble those models that improve over the base classifier on the development (dev) set. nict kyoto (t3): nict kyoto submission for the critical error detection task consists in large scale qe pretraining with synthetic data in a multilingual and multimetric setting. a total of six sentence-and word-level quality indicators were involved in continued training of an xlm-r checkpoint using qe oriented training objectives in a multi-task fashion, based on a corpus of 70 million sentence pairs including twelve languages. fine-tuning on the official dataset was then performed and resulting models from different initializations were ensembled to constitute the final submission. papago (t1): papago's submission is a multilingual quality estimation system that explores the combination of pre-trained language models and multi-task learning architectures. they propose an iterative training pipeline based on pretraining with large amounts of in-domain synthetic data and finetuning with gold (labeled) data. they then compress our system via knowledge distillation in order to reduce parameters yet maintain strong performance. postech (t2): postech's model uses two pre-trained monolingual encoders to first produce monolingual representations of the two input data separately and then exchanges the information of these representations through two additional cross attention networks. the two pre-trained monolingual encoders are an english electra and a german electra, respectively. they fine-tuned their system in two stages: the qe pre-training stage and the qe fine-tuning stage. in the former, they used a large quantity of artificial training data, and the loss value equals to the sum of the losses for the estimated hter (sentence-level qe), ok-bad for the source tokens, ok-bad for the mt tokens, and ok-bad for the gap tokens in between two mt tokens. in the latter, they only used human labelled training data, and the loss value is one of the four above mentioned loss values, depending on the targeted subtask. qemind (t1, t3): qemind propose novel glassbox qe features to estimate the uncertainty of machine translations and incorporate them into the transfer learning from the large-scale pre-trained model, xlm-roberta. in addition, three important strategies are particularly utilized for improving the qe system's performance such as multilingual training, data augmentation and model ensemble. rtm (t1, t2): referential translation machines (rtms) superlearner results combine individual machine learning model results via crossvalidation on the training set. the combined models guarantee lower error on the validation set than the model that minimises the overall error. a superlearner model improves the results over non-mixture results. smob-eceiit (t1): smob-eceiit's participation is fully unsupervised, as created without using any annotated data or even parallel bilingual data. the system is composed of two novel different methods. the first method is based on soft alignment of multilingual contextual embeddings, generated by pre-trained mbert or xlm-r (depending on the specific language). the soft alignment is calculated by the sinkhorn distance (cuturi, 2013), which is an optimal transportation distance with an entropic regularization term. the second method is based on the assumption that word embedding spaces are approximately isometric @xcite , and on an isometric-invariant method known as persistent homology @xcite . each sentence is represented by the distances between its own word embeddings (either static or contextual). the created distance matrices are compared using the wasserstein distance between their persistence barcodes (the output of persistent homology computation). finally, the two methods are linearly combined. tuda (t1): tuda's submissions are produced with pre-trained multilingual language models which they extended to new languages and unseen scripts using recent adapter-based methods. ## discussion in what follows, we discuss the main findings of this year's shared task based on the goals we had previously identified for it.