Findings of the WMT 2020 Shared Task on Quality Estimation (2020)

Abstract: We report the results of the WMT20 shared task on Quality Estimation, where the challenge is to predict the quality of the output of neural machine translation systems at the word, sentence and document levels. This edition included new data with open domain texts, direct assessment annotations, and multiple language pairs: English-German, English-Chinese, Russian-English, Romanian-English, Estonian-English, Sinhala-English and Nepali-English data for the sentence-level subtasks, English-German and English-Chinese for the word-level subtask, and English-French data for the document-level subtask. In addition, we made neural machine translation models available to participants. 19 participating teams from 27 institutions submitted altogether 1374 systems to different task variants and language pairs.

## introduction this shared task builds on its previous eight editions to further examine automatic methods for estimating the quality of neural machine translation (mt) output at run-time, without the use of reference translations. as in previous editions, it includes the (sub)tasks of word-level, sentencelevel and document-level estimation. important elements introduced this year are: a variant of the sentence-level task where sentences are annotated with direct assessment (da) 1 scores instead of labels based on post-editing; a new multilingual sentence-level dataset mainly from wikipedia articles, where the source articles can be retrieved for document-wide context; the availability of nmt 1 we note that the procedure followed for our data diverges from that proposed by @xcite in three ways: (a) we employ fewer but professional translators to score each sentence, (b) scoring is done against the source segment (bilingual annotation) and not the reference, and (c) we provide translators with guidelines on the meaning of ranges of scores. models to explore system-internal information for the task. in addition to advancing the state of the art at all prediction levels, our main goals are: we have three subtasks: task 1 aims at predicting da scores at sentence level (section 2.1); task 2 aims at predicting post-editing effort scores at both sentence and word levels, i.e. words that need editing, as well as missing words and incorrect source words (section 2.2); task 3 aims at predicting a score for an entire document as a function of the proportion of incorrect words in such a document, weighted by the severity of the different errors (section 2.3). tasks make use of large datasets produced from either post-editions or da annotations, or error annotation, all done by professional translators. the text domains vary for each subtask. neural mt systems were built on freely available data using an open-source toolkit to produce translations, and these models were made available to participants. we provide new training and test datasets for tasks 1 and 2, and a new test set for task 3. the datasets and models released are publicly available. participants are also allowed to explore any additional data and resources deemed relevant. baseline systems were entered in the platform by the task organisers (section 3). the shared task uses codalab as submission platform, where participants (section 4) could submit up to 30 systems for each task and language pair. results for all tasks evaluated according to standard metrics are given in section 5, while a discussion on the main goals and findings from this year's task is presented in section 6. ## subtasks in what follows we give a brief description for each subtask, including the datasets provided for them. ## baseline systems sentence-level baseline systems: for tasks 1 and 2, both word and sentence-level, we used the lstm-based predictor-estimator approach @xcite , implemented in openkiwi @xcite . the predictor model was trained on the same parallel data as the nmt systems for each language pair (made available at the task website), foot_3 while the the estimator was trained on the 7, 000 qe labelled data for each task. word-level baseline systems: for task 2, we also used the predictor-estimator as above, but it was trained to predict jointly word-level tags and sentence-level scores. document-level baseline system: for task 3, similarly as last year, we used a baseline which treats sentences independently and casts the problem as word-level qe, such that all words and gaps within an error span are given the tag bad. we then trained a predictor-estimator model, regrouping any contiguous sequence of tokens tagged as bad in a single error annotation. in order to get mqm scores, instead of computing the value according to its definition, we compute it simply as 1 minus the the ratio of bad tags. ## participants bergamot-latte (t1): bergamot-latte submitted two systems to the two variants of sentence-level predictions: (i) a black-box approach based on pre-trained representations; (ii) an unsupervised glass-box approach that leverages information extracted from the neural mt system. the black-box model consists of stacking a 2-layer multilayer perceptron on the vector representation of the cls token from the contextualised representation from xlm-r(conneau et al., 2020), using both the source and the target sentences as input. the glass-box approach explores the best-performing unsupervised quality indicators presented in fomicheva et al. (2020c) that rely on uncertainty quantification based on the monte carlo dropout method: d-tp and d-lex-sim. bergamot (t1, t2): bergamot explores recent work on glass-box qe that exploits nmt output distribution and attention to capture uncertainty as a proxy to mt quality. specifically, they use three groups of unsupervised quality indicators described in fomicheva et al. (2020c) as features for a regression model. bering lab (t2): bering lab proposes a finetuned version of a pre-trained xlm-r model. the model is first trained on a huge artificial qe data that is created by (i) translating a parallel corpus with an opennmt system; and (ii) using the ter tool to produce articifial labels for both word-and sentence-levels. the model is then fine-tuned using the shared task's data. for predictions at word-level, the final hidden vector of each token, including @xmath3, is fed into a linear layer with sigmoid activation in order to predict the probability of each of these token to be bad. quality labels for tokens and gaps are predicted separately with two distinct binary classification layers. for predictions at sentence-level, the final hidden vector of the @xmath4, considered as a pooled representation, is fed into two linear layers with tanh activation. submitted predictions are results of an ensemble of 5 models trained with different seeds: averaged predictions for sentence-level, and majority voting for word-level. ## discussion in what follows, we discuss the main findings of this year's shared task based on the goals we had previously identified for it. general progress. overall, participating systems achieved very promising results, with the best performing submissions showing moderate to strong correlation for sentence-level da and hter prediction tasks. one reason for high correlation levels is likely to be that top performing systems are based on pre-trained representations. like in other nlp tasks, for qe it had already been shown to substantially improve the results over models that do not use such representations, with heavier pre-trained embeddings contributing substantially more @xcite . strong pre-trained embeddings such as xlm-r were used by most submissions this year. when interpreting the results for all tasks, it should be noted that most of the participants use extremely resource-heavy systems, ensembles of multiple models with more than 500m parameters, which could make them difficult to use in practice. reporting the number of parameters could be a good practice for the future. comparison to previous years submissions are not possible as they use very different datasets, except for task 3, where a new test set was collected from the same initial larger dataset, but the training data is virtually the same. for the fine-grained 9 notice that if a gold annotation a g j has the highest overlap with a system annotation a s i , it does not necessarily mean that a s i has the highest overlap with a g j . comparison between hter and da. compared to the results from the previous editions of this shared task, participating systems show overall higher correlation with da labels. besides the qe systems getting much stronger, da labels might be easier to predict, as hter is a semi-automatic metric and may suffer from the same issues as ter, as it does not capture to what extent the overall quality of the sentence is affected by mt errors. we should note, however, that for the language pairs selected for post-editing this year (english-german and english-chinese) the correlation is higher for hter. a possible reason is a very skewed output distribution of the da scores for these particular language pairs. hter and da annotation capture different aspects of translation quality. in fact, as shown in @xcite , the correlation between the two types of scores is fairly low. an interesting question is whether the approaches that perform best for predicting da also achieve the best results for hter. figure 3 plots sentence-level pearson correlation with hter and direct assessments for the systems that participated in both tasks. while the systems with the highest and the lowest ranks are the same, results change considerably for the systems in the middle. specifically, transquest is one of the winning submissions for the prediction of da, but is outperformed by the submissions that use glass-box features, i.e. bergamot and ist and unbabel (kiwi-glass-box) for the hter task. multilingual approaches. most of the participating approaches rely on pre-trained multilingual representations and use the provided data annotated with quality labels for fine-tuning. this shows the potential for multilingual prediction in these systems making them much more appealing in prac- ## conclusions this year's edition of the qe shared task introduced a number of new elements: the largest number of languages ever, new types of annotation (direct assessment, in addition to labels derived from post-editing and manual error tagging), and number of samples annotated overall. it also attracted the largest number of teams and submissions. we believe the current set of tasks covers a broad enough range of challenges that are far from solved, such as improving performance for languages with skewed distributions, addressing low resource languages, predicting source words that lead to errors, multilingual or language-independent models, etc. in future editions, we hope to keep pushing for progress in these areas.