BERGAMOT - LATTE Submissions for the WMT 20 Quality Estimation Shared Task (2020)

Abstract: This paper presents our submission to the WMT2020 Shared Task on Quality Estimation (QE). We participate in Task and Task 2 focusing on sentence-level prediction. We explore (a) a black-box approach to QE based on pre-trained representations; and (b) glass-box approaches that leverage various indicators that can be extracted from the neural MT systems. In addition to training a feature-based regression model using glass-box quality indicators, we also test whether they can be used to predict MT quality directly with no supervision. We assess our systems in a multi-lingual setting and show that both types of approaches generalise well across languages. Our black-box QE models tied for the winning submission in four out of seven language pairs inTask 1, thus demonstrating very strong performance. The glass-box approaches also performed competitively, representing a light-weight alternative to the neural-based models.

## introduction quality estimation (qe) @xcite is an important part of machine translation (mt) pipeline. it allows us to evaluate how good a translation is without comparison to reference sentences. as part of the wmt20 shared task on quality estimation, two sentence-level tasks were proposed. in task 1, participants are asked to predict human judgements of mt quality generated following a methodology similar to direct assessment (da) @xcite . the goal of task 2 is to estimate the post-editing effort required in order to correct the mt outputs and measured using the hter metric @xcite . the rest of this paper is organised as follows. section 2 describes the glass-box (2.1) and blackbox (2.2) qe methods that we explore in our submissions. section 3 describes the dataset used for the wmt2020 shared task on quality estimation. section 4 provides our experimental settings, whereas section 5 presents the results. conclusions are given in section 6. ## approach below we first describe our glass-box submissions based on the quality indicators that can be obtained as a by-product of decoding with an nmt system. second, we present our neural-based qe submissions, which explore transfer learning with pretrained representations. in both cases, we describe how qe is addressed as a multilingual task. ## data this year two sentence-level qe tasks are available. for task 1 the participants are expected to predict da-style human judgements @xcite , whereas the goal of task 2 is to estimate the post-editing effort (hter). the data for task 1 includes six language pairs: sinhala-english (si-en), nepalese-english (ne-en), estonian-english (et-en), romanian-english (ro-en), english-german (en-de) and english-chinese (en-zh), where source sentences were extracted from wikipedia articles. for task 2, only english-chinese and english-german are available. we also experimented with an additional dataset collected by iqt labs in collaboration with the university of sheffield. this is an russian-english (ru-en) dataset that contains a combination of russian reddit forums (75%) (using the reddit api) and russian wikiquotes (25%). all mt outputs were generated by transformer-based nmt systems @xcite . all datasets contain at least three da judgements per mt segment by professional translators (0-100), with absolute quality scores standardised according to each annotator's mean and standard deviation. hter labels were obtained by having professional translators fixing any errors in the translations, followed by using the ter foot_0 tool. for each language pair the organisers provided training set (7000 sentences), development set (1000 sentences) and a blind test set (1000 sen-tences). for our final experiments we combined the training (7000 sentences) and development (1000 sentences) sets, set a grid for the hyperparameters of our regression models and performed 5-fold crossvalidation to choose the best hyperparameters. ## results in this section, we present and analyse the results for our submissions to task 1. we provide a general comparison of the glass-box and black-box systems and also look at some specific aspects of their performance. ## conclusions we presented glass-box and black-box models submitted to the wmt2020 qe shared task. blackbox models showed the results on a par with the top submissions. glass-box methods achieve from moderate to strong linear correlation with human judgments and can be used as a light-weight and cost-effective alternative in a scenario where the nmt model is available. besides that, we conducted experiments to test the performance of our qe systems in a multilingual setting. we showed that the performance of both approaches is comparable when training and predicting on the same language pair, and when training a single model to predict on multiple language pairs.