IST -Unbabel Participation in the WMT 20 Quality Estimation Shared Task (2020)

Abstract: We present the joint contribution of IST and Unbabel to the WMT 2020 Shared Task on Quality Estimation. Our team participated on all tracks (Direct Assessment, Post-Editing Effort, Document-Level), encompassing a total of 14 submissions. Our submitted systems were developed by extending the OpenKiwi framework to a transformer-based predictor-estimator architecture, and to cope with glass-box, uncertainty-based features coming from neural machine translation systems.

## introduction quality estimation (qe) is the task of evaluating a translation system's quality without access to reference translations (blatz et al., 2004; specia et al., 2018). this paper describes the joint contribution for instituto superior técnico (ist) and unbabel to the wmt20 quality estimation shared task, where systems were submitted to all three tasks: 1) sentence-level direct assessment; 2) word and sentence-level post-editing effort; and 3) documentlevel annotation and scoring. unbabel's participation in previous editions of the shared task (2016, 2017, 2019) used ensemble of strong individual systems, with varying architectures and hyper-parameters. while this strategy led to very strong results, large system ensembles are not a very practical solution, complicating model deployment and requiring expensive computation and memory usage. this year, in contrast, our focus was on simplicity: only single model systems were submitted and, in a few cases, an additional simple ensemble of the same model. transfer learning on top of pretrained multilingual models was also used for avoiding manual pretraining for each language pair. last year's winning submission (kepler et al., 2019a) combined strong individual systems built on top of the openkiwi framework (kepler et al., 2019b) and pretrained transformer models. we consolidated those changes with support for newly released pretrained models and packages and published a new version 2.0 of the openkiwi framework. @xcite we trained and submitted single model systems in openkiwi for all tasks, beating all baselines by a large margin. additionaly, we also used openkiwi with small adaptations to handle specific sources of information in tasks 1 and 3. task 1, in particular, was introduced this year with direct assessment scores as targets. further, it introduced the novelty of providing the trained nmt models that were used for producing the translations. previously, only black-box qe was considered in the wmt shared task, as it is one of the main uses cases. with the availability of the nmt models, new glass-box approaches can be explored. our best submitted systems drew inspiration from (fomicheva et al., 2020) to leverage this information, improving in performance and robustness over a black-box approach. our main contributions are: of noteworthy mention is that the nmt models for tasks 1 and 2 where provided along with the data, which opened up the possibility of using glassbox approaches. ## implemented systems to avoid the complexity of ensemble of several systems, all our submitted systems consisted of a single model type. in addition to standard openkiwi 2.0 systems submitted to tasks 1 and 2 ( §3.1), we implemented two types of extensions on top of openkiwi, one for exploring glass-box approaches for tasks 1 and 2 ( §3.2), and one for handling document-level qe for task 3 ( §3.3). ## conclusions our approach to this year's edition of the qe shared task was simplicity. our submissions consisted of either single models, or simple ensembles of multiple runs of the same model. moreover, we used multi-task models in task 2, where a system was trained on all three possible outputs (target and source word level and sentence level). we implemented a new version of openkiwi and used it as our baseline. it significantly outperformed the official shared task baseline across the board, which was based on the previous version of openkiwi. finally, we showed that having access to nmt models enables using glass-box approaches to qe, which in turn improves performance when used in