PATQUEST : Papago Translation Quality Estimation (2020)

Abstract: This paper describes the system submitted by Papago team for the quality estimation task at WMT 2020. It proposes two key strategies for quality estimation: (1) task-specific pretraining scheme, and (2) task-specific data augmentation. The former focuses on devising learning signals for pretraining that are closely related to the downstream task. We also present data augmentation techniques that simulate the varying levels of errors that the downstream dataset may contain. Thus, our PATQUEST models are exposed to erroneous translations in both stages of task-specific pretraining and finetuning, effectively enhancing their generalization capability. Our submitted models achieve significant improvement over the baselines for Task 1 (Sentence-Level Direct Assessment; EN-DE only), and Task 3 (Document-Level Score).

## introduction with the widespread use of machine translation systems, there is a growing need to evaluate translated results at low-cost. the task of quality estimation (qe) addresses this issue, where the quality of a translation is predicted automatically given the source sentence and its translation. the estimated quality can inform users about the reliability of the translation, or whether it needs to be post-edited. previous qe systems generally include pretraining and finetuning steps, where the former step involves masked language modeling (mlm) utilizing large parallel corpora, with the expectation that the models will learn cross-lingual relationships @xcite . the models are, in turn, finetuned with task-specific data. however, while the pretraining step involves training data with near-perfect translations, lowquality translations are only introduced during the finetuning step. in this work, we suggest two key strategies that could alleviate this pretrain-finetune discrepancy in qe tasks by: (1) adopting a task-specific pretraining objective which is close to that of the downstream task, and (2) generating abundant taskspecific erroneous sentence pairs and their learning signals. our approach, which is depicted in figure 1 , is motivated from bleurt @xcite , where we extend their general approach to the bilingual qe setting. our submitted systems achieve significant improvements in performance over the baseline systems on wmt20 shared tasks for qe @xcite : an absolute gain of +35.2% in pearson score for (task 1) sentence-level direct assessment (en-de), and +18.4% in pearson score for (task 3) document-level score. ## sentence-level qe: direct assessment the task of sentence-level qe for direct assessment (da) involves predicting the perceived quality of the translation given the source and the translated sentences. following the footsteps of the previous work on qe, our sentence-level system also utilizes the pretrained multilingual language models such as bert @xcite and cross-lingual language model (xlm) @xcite . as the size of the training corpus for the qe task is very limited (7k sentence pairs), it is crucial to align these models closely to the task using more data in the form of task-specific pretraining. as opposed to pretraining the models on parallel corpora using the standard mlm approach, we pretrain the models in a multi-task setting using learning signals and data that are arguably more task-specific similar to @xcite . ## document-level qe: mqm scoring given a source and its translated document, this task involves identifying translation errors and estimating the translation quality of the document based on the taxonomy of the multidimensional quality metrics (mqm) foot_7 . with the pre-defined mqm taxonomy, human annotators assess whether the translation satisfies the specifications, and from these annotations, an mqm score is obtained. in this work, we focus on building a system that predicts the mqm score for a given pair of source and translated document. the major difficulty that we encountered in this task was the lack of training data. as the amount of provided data is limited (8,591 sentence pairs), a model that is solely finetuned on this small-scale data was not capable enough to differentiate sentences with varying level of errors. to address this issue, we propose simple yet effective methods for task-specific data augmentation, and task-specific training framework foot_8 . ## conclusion in this paper, we present a task-specific pretraining scheme for the qe task. our pretraining objective is devised so that it is closely related (task 1) or identical (task 3) to the finetuning objective. in addition, the models are exposed to abundant amount of error-induced translations generated from large parallel corpora, effectively alleviating the issue of data scarcity. our proposed models yield significant improvement over the baseline systems for the two tasks.