Papago’s Submission for the WMT 21 Quality Estimation Shared Task (2021)

Abstract: This paper describes Papago submission to the WMT 2021 Quality Estimation Task 1: Sentence-level Direct Assessment. Our multilingual Quality Estimation system explores the combination of Pretrained Language Models and Multi-task Learning architectures. We propose an iterative training pipeline based on pretraining with large amounts of in-domain synthetic data and finetuning with gold (labeled) data. We then compress our system via knowledge distillation in order to reduce parameters yet maintain strong performance. Our submitted multilingual systems perform competitively in multilingual and all 11 individual language pair settings including zero-shot.

## introduction quality estimation (qe) evaluates the quality of machine translated output without human reference translation @xcite . qe has a variety of applications in the machine translation (mt) pipeline and is particularly useful in industry settings by informing translation quality to endusers. high performance in sentence-level qe @xcite is achieved by building a model on top of pretrained language model (plm); xlm-roberta-large @xcite performs particularly well as shown in previous wmt sentence-level qe shared task. however, such plms contain extremely large number of parameters. this year's task is different from the previous years' task as submitted systems are ranked based on both model size 1 and model performance 2 . for concurrent work @xcite applies knowledge distillation @xcite from a plm-based qe architecture to a much lighter 1 disk space without compression and number of parameters. 2 pearson's correlation coefficient, root mean square error (rmse), mean absolute error (mae). birnn-based architecture, reducing memory requirements. data scarcity is another issue relevant to qe tasks where there are often limited amount of gold training data. previous wmt systems incorporate data augmentation techniques and show improvements in model performance when training with additional sources of data @xcite . our system builds a model on top of plm and trains with multi-task learning (mtl) @xcite . @xcite ; @xcite where back-translation is iteratively applied to the same monolingual corpus to successively generate higher quality synthetic training data in the context of neural machine translation (nmt), our proposed approach consists of an iterative knowledge transfer procedure which aims to repeatedly produce better quality pseudo labels for large amounts of synthetic training data. during the final stage of our training pipeline, knowledge distillation is applied from teacher to student model in order to reduce model size while maintaining competitive performance. we participate in wmt 2021 quality estimation @xcite task 1 for multilingual and all individual language pair settings. our system is a single multilingual sentence-level qe model that performs very strongly in both multilingual and individual language pair settings. ## data in this year's task, participants are provided with 7k train set (train), 1k development set (dev), and 1k test set (test20) for 7 language pairs: high-resource english-german (en-de) and english-chinese (en-zh), medium-resource romanian-english (ro-en) and estonian-english (et-en), and low-resource sinhalese-english (si-en) and nepalese-english (ne-en), as well as russian-english (ru-en). the source side sentences of language pairs excluding ru-en are col-lected from wikipedia data; the source side sentences of ru-en is collected from a combination of wikipedia articles and reddit articles. target side sentences are collected by translating source side sentences using nmt models and each sentence pair is annotated by at least three professional translators with a score between 0-100 according to the perceived translation quality. systems are required to inference z-standardized direct assessment (da) scores for 1k blind test set for each language pair. this year's task also include zero-shot scenario for 4 new language pairs: english-czech (en-cs), english-japanese (en-ja), pashto-english (ps-en), and khmer-english (km-en). as additional resource, participants are also provided with parallel data used to train nmt models (except for ru-en and zero-shot language pairs) and nmt models used to generate target side sentences of the dataset. ## approach figure 1 summarizes our approach. below we describe relevant components to our sentence-level qe model. ## settings for all training phases and experiments, we train our model in data parallelism on multiple nvidia tesla v100 gpus for 3 epochs with batch size of 8 and is optimized with adam (kingma and ba, 2015) with a learning rate of 7e -6 . dropout @xcite with 0.15 is applied to activation function outputs in equation 1. each model variant is trained 3 times with different random seeds, and for each model variant the best performing system in terms of pearson's correlation coefficient is reported. all models trained within the scope of this paper are multilingual qe models. we concatenate the train set of each individual language pair to create a single multilingual train set for training. we apply the same for dev and test20 set such that validation, model selection and evaluation can be performed at a multilingual level. ## results in this section, we present results of our architectures described in §3. pearson's correlation coefficient between predictions and gold standard scores is the main evaluation metric to measure performance; this year's task also considers model size to rank systems. our final compressed model base large +mtl+ikt @xmath2. we submit two systems: base large +mtl+ikt and base large +mtl+ikt+kd. systems can be evaluated on two ranking schemes: r1 indicates overall ranking foot_1 which considers both model performance and size, while r2 foot_2 ranks systems based only on model performance. as shown in we observe in ## conclusions in this paper, we describe our submission to the wmt 2021 quality estimation task 1: sentence-level direct assessment. we introduce a qe model architecture trained with multi-task objective and show improvements in performance. we show that iterative knowledge transfer techniques applied in qe tasks can further improve model's performance and demonstrate that knowledge distillation is effective for building a competitive lighter-weight qe model, making it more suitable for practical use. although our submitted systems show strong performance in general, we observe that our compressed model becomes relatively less competitive under zero-shot settings. further analysis of this phenomenon and improvements on zero-shot are challenges that we need to overcome in future work.