Papago’s Submission to the WMT 22 Quality Estimation Shared Task (2022)

Abstract: This paper describes anonymous submission to the WMT 2022 Quality Estimation shared task. We participate in Task 1: Quality Prediction for both sentence and word-level quality prediction tasks. Our system is a multilingual and multi-task model, whereby a single system can infer both sentence and word-level quality on multiple language pairs. Our system’s architecture consists of Pretrained Language Model (PLM) and task layers, and is jointly optimized for both sentence and word-level quality prediction tasks using multilingual dataset. We propose novel auxiliary tasks for training and explore diverse sources of additional data to demonstrate further improvements on performance. Through ablation study, we examine the effectiveness of proposed components and find optimal configurations to train our submission systems under each language pair and task settings. Finally, submission systems are trained and inferenced using K-folds ensemble. Our systems greatly outperform task organizer’s baseline and achieve comparable performance against other participants’ submissions in both sentence and word-level quality prediction tasks.

## introduction quality estimation (qe) evaluates the quality of machine translated output without human reference translation @xcite . apart from qe models' most obvious usage as being reference-less metrics for mt, it has variety of other applications in machine translation (mt) pipeline including but not limited to: parallel corpus filtering @xcite , curriculum learning @xcite and decoding @xcite . high performance in both sentence and wordlevel quality prediction tasks is achieved by incorporating plm as part of qe model architecture as demonstrated in previous wmt qe findings * these authors contributed equally to this work @xcite . previous years' top performers generally incorporate various data augmentation techniques in order to account for limited amount of annotated gold data @xcite . multi-task training, ensembling, or incorporating features extracted from external models are few other popular approaches that proved to work well @xcite @xcite . our system is a multilingual and multi-task model, whereby a single system can infer both sentence and word-level quality on multiple language pairs. our system's architecture ( §3.1) consists of plm and task layers, and is jointly optimized for both sentence and word-level quality prediction tasks ( §3.2.1) using multilingual dataset. we propose novel auxiliary tasks ( §3.2.2) for training and explore diverse sources of additional data ( §3.3) to demonstrate further improvements on performance. through ablation study ( §5), we evaluate each components of our proposed model and use optimal configurations to train our submission systems under each language pair and task settings. finally, submission systems are trained and inferenced using k-folds ensemble ( §3.4.2). our systems greatly outperform task organizer's baseline and perform very competitively against other participants' submissions in both sentence and word-level quality prediction tasks. ## quality prediction task and dataset in this section we briefly overview two subtasks and their datasets in task 1. apart from provided gold data as described below, participants are allowed to leverage additional sources of data. ## approach below we describe relevant components of our proposed qe model. ## ablation in this section, we present ablation study of individual components to our model described in §3. all evaluations for ablation in ## results ## conclusions in this work, we describe our system submission to the wmt 2022 quality estimation shared task. our system is a multilingual and multi-task model for both sentence and word level quality prediction tasks. we demonstrate through ablation study that additional training objectives and data can further improve quality prediction performance. our final model is trained and inferenced using k-folds ensemble which show remarkable performance in all language pairs and tasks. however, we find that multi-task or auxiliary-task training do not help in word-level as much as they do in sentence-level quality prediction. further analysis to understand the dynamics of training with multiple objectives and improvements on word-level quality prediction are challenges that we need to overcome in future work.