NJUNLP ’s Participation for the WMT 2022 Quality Estimation Shared Task (2022)

Abstract: This paper presents submissions of the NJUNLP team in WMT 2022Quality Estimation shared task 1, where the goal is to predict the sentence-level and word-level quality for target machine translations. Our system explores pseudo data and multi-task learning. We propose several novel methods to generate pseudo data for different annotations using the conditional masked language model and the neural machine translation model. The proposed methods control the decoding process to generate more real pseudo translations. We pre-train the XLMR-large model with pseudo data and then fine-tune this model with real data both in the way of multi-task learning. We jointly learn sentence-level scores (with regression and rank tasks) and word-level tags (with a sequence tagging task). Our system obtains competitive results on different language pairs and ranks first place on both sentence- and word-level sub-tasks of the English-German language pair.

## introduction quality estimation (qe) of machine translation (mt) is a task to predict the quality of translations at run-time without relying on reference translations @xcite . this paper describes the contribution of the njunlp team to the wmt2022 qe shared task @xcite on sentence-and word-level sub-tasks (task 1) 1 . for the sentence-level task, participating systems are required to predict the quality score for each translation output, and all scores are standardized using the z-score by the rater. the result is evaluated using spearman's rank correlation coefficient as the primary metric. for the word-level task, participating systems are required to tag each token of the translation output with ok and bad. the bad tag denotes this token is wrong, or there is one or more missing token(s) on the left side. the result is evaluated in terms of matthews correlation coefficient (mcc) as the primary metric. inspired by directqe @xcite , we further explore pseudo data and multi-task learning for the qe shared task. our main contributions are as follows: our system obtains competitive results on different language pairs. moreover, we rank first place on both sentence-and word-level of the english-german language pair with the spearman score of 63.47 (+1.33 than the second best system) and mcc score of 35.19 (+3.33). ## methods to handle the few-shot and zero-shot settings, we follow directqe @xcite framework. specifically, we first generate pseudo data using parallel data, then pre-train the qe model with generated data, and fine-tune the pre-trained model with real qe data if provided. we will describe these steps as follows. ## conclusion we present njunlp's work to the wmt 2022 shared task on quality estimation. we propose several novel pseudo data generation methods to bridge the gaps between existing pseudo data and real qe data. to learn the ordinal information, we extend multi-task learning for the qe task with the rank task. we also explore the pe data for the mqm annotation sub-task and propose to ensemble output scores with different scales using the z-score. experiments show that our pseudo data significantly improve the performance over the baseline. meanwhile, rank loss and pe data do help. in future research, we will conduct more ablation studies to reveal the contributions of each part.