The N iu T rans System for the WMT 20 Quality Estimation Shared Task (2020)

Abstract: This paper describes the submissions of the NiuTrans Team to the WMT 2020 Quality Estimation Shared Task. We participated in all tasks and all language pairs. We explored the combination of transfer learning, multi-task learning and model ensemble. Results on multiple tasks show that deep transformer machine translation models and multilingual pretraining methods significantly improve translation quality estimation performance. Our system achieved remarkable results in multiple level tasks, e.g., our submissions obtained the best results on all tracks in the sentence-level Direct Assessment task.

## introduction quality estimation (qe) evaluates the quality of machine translation output without human reference translations @xcite . it has a wide range of applications in post-editing and quality control for machine translation. we participated in all tasks and language pairs at the wmt 2020 qe shared task foot_1 , including sentence-level direct assessment tasks, word and sentence-level post-editing effort tasks, and document-level qe tasks. we investigated transfer learning and ensemble methods using recently proposed multilingual pre-trained models @xcite as well as deep transformer models @xcite . our main contributions are as follows: results on different level tasks show that our methods are very competitive. our submissions achieved the best pearson correlation on all language pairs of the sentence-level direct assessment task and the best results on english-chinese post-editing effort tasks. we present methods for the sentence-level direct assessment task in §2. then in §3 and §4, we describe our approaches to post-editing tasks and document-level tasks, respectively. system ensemble methods are discussed in §5. we show the detail of our submissions and the results in §6. we conclude and discuss future work in §7. ## sentence-level direct assessment task the sentence-level direct assessment task is a new task where sentences are annotated with direct assessment (da) scores by professional translators rather than post-editing labels. da scores for each sentence are rated from 0 to 100, and participants are required to score sentences according to z-standardized da scores. the da task consists of seven tracks for different language pairs and one multilingual track. submissions were evaluated in terms of pearson's correlation metric for the da prediction against human da (z-standardized mean da score, i.e., z-mean). ## word and sentence-level post-editing effort task this task consists of the word-and sentence-level tracks to evaluate post-editing effort. the wordlevel tasks predicts ok or bad tags in both source and target sequences. it evaluates the matthews correlation coefficient foot_4 (mcc) for tags. the sentencelevel task predicts hter scores, which is the ratio between the number of edits needed and the reference translation length. it evaluates pearson's correlation for the hter prediction. there are two language pairs in both the word-and sentencelevel tasks, including english-german (en-de) and english-chinese (en-zh). ## system ensemble in addition to training models for each task, we also explored effective ensemble methods to combine outputs for different level tasks. ## conclusion this paper describes the submissions of the niu-trans team to the wmt 2020 qe task. we explored the combination of transfer learning, multitask learning, and model ensemble. different level tasks show that deep transformer nmt models and multilingual pretraining methods significantly boost qe models' performance. although our system achieved impressive results in all tasks and language pairs, there are still many problems. for instance, the translation quality estimation of low-resource languages performs much better than that of high-resource. it raises the concern of whether our model learns the evaluation criteria instead of memorizing data, as suggested in @xcite . besides, strong nmt models help quality estimation, but can we use qe models to improve nmt systems' learning? we plan to answer these questions in the future and promote the joint improvement of qe and nmt models. china (no.2019qy1801). the authors would like to thank anonymous reviewers for their comments.