NICT K yoto Submission for the WMT ’20 Quality Estimation Task: Intermediate Training for Domain and Task Adaptation (2020)

Abstract: This paper describes the NICT Kyoto submission for the WMT’20 Quality Estimation (QE) shared task. We participated in Task 2: Word and Sentence-level Post-editing Effort, which involved Wikipedia data and two translation directions, namely English-to-German and English-to-Chinese. Our approach is based on multi-task fine-tuned cross-lingual language models (XLM), initially pre-trained and further domain-adapted through intermediate training using the translation language model (TLM) approach complemented with a novel self-supervised learning task which aim is to model errors inherent to machine translation outputs. Results obtained on both word and sentence-level QE show that the proposed intermediate training method is complementary to language model domain adaptation and outperforms the fine-tuning only approach.

## introduction this paper presents the nict kyoto submission for the ninth edition of the quality estimation (qe) shared task organized at the fifth conference for machine translation (wmt'20). the goal of qe is to estimate the quality of machine translation (mt) output without using a translation reference. the system developed for the task and described in this paper is based on pre-trained cross-lingual language models (xlm) @xcite , domain and task-adapted through intermediate training @xcite and fine-tuned in a multi-task fashion for the sentence and word-level qe objectives. it was shown during the qe shared task at wmt'19 @xcite that pre-trained language models (lm) fine-tuned for qe reach state-of-the-art results at the levels of sentence and word following the predictor-estimator architecture @xcite or using a fully endto-end approach @xcite @xcite . however, fine-tuning pretrained lms is highly unstable when the dataset used for fine-tuning is small @xcite , which is usually the case for qe, as annotated datasets are scarce and expensive to produce, and wmt qe datasets are no exceptions (the shared task datasets are presented in our approach is complementary to lm domain adaptation and we propose to conduct both tasks, i.e. domain and final task adaptation, jointly during intermediate training and prior to fine-tuning. more details about the intermediate training approach, including masked lm modifications and the datasets used, are presented in section 2, followed by the qe task fine-tuning and evaluation in section 3. finally, a conclusion and future work are given in section 4. ## qe fine-tuning the objective of fine-tuning masked lms for qe is to predict sentence-level human translation edit rate (hter) and word-level good and bad classes. 3 note that in our models, for the target sequence word-level qe, we considered gaps between target words (missing translations) as part of the target sequence and did not use a loss nor a training objective specific to gaps. ## conclusion we have presented in this paper the nict kyoto submission for the wmt'20 qe shared task on predicting post-editing effort at the sentence and word-level. our submissions consisted of ensembles of several fine-tuned masked lms, pre-trained using the translation lm objective, domain and task adapted in a self-supervised fashion using domainrelevant data and a modified masking approach during intermediate training. this novel intermediate training objective allows for a smooth transition from a pre-trained masked lm towards the final qe task without requiring annotated data. we have shown empirically that both domain and task adaptation reach good results compared to out-of-the-box pre-trained models and compared to fine-tuning only. our final submissions were ranked among the top systems both at the sentence and word-level for two language pairs.