Two-Phase Cross-Lingual Language Model Fine-Tuning for Machine Translation Quality Estimation (2020)

Abstract: In this paper, we describe the Bering Lab’s submission to the WMT 2020 Shared Task on Quality Estimation (QE). For word-level and sentence-level translation quality estimation, we fine-tune XLM-RoBERTa, the state-of-the-art cross-lingual language model, with a few additional parameters. Model training consists of two phases. We first pre-train our model on a huge artificially generated QE dataset, and then we fine-tune the model with a human-labeled dataset. When evaluated on the WMT 2020 English-German QE test set, our systems achieve the best result on the target-side of word-level QE and the second best results on the source-side of word-level QE and sentence-level QE among all submissions.

## introduction machine translation quality estimation (qe) is the task of estimating the quality of machine-translated (mt) output given just the source text at various granularity levels (word, sentence, and document) @xcite . word-level qe can be divided into target-side and source-side tasks. on the target-side, the goal is to predict whether each word in the mt sentence is ok or bad and whether there are missing words between each word. the goal on the source-side is to predict whether each word in the source sentence is correctly translated or not. on the other hand, sentence-level qe aims to predict the human translation error rate (hter) @xcite of the mt sentence, which measures the required amount of human editing to fix the mt sentence. in this paper, we propose a cross-lingual language model fine-tuning approach with a few additional parameters for word-level and sentence-level qe. as a pre-trained cross-lingual language model, we use xlm-roberta (xlm-r) @xcite , which shows state-of-the-art performance for a wide range of cross-lingual transfer tasks. in addition, since labeling the qe dataset requires a large amount of human labor, we generate and utilize a huge artificial qe dataset to improve the performance of our model. our contributions are summarized as follows. in the experiment using the wmt 2020 english-german word-level qe test set, we achieve an mcc of 0.597 and 0.454 for the target-side and source-side, respectively, showing the best and second best performance among all submitted systems, respectively. for the sentence-level qe test set, we achieve a pearson correlation of 0.723, which ranks second among all submissions. ## methodology we fine-tune xlm-r @xcite with a few additional parameters for sentence-level and word-level qe as described in figure 1 . we train our model in two phases: 1) pre-training with a huge artificial dataset and 2) fine-tuning with a human-labeled dataset. ## conclusion this paper describes bering lab's submissions to the wmt 2020 qe shared task. we propose a twophase cross-lingual language model fine-tuning approach for word-level and sentence-level translation quality estimation. the experimental results show that pre-training with an artificially generated dataset significantly improves performance for both tasks. overall, our submitted systems achieve the best result on the target side of word-level qe and the second best results on the source side of word-level qe and the sentence-level qe among all submissions.