Intermediate Self-supervised Learning for Machine Translation Quality Estimation (2020)

Abstract: Pre-training sentence encoders is effective in many natural language processing tasks including machine translation (MT) quality estimation (QE), due partly to the scarcity of annotated QE data required for supervised learning. In this paper, we investigate the use of an intermediate self-supervised learning task for sentence encoder aiming at improving QE performances at the sentence and word levels. Our approach is motivated by a problem inherent to QE: mistakes in translation caused by wrongly inserted and deleted tokens. We modify the translation language model (TLM) training objective of the cross-lingual language model (XLM) to orientate the pre-trained model towards the target task. The proposed method does not rely on annotated data and is complementary to QE methods involving pre-trained sentence encoders and domain adaptation. Experiments on English-to-German and English-to-Russian translation directions show that intermediate learning improves over domain adaptated models. Additionally, our method reaches results in par with state-of-the-art QE models without requiring the combination of several approaches and outperforms similar methods based on pre-trained sentence encoders.

## introduction machine translation (mt) quality estimation (qe) @xcite @xcite aims at evaluating the quality of translation system outputs without relying on translation references, which are required by automatic evaluation metrics such as bleu @xcite or ter @xcite . current state-of-the-art (sota) qe approaches have switched from hand-crafted features to large data-driven neural-based models @xcite . best performing qe methods from the latest wmt qe shared task @xcite are based on two approaches: predictor-estimator @xcite and qe-specific output layers on top of pre-trained contextual embeddings @xcite . while both approaches make use of sentence encoder models, such as bert @xcite or xlm @xcite , only the second approach allows for straightforward end-to-end learning and direct fine-tuning of the pre-trained language model. however, fine-tuning pre-trained models is highly unstable when the dataset is small @xcite , which is the case in qe for mt as annotated datasets are scarse. to provide a smooth transition between pre-training and fine-tuning, an intermediate training step has been proposed @xcite , using large scale labeled data relevant to the target task. this approach is nonetheless limited by its reliance on annotated data for supervised learning. in this paper, we focus on providing a novel self-supervised intermediate training approach to adapt a pre-trained model to qe by modifying the popular masked lm objective. our model is based on the translation language model of xlm with a novel training objective which simulate common mistakes observed in translations, namely deletions and insertions in translations. our contribution focuses on intermediate training of pre-trained lms for qe and is twofold: evaluating the impact of domain adaptation on pre-trained model and designing a self-supervised learning for intermediate training. the approach proposed in this paper is detailed in section 2, and the experimental setup is introduced in section 3. the results are presented in section 4 before concluding in section 5. pre-trained language model (lm) fine-tuning has shown to improve the results of many natural language processing tasks such as grammatical sentence classification, paraphrases detection or textual entailment to name a few popular tasks @xcite . some of the prevailing fine-tuned pre-trained models studied in the literature are bert @xcite and xlm @xcite , among others. at the core of these approaches are similar lm techniques, using the sequentiality of languages to learn probabilities over sequences (x) of words (x i , @xmath0; n]) as in p(x) = n @xmath1, ..., x n ) (causal lm) or randomly masking some input tokens and learning to retrieve them based on both left and right contexts (masked lm). the masked lm approach introduced in bert was extended in xlm to learn relations between translated sentences based on bilingual parallel corpora, integrating a new training objective called translation lm. the translation lm is particularily suited for qe, as it allows the model to learn bilingual context information when predicting masked tokens. however, fine-tuning pre-trained models was shown to be unstable with small datasets @xcite . this instability finds some explainations in recent work @xcite . a proposed approach to reduce instability is to use a second stage pre-training step, between the initial lm training and the final task-oriented fine-tuning. it is based on a large amount of labeled data for a task related to the target objective @xcite . in addition to providing a smooth transition between initial pre-training and fine-tuning by coercing the model towards the final training objective, the intermediate step allows for domain adaptation when there is a domain mismatch between the datasets used for each training step @xcite . as a variant to the intermediate training approach, which originally makes use of labeled data, we propose a self-supervised intermediate step, alleviating the need for annotated data. we aim at jointly adapting a pre-trained lm to the domain and to the final task through intermediate training. for domain adaptation, we relied on continued training by using a dataset relevant to the final task @xcite . for task adaptation, we modified the masked lm approach used in the masked translation lm task. more precisely, in addition to predicting the masked tokens in the input parallel sequences, we introduced fake masks for which a null token has to be predicted. this method forces the model to distinguish between missing words, which often occure in translated sentences when source words are not translated, and wrongly introduced words, similar to mistranslations when source words are wrongly translated. the proposed intermediate self-supervised learning task is illustrated in figure 1 . our model is inspired by the approach of @xcite , however, it is based on xlm in our work instead of bert. to allow for sentence and word-level qe from pre-trained lms, two parallel outputs on top of xlm composed of parametrised linear layers were added. the first output layer corresponds to the word-level qe task, takes as input the word-level final hidden states given by xlm, and outputs word-level probabilities for the two classes (good and bad) using a softmax function. the second output layer corresponds to the sentence-level qe task, takes as input the final hidden state of the first token (@xmath2, and outputs a sequence-level probability using a sigmoid function. two loss functions, calculated based on each output and the training gold labels, were linearly summed to compose the final loss. the loss was then back-propagated, whether through the output layers only when the xlm parameters were frozen, or through all layers otherwise, to update the model parameters. more details about the training procedure and neural network hyperparameters are presented in section 3.3. ## experimental setup this section presents our experimental setup, including the qe task and the evaluation methodology, as well as the datasets used. ## conclusion we presented in this paper a novel intermediate self-supervised learning approach applied to a pre-trained language model for mt qe. results show that our approach is helfpul jointly with domain adaptation and leads to qe performances on par with sota methods, without involving the combination of a large amount of approaches and models. we believe that the results obtained on the english-russian pair can be improved by using a more accurate data selection method from non-synthetic domain relevant parallel corpora for the intermediate training task and will explore this research direction in future work. our proposed approach can be seen as an extension of the work done by @xcite , where the authors evaluated the impact of continued pre-training lms on domain and task relevant data, but keeping the training objective similar to the pre-training step. our approach differs from this work by adding a training component relevant to the final qe task, i.e. fake masking. we would like to explore masking variants in future work, for instance by allowing variable masking spans such as the ones proposed in @xcite . finally, an extension of our work inspired by @xcite is to explore the pre-training masking hyper-parameters, namely the number of tokens masked and randomly replaced by other tokens sampled from the vocabulary. because detecting mistranslations in mt output is crucial for qe, increasing the number of replaced tokens would force the model to learn more accurately which tokens are mistranslated or not. this hyper-parameter search could be part of the intermediate training procedure to avoid increasing the computational costs of large lms pre-training.