Self-Supervised Quality Estimation for Machine Translation (2021)

Abstract: Quality estimation (QE) of machine translation (MT) aims to evaluate the quality of machine-translated sentences without references and is important in practical applications of MT. Training QE models require massive parallel data with hand-crafted quality annotations, which are time-consuming and labor-intensive to obtain. To address the issue of the absence of annotated training data, previous studies attempt to develop unsupervised QE methods. However, very few of them can be applied to both sentence- and word-level QE tasks, and they may suffer from noises in the synthetic data. To reduce the negative impact of noises, we propose a self-supervised method for both sentence- and word-level QE, which performs quality estimation by recovering the masked target words. Experimental results show that our method outperforms previous unsupervised methods on several QE tasks in different language pairs and domains.

## introduction in recent years, neural approaches @xcite @xcite have significantly improved the quality of machine translation (mt). despite their apparent success, neural machine translation (nmt) systems still inevitably generate erroneous translations in real-world scenarios @xcite , especially for low-resource language pairs. therefore, the evaluation of translation quality plays an important role in some applications of mt. for example, in computer-assisted translation (cat) @xcite , the evaluation of translation quality can significantly reduce human efforts for postediting @xcite . quality estimation (qe) of mt aims to evaluate the quality of the outputs of an mt system without references. training qe models often requires massive parallel data, which are composed of authentic source sentences and machine-translated target sentences with quality annotations produced by manual evaluation or human post-editing @xcite @xcite . as obtaining such annotated data is time-consuming and labor-intensive in practice, unsupervised qe has received increasing attention @xcite @xcite @xcite . most of the aforementioned methods use various features to conduct unsupervised qe @xcite @xcite @xcite . these methods are simple and effective but limited to sentence-level tasks. compared with sentencelevel qe, word-level qe can provide more finegrained quality information @xcite , and thus it can better assist post-editing in cat when combined with sentence-level qe. recently, @xcite use synthetic data to train unsupervised qe models, which can be applied for both sentence-and word-level tasks. specifically, they construct synthetic target sentences using mt models or masked language models (mlms) and generate quality annotations by comparing the synthetic target sentences with the references using the ter tool @xcite . however, the method proposed by @xcite still has two major weaknesses. first, synthetic data contain biased noise and may negatively affect the model performance. on the one hand, the differences between mt outputs and references are usually larger than the differences between mt outputs and their post-editions @xcite , and thus more errors will be annotated in the synthetic data. on the other hand, sentences that are our method performs quality estimation by checking whether the masked target words can be successfully recovered using the source sentence and the observed target words. masked words are highlighted by shading. in this work, we propose a self-supervised qe method to overcome the aforementioned weaknesses. the basic idea is to mask some target words in the machine-translated sentence and use the source sentence and the observed target words to recover the masked target words. intuitively, a target word is correct if it can be recovered according to its surrounding context. for example, in figure 1 , since the masked target word "er" can be successfully recovered but another masked target word "lieder" is not identical to the recovered word "musik", we identify "er" as correct and "lieder" as erroneous. based on this intuition, our method estimates the translation quality of the target words by checking whether they can be correctly recovered. finally, we obtain the sentencelevel quality score by summarizing the word-level predictions. obviously, our method is not affected by the noise and is easier to train, since it involves no synthetic data. experimental results show that our self-supervised method outperforms previous unsupervised methods. ## quality estimation for machine translation quality estimation for machine translation aims to evaluate the quality of machine-translated sentences without using references. currently, there are different types of qe tasks, including sentence-, word-, phrase-and document-level qe. ity annotations are generated by comparing the machine-translated target sentences with their posteditions using the ter tool @xcite . for word-level annotations, each target word is annotated with "ok" or "bad", where "ok" denotes correct words and "bad" denotes erroneous words. for sentence-level annotations, target sentences are annotated with human translation error rate (hter) scores, which measure the percentage of human edits to correct mt outputs: according to the equation above, sentence-level quality scores are calculated based on the wordlevel errors in the target sentences. in other words, hter scores can be approximately regarded as a summary of word-level quality tags. ## self-supervised quality estimation our self-supervised qe method is implemented based on the architecture of mlm @xcite (section 3.1). we train the model to recover the masked target words in the authentic parallel corpora and estimate the translation quality by recovering the masked target words in the target sentence (section 3.2). besides, monte carlo (mc) dropout @xcite ) is utilized to better calculate quality scores (section 3.3). ## conclusion and future work we have presented a self-supervised method for quality estimation of machine-translated sentences. the central idea is to perform quality estimation by recovering masked target words using the surrounding context. our method is easy to implement and is not affected by noisy synthetic data. experimental results show that our method outperforms previous unsupervised qe methods. in the future, we plan to extend our self-supervised method to phrase-and document-level tasks. algorithm 1 calculating quality scores with monte carlo dropout input: source sentence x, target sentence ŷ = (ŷ 1 , • • • , ŷt ), number of samples for each target token n , number of estimations n ′ , model parameter θ output: quality scores of all target tokens score(ŷ 1 ), • • • , score(ŷ t )