Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation (2021)

Abstract: Quality Estimation (QE) plays an essential role in applications of Machine Translation (MT). Traditionally, a QE system accepts the original source text and translation from a black-box MT system as input. Recently, a few studies indicate that as a by-product of translation, QE benefits from the model and training data’s information of the MT system where the translations come from, and it is called the “glass-box QE”. In this paper, we extend the definition of “glass-box QE” generally to uncertainty quantification with both “black-box” and “glass-box” approaches and design several features deduced from them to blaze a new trial in improving QE’s performance. We propose a framework to fuse the feature engineering of uncertainty quantification into a pre-trained cross-lingual language model to predict the translation quality. Experiment results show that our method achieves state-of-the-art performances on the datasets of WMT 2020 QE shared task.

## introduction the emergence of neural machine translation (nmt) has brought about a revolutionary change in translation technology, resulting in translation with much higher quality. even though nmt can produce a fairly smooth translations at present, it is still not error-free. the outputs of a machine translation (mt) system must be proofread by humans in a post-editing phase, especially in those scenes with zero tolerance for translation quality, such as in the legal domain. therefore, it is essential to find out how good or bad the translations produced by an mt system are at run-time. quality estimation (qe) aims to predict the quality of a mt system's output without any access to ground-truth translation references or human intervention. qe methods have been explored broadly @xcite @xcite on * indicates corresponding author. wmt's benchmark qe datasets foot_0 . typical topranked qe systems @xcite need a large amount of parallel corpora for pre-training and in-domain translation triplets of source texts, machine translations and corresponding quality labels/scores @xcite for qe fine-tuning. starting from 2019, in replacement of pre-training a model from scratch, state-of-theart (sota) qe systems @xcite have achieved better results via taking advantage of sota pre-trained neural network models such as mbert @xcite and xlm-r @xcite with transfer learning to qe tasks. in recent years, the information of the nmt systems and the corresponding training data are open to participants in wmt qe shared task, which is helpful for us to gain more qe insights. essentially, it extends the traditional qe "black-box" nmt, where any information of the mt system is unknown to the "glass-box" stage. in fact, the concept of "glass-box" qe features has been introduced by @xcite , which provides an indication of the confidence of a mt system by extracting the outputs of moses-like statistical machine translation (smt) systems, for example the word-and phrase-alignment information and n-gram language model (lm) probabilities. a qe system using these features and svm regression is considered to be the baseline model of the sentence-level tasks of wmt qe from 2013 to 2018. even though the lm probability, as one of the glass-box qe features, has been widely used to estimate confidence of smt systems @xcite , the performance of using these features alone is not good enough, that can be seen from the performance of baseline results @xcite . ## related work a variant of sentence-level task where the quality of machine translations is annotated with direct assessment (da), instead of hters @xcite based on human post-editing. at least three different raters rate the mt sentences according to a continuous scoring scale from 0 to 100 in the respect of translation quality. da scores are standardised using the z-score by rater to be the final qe prediction. da estimation 2 is much closer to practical applications of qe, because the human post-editing work is expensive under the actual production deployment cost control. therefore, 2 http://www.statmt.org/wmt20/ quality-estimation-task.html our experiment designs are focusing on the direct assessment tasks. most of previous work on qe is based on the studies of feature engineering that explore how to extract useful features as inputs from source and machine translations to estimate the translation quality by a feature-enriched model. such early work on qe, for instance, uses manually crafted features extracted from source and machine translations, and some "glass-box" features from smt systems to build svm regression models with rbf kernel @xcite . as time went by, with the development of deep learning applied in nlp, predictor-estimator architecture using neural net-works for qe was proposed @xcite , which relies highly on a pre-trained word prediction model with a bidirectional rnn structure, called predictor. its training requires large amount of parallel data. the latent representations generated by the word prediction model are treated as features to be fed into a downstream estimator mode for qe fine-tuning. motivated by transformer @xcite framework of neural machine translation, @xcite modified the rnn-based word predictor with a bidirectional transformer structure to make an improvement on word prediction, leading to better qe results. sota pre-trained models have achieved successes in various nlp tasks @xcite with transfer leaning. current sota qe systems @xcite @xcite , profit from sota pre-trained models to gain cross-lingual representations of source and machine translations and fine tune the model with additional layers to meet the goals of qe. they passingly remove the dependency of large amount of parallel data and ease the burden of pre-training complex neural network models. encouraged by the top ranked qe system @xcite , we design a featureenhanced model similar to their work, that also relies on the pre-trained xlm-r model, but it is enhanced by a mixture of other useful "glass-box" and "black-box" qe features. the model structure can be first glimpsed in figure 1 and details will be introduced in section 3. ## methodology in this section, we provide a complete view of our uncertainty quantification approaches: (1) the predictive information of softmax distribution from the nmt model is still used as a "glass-box" qe feature due to its indication in qe explored from previous work @xcite . we describe it simply in section 3.1; (2) stimulated by @xcite , more useful derivatives of mc dropout sampling for uncertainty quantification are investigated as "glass-box" features in section 3.2; (3) we extend the meaning of "glassbox" in a broader sense and shift our gaze from model confidence to data confidence in section 3.3. more creatively, (4) a combination of "glass-box" and "black-box" approaches is proposed to estimate the uncertainty of source texts in section 3.4. in particular, (5) the "black-box" approach utilizing a sota pre-trained nlp model in (4) can inherently estimate the confidence of the sources via the masking strategy in section 3.5. in section 3.6, a model enhanced by above uncertainty features is carried out for the final goal of quality estimation. ## conclusion in this paper, we extend previous work on "glassbox" qe for uncertainty quantification and explore how sota transfer learning method can benefit from uncertainty features. first, we re-organize "glass-box" features from the softmax distribution and monte carlo dropout sampling in previous work and derive useful variants. secondly, based on the information of training data of the nmt model, the "glass-box" features in the respect of data attributes are extracted to predict uncertainty as well. more importantly, we propose a new method, which utilizes the masking mechanism of the pre-trained model to quantify uncertainty through robustness testing via several pre-designed "glass-box" features. finally, we evaluate all the "black-box" and "glass-box" approaches by an uncertainty feature enhanced model on the benchmark da datasets of wmt 2020 qe shared task. the experimental results show that our proposed features for uncertainty estimation are effective, and the uncertainty feature-enhanced qe model is superior to sota qe systems.