Unsupervised Quality Estimation for Neural Machine Translation (2020)

Abstract: Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation, and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By utilizing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivaling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.

## introduction with the advent of neural models, machine translation (mt) systems have made substantial progress, reportedly achieving near-human quality for high-resource language pairs @xcite . however, translation quality is not consistent across language pairs, domains, and datasets. this is problematic for low-resource scenarios, where there is not enough training data and translation quality significantly lags behind. additionally, neural mt (nmt) systems can be deceptive to the end user as they can generate fluent translations that differ in meaning from the original @xcite . thus, it is crucial to have a feedback mechanism to inform users about the trustworthiness of a given mt output. quality estimation (qe) aims to predict the quality of the output provided by an mt system at test time when no gold-standard human translation is available. state-of-the-art (sota) qe models require large amounts of parallel data for pretraining and in-domain translations annotated with quality labels for training @xcite . however, such large collections of data are only available for a small set of languages in limited domains. current work on qe typically treats the mt system as a black box. in this paper we propose an alternative glass-box approach to qe that allows us to address the task as an unsupervised problem. we posit that encoder-decoder nmt models @xcite @xcite offer a rich source of information for directly estimating translation quality: (a) the output probability distribution from the nmt system (i.e., the probabilities obtained by applying the softmax function over the entire vocabulary of the target language); and (b) the attention mechanism used during decoding. our assumption is that the more confident the decoder is, the higher the quality of the translation. while sequence-level probabilities of the top mt hypothesis have been used for confidence estimation in statistical mt @xcite , the output probabilities from deep neural networks (nns) are generally not well calibrated, that is, not representative of the true likelihood of the predictions @xcite @xcite . moreover, softmax output probabilities tend to be overconfident and can assign a large probability mass to predictions that are far from the training data @xcite . to overcome such deficiencies, we propose ways to exploit output distributions beyond the top-1 prediction by exploring uncertainty quantification methods for better probability estimates @xcite . in our experiments, we account for different factors that can affect the reliability of model probability estimates in nns, such as model architecture, training, and search @xcite . in addition, we study attention mechanism as another source of information on nmt quality. attention can be interpreted as a soft alignment, providing an indication of the strength of relationship between source and target words @xcite . although this interpretation is straightforward for nmt based on recurrent neural networks (rnn) @xcite , its application to current sota transformer models with multihead attention @xcite is challenging. we analyze to what extent meaningful information on translation quality can be extracted from multihead attention. to evaluate our approach in challenging settings, we collect a new dataset for qe with 6 language pairs representing nmt training in high, medium, and low-resource scenarios. to reduce the chance of overfitting to particular domains, our dataset is constructed from wikipedia documents. we annotate 10k segments per language pair. by contrast to the vast majority of work on qe that uses semi-automatic metrics based on post-editing distance as gold standard, we perform quality labeling based on the direct assessment (da) methodology @xcite , which has been widely used for popular mt evaluation campaigns in the recent years. at the same time, the collected data differs from the existing datasets annotated with da judgments for the well known wmt metrics task foot_0 in two important ways: we provide enough data to train supervised qe models and access to the nmt systems used to generate the translations, thus allowing for further exploration of the glass-box unsupervised approach to qe for nmt introduced in this paper. our main contributions can be summarized as follows: (i) a new, large-scale dataset for sentence-level foot_1 qe annotated with da rather than post-edit ing metrics ( §4); (ii) a set of unsupervised quality indicators that can be produced as a by-product of nmt decoding and a thorough evaluation of how they correlate with human judgments of translation quality ( §3 and §5); (iii) the first attempt at analysing the attention distribution for the purposes of unsupervised qe in transformer models ( §3 and §5); and (iv) the analysis on how model confidence relates to translation quality for different nmt systems ( §6). our experiments show that unsupervised qe indicators obtained from well-calibrated nmt model probabilities rival strong supervised sota models in terms of correlation with human judgments. currently, the best performing approaches to qe use nns to learn useful representations for source and target sentences @xcite @xcite . a notable example is the predictor-estimator (predest) model @xcite , which consists of an encoder-decoder rnn (predictor) trained on parallel data for a word prediction task and a unidirectional rnn (estimator) that produces quality estimates leveraging the context representations generated by the predictor. despite achieving strong performances, neural-based approaches are resource-heavy and require a significant amount of in-domain labeled data for training. they do not use any internal information from the mt system. existing work on glass-box qe is limited to features extracted from statistical mt, such as language model probabilities or number of hypotheses in the n-best list @xcite . the few approaches for unsupervised qe are also inspired by the work on statistical mt and perform significantly worse than supervised approaches @xcite @xcite . for example, @xcite use lexical translation probabilities from word alignment models and language model probabilities. their unsupervised approach averages these features to produce the final score. however, it is largely outperformed by the neural-based supervised qe systems @xcite . the only works that explore internal information from neural models as an indicator of translation quality rely on the entropy of attention weights in rnn-based nmt systems @xcite . however, attention-based indicators perform competitively only when combined with other qe features in a supervised framework. furthermore, this approach is not directly applicable to the sota transformer model that uses multihead attention mechanism. recent work on attention interpretability showed that attention weights in transformer networks might not be readily interpretable @xcite . @xcite show that different attention heads of transformer have different functions and some of them are more important than others. this makes it challenging to extract information from attention weights in transformer (see §5). to the best of our knowledge, our work is the first on glass-box unsupervised qe for nmt that performs competitively with respect to the sota supervised systems. ## unsupervised qe for nmt we assume a sequence-to-sequence nmt architecture consisting of encoder-decoder networks using attention @xcite . the encoder maps the input sequence @xmath0, . . . , x i into a sequence of hidden states, which is summarized into a single vector using attention mechanism @xcite . given this representation the decoder generates an output sequence @xmath1, . . . , y t of length t . the probability of generating y is factorized as: in what follows, we propose unsupervised quality indicators based on: (i) output probability distribution obtained either from a standard deter-ministic nmt ( §3.1) or (ii) using uncertainty quantification ( §3.2), and (iii) attention weights ( §3.3). ## experiments and results below we analyze how our unsupervised qe indicators correlate with human judgments. ## discussion in the previous section we studied the performance of our unsupervised quality indicators for different language pairs. in this section we validate our results by looking at two additional factors: domain shift and underlying nmt system. ## conclusions we have devised an unsupervised approach to qe where no training or access to any additional resources besides the mt system is required. besides exploiting softmax output probability distribution and the entropy of attention weights from the nmt model, we leverage uncertainty quantification for unsupervised qe. we show that, if carefully designed, the indicators extracted from the nmt system constitute a rich source of information, competitive with supervised qe methods. we analyzed how different mt architectures and training settings affect the relation between predictive probabilities and translation quality. we showed that improved translation quality does not necessarily imply a stronger correlation between translation quality and predictive probabilities. model ensemble have been shown to achieve optimal results both in terms of translation quality and when using output probabilities as an unsupervised quality indicator. finally, we created a new multilingual dataset for qe covering various scenarios for mt development including low-and high-resource language pairs. both the dataset and the mt models needed to reproduce the results of our experiments are available at https://github.com/ facebookresearch/mlqe . this work can be extended in many directions. first, our sentence-level unsupervised metrics could be adapted for qe at other levels (word, phrase, and document). second, the proposed metrics can be combined as features in supervised qe approaches. finally, other methods for uncertainty quantification, as well as other types of uncertainty, can be explored.