Quality In, Quality Out: Learning from Actual Mistakes (2020)

Abstract: Approaches to Quality Estimation (QE) of machine translation have shown promising results at predicting quality scores for translated sentences. However, QE models are often trained on noisy approximations of quality annotations derived from the proportion of post-edited words in translated sentences instead of direct human annotations of translation errors. The latter is a more reliable ground-truth but more expensive to obtain. In this paper, we present the first attempt to model the task of predicting the proportion of actual translation errors in a sentence while minimising the need for direct human annotation. For that purpose, we use transfer-learning to leverage large scale noisy annotations and small sets of high-fidelity human annotated translation errors to train QE models. Experiments on four language pairs and translations obtained by statistical and neural models show consistent gains over strong baselines.

## introduction quality estimation (qe) for machine translation (mt) is the task of predicting the overall quality of an automatically generated translation e.g., on either word, sentence or document level @xcite . in opposition to automatic metrics and manual evaluation which rely on gold standard reference translations, qe models can produce quality estimates on unseen data, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. and at runtime. qe has already proven its usefulness in many applications such as improving productivity in post-editing of mt, and recent neuralbased approaches to qe have been shown to provide promising performance in predicting quality of neural mt output @xcite . qe models are trained under full supervision, which requires to have quality-labelled training data at hand. obtaining annotated data for all the domains and languages of interest is costly and often impractical. as a result, qe models can suffer from the same limitations as neural mt models themselves, such as drastic degradation of their performance on out-of-domain data. as an alternative, qe models are often trained under weak supervision, using training instances labelled from noisy or limited sources (e.g. data labelled with automatic metrics for mt). here, we focus on sentence-level qe, where given a pair of sentences (the source and its translation), the aim is to train supervised machine learning (ml) models that can predict a quality label as a numerical value. the most widely used label for sentence-level qe is the human-mediated translation edit rate (hter) @xcite , which represents the post-editing effort. hter consists of the minimum number of edits a human language expert is required to make in order to fix the translation errors in a sentence, taking values between 0 and 1. the main limitation of hter is that it does not represent an actual translation error rate, but its noisy approximation. the noise stems mostly from errors in the heuristics used to automatically align the machine translation and its post-edited version, but also from the fact that some edits represent preferential choices of humans, rather than errors. to overcome such limitations, qe models can be improved by using data that has been directly annotated for translation errors by human experts. figure 1 shows an example of the discrepancy between the hter score and the proportion of actual errors from expert annotation, for a raw translation and its post-edited version. annotations of mt errors usually follow finegrained error taxonomies such as the multidimensional quality metrics (mqm) framework @xcite . while such annotations provide highly reliable labelled data, they are more expensive to produce than hter. this often results in datasets that are orders of magnitude smaller than hter-based ones. this makes it hard to only use such high-quality resources for training neuralbased qe models, which typically require large amounts of training data. in this paper, we use transfer-learning to develop qe models by exploiting the advantages of both noisy and high-quality labelled data. we leverage information from large amounts of hter data and small amounts of mqm annotations to train more reliable sentence-level qe models. our aim is to predict the proportion of actual errors in mt outputs. more fine-grained error prediction is left for future work. ## transfer-learning approach we use inductive transfer-learning @xcite , where given a source learning task t s and a target task t t , the aim is to improve performance in the latter by re-using knowledge from t s , where t @xmath0. here, t s corresponds to predicting postediting effort based on noisy hter annotations, and t t to predicting the proportion of actual proportion of errors based on mqm annotations. ## results tables 3 and 4 show respectively the average absolute pearson's r correlation co-efficient and the root mean square error (the official metrics for this task @xcite ) between actual and predicted mqm error proportions in six combinations of mt models (pbmt, nmt) and language pairs @xcite . first, we observe that the baseline model (lr-qefeat) performs fairly well on predicting the proportion of errors, especially for the en-de and en-cs pbmt. however, it is not robust across language pairs and types of translation systems. second, the birnn-hter model, trained on hter data and used as is, is not able to predict the proportion of actual mqm errors. surprisingly, the birnn-mqm model trained on mqm data directly achieves relatively good performance for all language pairs. this seems to confirm that (i) the birnn architecture, as simple as it may be, allows to train models that perform well while keeping low the computational resources required; and (ii) that hter is a noisy approximation of the quality of a translation and post-edits are not actually well-aligned to actual translation errors. overall, the best performing model is birnn-mqm t l with transfer-learning and fine-tuning, while our hybrid model seems to further improve performance in predicting quality on statistical mt output. this is in line with recent findings demonstrating the benefits of feature-based approaches for predicting the quality of statistical mt, but not for predicting the quality of neural mt, which is better modelled with learned representations using neural networks @xcite . this also confirms our main hypothesis that noisy data, but from a closely related task, encapsulates useful information that our tl model is able to leverage. ## leveraging pre-trained token-level representations as reported in @xcite , state-ofthe-art models for supervised qe follow current trend in the nlp community in 2019: leveraging large-scale pre-trained language models to compute word-or sentence-level representations. following @xcite and their transformerbased predictor-estimator model, we considered two variants of our birnn-hter model introduced in section 3: lm-birnn by default, the weights of both the source and target bidirectional gru encoders of the birnn model are first randomly initiated and then learned, simultaneously, during training of the task at hand. in this variant, we first learn the weights of each encoder independently in a language modelling fashion with a cross-entropy loss, using the additional resources provided by the organisers of the wmt'18 qe shared task 6 . we then reuse the learned weights to initiate each encoder of the birnn model. ## conclusions we introduced a new task of predicting the proportion of actual errors in a translated sentence as an alternative to the commonly used noisy estimate hter. the reported results from using inductive transfer-learning are particularly encouraging considering the simplicity of our birnn model. our transfer-learning method helps to train models which are better at predicting the proportion of actual errors for different language pairs and trans-lation systems, compared to models trained on the target task only. however, whereas we were expecting to observe significant gains with the use of more advanced token-level pre-trained representations (here from bert), we report drastic degradation in performances for this configuration when re-purposing the qe models via transfer-learning. these somewhat counter-intuitive results are an indication that further work can be done in this area to refine our transfer-learning approach, as the use of large scale pre-trained representations has become a common practice in nlp applications, including qe. in addition to this, we plan in furture to estimate the quality of machine translation using more finegrained mqm annotations for subsentence-level qe.