Learning from Past Mistakes: Quality Estimation from Monolingual Corpora and Machine Translation Learning Stages (2023)

Abstract: Quality Estimation (QE) of Machine Translation output suffers from the lack of annotated data to train supervised models across domains and language pairs. In this work, we describe a method to generate synthetic QE data based on Neural Machine Translation (NMT) models at different learning stages. Our approach consists in training QE models on the errors produced by different NMT model checkpoints, obtained during the course of model training, under the assumption that gradual learning will induce errors that more closely resemble those produced by NMT models in adverse conditions. We test this approach on English-German and Romanian-English WMT QE test sets, demonstrating that pairing translations from earlier checkpoints with translations of converged models outperforms the use of reference human translations and can achieve competitive results against human-labelled data. We also show that combining post-edited data with our synthetic data yields to significant improvements across the board. Our approach thus opens new possibilities for an efficient use of monolingual corpora to generate quality synthetic QE data, thereby mitigating the data bottleneck.

## introduction significant improvements have been achieved in machine translation (mt) in recent years, in particular with the advent of neural machine translation (nmt) @xcite @xcite . however, the quality of automated translations can vary significantly depending on training data volumes, domain of application, language pairs or the complexity of specific source segments. machine translation errors can significantly increase the risks and costs of using mt and the automatic estimation of mt quality becomes increasingly necessary to pinpoint or discard erroneous automatic translations. traditionally, the quality of mt output has been assessed against human references, via automated metrics such as bleu @xcite or ter @xcite . however, such references are not always available and are costly to produce, which has led to the development of quality estimation (qe) approaches based on the sole properties of the source and machine-translated sentences @xcite . most approaches to qe are based on supervised learning, traditionally via feature engineering @xcite , and, in recent years via neural models @xcite @xcite . although they provide the most accurate estimates to date, supervised methods depend on human annotations or post-edited translations to perform the task. the cost of producing quality qe training datasets hinders the development of qe models for the large number of possible domains and language pairs. two main alternatives address the lack of training qe data. on the one hand, unsupervised and self-supervised approaches @xcite @xcite @xcite discard the need for qe training data altogether, but typically fail to consistently meet the accuracy of supervised alternatives or may require access to additional information such as internal states of the mt model. on the other hand, methods that exploit synthetic training data have also been proposed in recent years, leveraging parallel dataset references. under this approach, parallel training data can be exploited, for instance, by taking a target reference translation as the approximated post-edited version of a machine-translated source segment and generating artificial qe labels @xcite . the two may differ significantly however, thereby introducing noise in the qe training data. alternatively, synthetic data can be generated by devising qe error generation pipelines from the parallel data @xcite , although this requires approximating errors that may not correspond to actual mt ones. in this work, we describe and evaluate a novel approach to synthetic qe data generation by exploiting the actual errors committed by nmt models at different learning stages. the hypothesis underlying this approach is that this type of errors might resemble more closely the errors produced by mt systems in scenarios where they typically fail, such as language pairs for which parallel training data are insufficient, or domains that deviate from those represented in the training sets. to test this hypothesis, we train nmt models on generic parallel data and select model checkpoints of varying quality to contrast their translations with either human reference translations or translations from the best converged nmt models. the generated synthetic data are then used to train neural qe estimators, either in isolation or in combination with human-generated data. we demonstrate the potential of this novel approach on wmt 2021 datasets in english-german and romanian-english. we notably show that it outperforms the use of human reference translations, directly or via self-supervised learning, is competitive with the use of human post-edited data, and can complement the latter to achieve further gains in qe accuracy. additionally, contrasting checkpoint translations with those of converged nmt models allows for a direct exploitation of monolingual data, thus opening new possibilities for the effective generation of synthetic qe data across languages and domains. in recent years, approaches based on artificial neural networks have been successfully applied to the task as well, either as additional features @xcite or as endto-end quality estimation systems @xcite @xcite . the predictor-estimator framework proposed by @xcite can be considered the current standard, since it outperformed alternatives in recent wmt qe tasks @xcite and now serves as baseline in the latest editions of the task @xcite @xcite . in this framework, a contextual word predictor component acts as a feature extractor and an estimator exploits the extracted features to predict qe labels. a neural word prediction model can be trained on the parallel data @xcite , though in recent years, pretrained large language models, such as bert @xcite and xlm-r @xcite , have also been successfully employed for this task @xcite @xcite . as previously noted, supervised approaches depend on the availability of annotated datasets, typically hter scores obtained from post-edited machine translation output, quality values on a predefined quality scale, or ok/bad annotations at the word-level. creating quality annotated datasets is a costly process, hindering the development of quality supervised qe models. to date, most publicly available qe datasets are those prepared for the wmt shared tasks, which are only available for a limited number of language pairs and domains. to address this data bottleneck, alternatives to supervised modelling have been explored for the qe task. thus, @xcite tackled weakly supervised and single-feature unsupervised methods, as a means to minimise the dependency on annotated data. @xcite describes an unsupervised method based on combining ibm1 models with language models over morphemes and part-of-speech tags, with a dependence on external tagging tools. in @xcite , unsupervised quality estimation is performed via lexical translation overlaps and n-gram language model scores, outperforming some feature-based supervised models but falling short against more sophisticated neural qe models. an unsupervised glassbox approach, based on the confidence of nmt models, was proposed by @xcite , achieving promising results, though it requires access to the nmt models that generate the evaluated translations. recently, @xcite proposed a self-supervised approach based on target token masking in parallel data, outperforming other methods based on unsupervised modelling or synthetic data generation. another approach to the lack of human-annotated training qe data is to leverage existing parallel corpora, similarly to what was suggested for automated post-editing @xcite . thus, lee (2020) and @xcite explored the use of target reference translations as post-edited versions of machine-translated source sentences, showing that it can provide a basis for supervised qe models. in this type of approach, however, target references may differ significantly from mt output and therefore introduce noisy training tuples in the qe data. alternatively, synthetic data can be generated by devising qe error generation pipelines from the parallel data @xcite , although this requires approximating errors that may not correspond to the actual ones produced by mt models. the study most related to ours is that of @xcite , who evaluated their levenshtein transformer approach to word-level quality estimation using synthetic data, part of which was generated by using the output from a weaker mt model and contrasting it with the output of another mt model of higher quality, taken as reference translator. although the idea of contrasting weaker and stronger mt models is similar, this differs from our approach in important respects: their synthetic data results are only established for their proposed qe framework based on the levenshtein transformer, only for word-level qe, and, most importantly, they use the output of unrelated converged translation models, instead of the related learning stages of the same model which we explore in this work. ## methodology as previously indicated, our approach is based on the assumption that nmt models at different training stages might produce errors that resemble those committed by fully trained mt systems, in scenarios where they fail to properly translate such as domain shifts or insufficient training data. the methodology can be summarised as follows: under this approach, synthetic qe data can be generated from monolingual or parallel source data, in any domain or language pair for which an nmt model was trained. several aspects need to be examined to determine an optimal setup for this method, mainly the impact of: (i) using best model translations as opposed to an existing reference in parallel data; (ii) using different volumes of synthetic data; (iii) creating synthetic data from different domains; (iv) combining synthetic data from different model checkpoints; and (v) combing synthetic data and human post-edited translations. in the next sections, we describe the experimental protocols to tackle these aspects and evaluate the potential of our approach. ## experimental setup our experiments centred on two language pairs, english-german (en-de) and romanian-english (ro-en), and the datasets of the wmt 2021 shared qe task @xcite . the selected datasets and models for our experiments are described in turn below. ## comparative results in this section, we compare our results with the selected alternative approaches, namely: baselines trained on the 7k post-edited data of the wmt-qe-train datasets; self-supervised models trained on the available parallel and comparable corpora, as these models require aligned data; the nmt qe model based on contrasting the nmt translation and the parallel or comparable target human reference (<best, @xmath0; and the best variants of our approach as determined in the previous section, all based on checkpoint translations of the source data and translations of the converged nmt models as references. in ## natural and synthetic data combination synthetic data can be used to fully train qe models when no human-labelled data are available, thus alleviating the training data bottleneck for supervised models. when human post-edited data are available however, it remains to be determined if checkpoint-based synthetic data can be used in a complementary manner to further improve the accuracy of qe models. to study this question, we trained qe models on datasets that merged the qe training data of the wmt shared task with synthetic data generated from a separate dataset. for both english and romanian, we thus randomly sampled sentences from the selected source monolingual datasets and generated synthetic data with @xmath3, @xmath4, which provided robust results across the board. 7 since the shared task training datasets consist of 7k data points, we considered three different merged data partitions: (i) merging the 7k wmt qe training data with 7k tuples from the synthetic data; (ii) merging the wmt qe 7k with 70k synthetic tuples, corresponding to our medium datasets in the previous experiments; and (iii) upsampling the qe training data to 70k and merging them with 70k synthetic tuples. there were thus two balanced datasets, and one unbalanced with an order of magnitude more synthetic data points. the results of these experiments are shown in the synthetic data generated via checkpoint translation can thus provide additional accuracy to qe models based on human post-edited data, at both word and sentence levels. we left further experimentation for future research, notably the combination of natural data with mixed synthetic data sampled from different domains. ## conclusions in this work, we described a novel approach to synthetic data generation for translation quality estimation, based on translation models at different learning stages. we exploited nmt model checkpoints, derived from standard training processes, to generate faulty translations that can be contrasted with either human references in parallel datasets, or the translations produced by the converged nmt model. we showed that the latter approach outperformed the use of human references by a significant margin, demonstrating the effectiveness of our method to directly exploit monolingual corpora for synthetic qe data generation. we also showed that checkpoint-based qe performed markedly better than both self-supervised qe and contrasting mt output with human references on parallel data. the synthetic data generated under our approach was shown to match, or be competitive with, human post-edited data, with a relatively minor impact of domain relatedness between the synthetic training data and the test data in our experiments. we also demonstrated that combining human-generated and synthetic data led to significant improvements on the qe tasks, showing the potential of our approach as both a standalone solution when no human-labelled data are available, and as a complementary option when such data are available. the main drawback of the checkpoint-based approach is the need to train a separate nmt model for synthetic data generation. however, since the goal of these models is to generate pairs of translations of differing relative quality, there is no requirement for them to be trained on large volumes of data to achieve high translation quality. as shown by our results in romanian-english, using a relatively small mt training corpus can lead to quality qe synthetic datasets. our approach could be further explored along different lines. in this work, we only selected two arbitrary checkpoint models for our experiments, based on their distance to the converged model in terms of bleu. additional checkpoints could be used to enrich the synthetic datasets, exploiting earlier or later training stages. the relative distances between checkpoints, or alternative selection metrics beyond bleu, could also be used to determine optimal checkpoints for qe data generation. further experimentation will also be relevant to assess optimal data sampling and combination strategies, for specific domains in particular. finally, determining if the errors learned from checkpoints may bias the qe system towards model-specific error types would require a dedicated analysis as well. we leave these research questions for future work.