Quality Estimation without Human-labeled Data (2021)

Abstract: Quality estimation aims to measure the quality of translated content without access to a reference translation. This is crucial for machine translation systems in real-world scenarios where high-quality translation is needed. While many approaches exist for quality estimation, they are based on supervised machine learning requiring costly human labelled data. As an alternative, we propose a technique that does not rely on examples from human-annotators and instead uses synthetic training data. We train off-the-shelf architectures for supervised quality estimation on our synthetic data and show that the resulting models achieve comparable performance to models trained on human-annotated data, both for sentence and word-level prediction.

## introduction the adoption of machine translation (mt) has been increasing in areas ranging from government and finance, to even social media due to the substantial improvements achieved from neural machine translation (nmt). however, even with improved performance, translation quality is not consistent across language pairs, domains, and sentences. this can be detrimental to end-user's trust and can cause unintended consequences arising from poor translations. thus, having metrics to assess the quality of translated content is crucial to ensure that only high-quality translations are provided to end-users or downstream tasks. quality estimation (qe) metrics aim to predict translation quality without access to reference translations @xcite @xcite . state-of-the-art qe techniques have leveraged mt systems and language-specific human annotations as supervision, including direct assessment and post-editing @xcite @xcite . however, these annotations are costly and time-consuming, particularly for word-level qe, where each token needs a label. some unsupervised approaches take inspiration from statistical mt @xcite @xcite or apply uncertainty quantification @xcite for qe. however, their performance is inferior to that of supervised models. in related areas such as automatic post-editing, parallel data has been used to create synthetic post-editing data @xcite , however this technique only compares machine-translated sentences to references. our approach augments mt errors with additional errors via masked language model rewriting. we leverage noisy, mined comparable sentences obtained by weakly-supervised techniques @xcite . these noisy bitexts have been mined from a variety of domains such as wikipedia @xcite and large webcrawls @xcite @xcite and have been shown to be an invaluable source of training data for nmt models. using this data is crucial to avoid data leakage between a trained nmt model and the data we use to create synthetic qe data. for each source-target sentence pair from the mined data, we apply an mt system to generate a candidate translation of the source sentence. additionally we rewrite each target reference sentence using a masked language model to introduce errors. these two approaches generate two alternative "translations" of the source sentence. we then produce pseudo-labels for each token in these translations by edit distance alignment to the original reference sentence. this results in each translated word being pseudo-labelled as correct or incorrect, which is our synthetic qe training data. analogously, sentence-level training data is derived as the proportion of incorrect words per sentence. our main contributions are: (i) we explore a simple technique to effectively generate synthetic data for qe that allows for both word-level and sentence-level estimation (ii) we demonstrate that our technique performs comparably to off-the-shelf models trained on human-annotated data. ## qe task description word-level qe has been mainly framed as the task of predicting which words in the translation need to be post-edited. as such, word-level qe aims to assign a tag for each word and gap between words in a machine-generated translation as correct, i.e., the word does not need editing, or incorrect, i.e., the words should be substituted, deleted, or inserted (tags for gaps) @xcite . for word-level, we denote the tag of each word in a translation as m @xmath0, bad}, where @xmath1, t ] and t is the length of the translation. also, we denote the tag of each gap between two words (including the beginning and the end) as g @xmath2, bad}, where @xmath3, t + 1]. in traditional qe, data is collected by first translating source sentences using an mt model. second, experts post-edit these translations. third, the post-edits and machine translations are aligned in such a way that induces the minimum edit distance between the tokens of each. finally, each m t is labelled as bad if it should be deleted or substituted and each g t is labelled as bad if at least a word should be inserted there. sentence-level qe labels can be generated by computing the human-targeted translation error rate (hter) @xcite , which is the minimum ratio of edit operations needed to fix the translation to the number of its tokens. we explore the possibility to skip the costly human post-editing process by proposing a data synthesis pipeline, which we then test on human labelled data. ## approach to data synthesis as depicted in figure 1 , we synthesize data from mined wikipedia datasets, where each example consists of a (source, target) sentence pair. we create candidate translations of source sentences in two ways: for the first approach, we apply the nmt model to translate each source sentence. for the second approach, we rewrite each reference target sentence using a masked language model (mlm), as shown in the mlm rewrites block in figure 1 . the two approaches create two forms of translations. then, by treating target sentences as if they were post-edited data (pseudo post-edits), we identify errors in each candidate translation by looking at the insertions, deletions, and substitutions between the references and generated translations. ## experiments and results we focus on data released by the wmt20 shared task on qe for predicting post-editing effort, which includes english-to-german (en-de) and english-to-chinese (en-zh) word-level data and their sentence-level hter @xcite . 1 as the human-annotated data is sampled from wikipedia, we choose to synthesize data from wiki-matrix @xcite , which consists of mined wikipedia parallel data from which we sample pairs with a laser (artetxe and schwenk, 2019) margin score threshold of 1.06 to ensure high-quality pairs. we note that the original qe data is not a subset of wikimatrix. the german and chinese text were tokenized using the moses foot_1 and jieba foot_2 tokenizers, respectively. we list the statistics of the filtered wikimatrix data as well as our resulting synthetic data in for the off-the-shelf qe model, we choose the for a fair comparison, we take the pre-trained predictor provided by the wmt20 qe shared task, fine-tune the whole model on the human annotated data, and compare results to those when fine-tuned on our synthetic data. we test by comparing model predictions and held-out human-annotated qe at word and sentence-level. at the word level, we measure qe performance with matthew's correlation coefficient (mcc) @xcite (main metric), as well as f1 scores for bad and ok tags. at the sentence-level, we measure the sentencelevel pearson's correlation @xcite , mean absolute error (mae) and root-mean-square deviation (rmse). as shown in in ## discussion in this section, we further analyze how the quantity of synthetic data impacts performance, and what types of errors are represented in each of the mlm and nmt portions of the synthetic data. ## conclusions and future work in this work we devise a technique for building word and sentence-level qe models by creating synthetic training data. by training an off-the-shelf model on our synthetic data, we achieve performance comparable to and often better than training on human-annotated data. this technique for data synthesis can be invaluable if human annotation is difficult to come-by, for example when dealing with low-resource scenarios. this work can be extended in various ways. while we investigate the scenario of utilizing solely synthetic data, further work can study the effects of augmenting human-labeled data with synthetic data. further work can analyze the efficacy of this technique into low-resource language pairs where such human-annotation is difficult to obtain. additionally, instead of a simple mlm re-writer, adversarial training to generate and detect errors could provide more realistic synthetic data.