P re Q u EL : Quality Estimation of Machine Translation Outputs in Advance (2022)

Abstract: We present the task of PreQuEL, Pre-(Quality-Estimation) Learning. A PreQuEL system predicts how well a given sentence will be translated, without recourse to the actual translation, thus eschewing unnecessary resource allocation when translation quality is bound to be low. PreQuEL can be defined relative to a given MT system (e.g., some industry service) or generally relative to the state-of-the-art.From a theoretical perspective, PreQuEL places the focus on the source text, tracing properties, possibly linguistic features, that make a sentence harder to machine translate.We develop a baseline model for the task and analyze its performance. We also develop a data augmentation method (from parallel corpora), that improves results substantially. We show that this augmentation method can improve the performance of the Quality-Estimation task as well.We investigate the properties of the input text that our model is sensitive to, by testing it on challenge sets and different languages. We conclude that it is aware of syntactic and semantic distinctions, and correlates and even over-emphasizes the importance of standard NLP features.

## introduction can we tell if a sentence is difficult to automatically translate, without actually translating it? we argue that this question has important practical and theoretical value. from a practical standpoint, a prequel system can save trouble and cost. for individuals or companies who do not maintain their in-house machine translation (mt) system, translating a big amount of data might turn out to be expensive. there are several automatic translation services offered, but there is no prior indication that any of them would do a good enough job. therefore, after purchasing translations, one would probably want to estimate the quality of the translations, with human annotators or an automatic quality estimation (qe) system @xcite . after spending these resources, there is still a chance that the translation quality will be judged unsatisfying, and there will be no choice but to hire human translators. the decision between the automatic route and the much more expensive alternative of hiring human translators can be informed by a prequel model, before investing the resources to actually translate. see figure 1 . theoretically, prequel allows insight into the performance boundaries of state-of-the-art mt methods. it is common practice to search in retrospect what kinds of text phenomena are difficult for existing mt systems to handle @xcite . such analysis helps in finding flaws in the system, and getting some notion of what is needed to overcome them. challenge sets are also used for that purpose -testing the system, analyzing its performance on carefully chosen cases @xcite @xcite . however, no attention was heeded to the source sentence itself and how it affects the translation quality. are there any specific properties, maybe linguistic features, that make a sentence more difficult for mt? by introducing the prequel task, we place the focus on the input text, and learn which source texts are easy to translate and which are difficult. we consider two variants of this approach. one, where the system is given and another where the prediction is done relative to the state-of-the-art in the field. the latter of course bears a tacit assumption that there are underlying properties that make a sentence easier or harder to translate for a stateof-the-art system. supporting empirical evidence for this assumption is presented in §7.1. we develop a baseline model for the task, based on a state-of-the-art qe system, and report its results in §6. we further develop an automatic data augmentation method from parallel corpora and use it in intertraining @xcite and multi-task settings, outperforming our initial results ( §3.2). in §9, we show that this augmentation method can improve the performance of the qe task as well. our results, surprisingly, do not fall far behind the results of state-of-the-art qe systems, despite not being exposed to the actual translation. to the best of our knowledge, this work is the first to address prequel. @xcite did point out that it is possible to perform qe using only the input sentence, but viewed this finding as an artifact of their dataset. we revisit their claims and argue that the prequel model can potentially simulate the mt system, and there is thus no theoretical reason to consider the dataset as "cheatable". we discuss their claims extensively in §9. analyzing the proposed model, we examine in §7.1 the question of whether the predictions made by the model are specific to the performance of one particular system (whose outputs were used for supervision), or whether they generalize to other recent neural systems. our results indicate that results do generalize to other systems, finding a drop of correlation of only 4.2 points. we further examine the contribution of the syntactic structure of the output in predicting its difficulty. to do so we experiment with a model that was fine-tuned with syntactic parsing, but find that its improvement to the performance is relatively low ( §4.2). however, we do find a correlation be-tween the baseline model's predictions and some syntactic features, suggesting that the model is aware of syntax ( §7.2). we further explore what features of the input sentence our model is sensitive to, and report results in §7.5. ## task definition prequel is the task of predicting the likelihood of an mt system to correctly translate a given sentence to a given target language. the task has two variants. the system-specific variant is given a source language s, a target language t and an mt system between them m . the goal is to learn a function g : @xmath0, such that for every sentence @xmath1. as standard in qe evaluation, the prequel predictions are evaluated against the gold labels using pearson's r correlation coefficient @xcite . the definition for the second variant, namely of general state-of-the-art mt, is equivalent but with no specific m . instead, g satisfies the above for all m such that m is a state-of-the-art mt. unlike the first, the correlation achievable in this variant is inherently bound, due to the non-uniform behavior exhibited by different mt systems. nevertheless, to the extent that systems do share performance patterns, mapping them is of theoretical and applicative value. we focus on the first variant at first and examine the second in §7.1. ## experimental setup we train and test our prequel models each time on one language pair. the first pair we examine is the high-resource en-de. as discussed in @xcite , the translation quality for translations in this pair has little variability, with a mean score of 73.25 and std 8.13. the vast majority of translations were assigned high da scores, which makes differentiating between them challenging. to balance this, we select as our second pair the medium-resource et-en. non-high-resource pairs give qe models an advantage -outputs are occasionally 'hallucinated', i.e., they do not have anything to do with the original sentences. detecting such cases should be simple for qe systems, which explains the high qe scores on those pairs. we would therefore like to examine whether this effect persists in the prequel settings, where the outputs are not available to the model. we use the pipeline described in §3.2 to create scores for en-de and en-zh (when avail-able) wmt-news @xcite , bibleuedin @xcite , tatoeba @xcite and globalvoices @xcite , all taken from opus (tiedemann, 2012). we remove duplicate sentences and randomly split each of the datasets to train/dev/test, 80%/10%/10%. for the mt system we use the opus-mt released model @xcite , based on marian-mt @xcite . the selected datasets are diverse in their domain. wmt-news (28,887 sentences) is a parallel corpus of news provided by wmt for testing mt performance. bible-uedin (48,705 sentences) is a multilingual parallel corpus created from translations of the bible. tatoeba (197,381 sentences) is a crowd-sourced collection of user-provided translations. globalvoices (55,822 sentences) is a news parallel corpus from the website global voices. we examine the out-of-domain effect in app. §c.2. we also ensure that for each language pair at least two datasets were not found in marian training set, making prequel predictions more interesting. we note that despite we translate from english to german/chinese, the translation direction is not always from english. this might lead to artifacts in the metric evaluation @xcite . however, we speculate this is more of a concern for evaluation. most of our experiments rely on the data as an augmentation method, which either helps the main task or not. for the automatic metric we use comet @xcite . where more than one metric is required (for training multitask), we use also chrf++ @xcite and bertscore @xcite . these three metrics cover different kinds: chrf++ is string-based, bertscore is an unsupervised embedding-based model, and comet is a supervised model trained end to end. we train instances of simple for en-de and eten. for en-de and en-zh we train also a simple aug version with comet as intertraining. for the en-de en-zh comparison ( §7.4), we use datasets that are available in both languages: newstests and bible-uedin, and test them on the subset of source sentences that are shared between en-de and en-zh development da. for en-de we also train the more advanced architectures, combined and multi-task. we train two versions of combined, com-bined+ and combined-, the first concatenated to an actual ud parser and the latter to another pre-trained roberta to control for the size. we use the augmentation intertraining for both versions. where we compare two or more versions, we train 3 seeds for each. we take the hyperparameters from the tran-squest implementation, adam optimizer with a learning rate 1e -5, and a linear learning rate warm-up over 10% of the training data. we adjust the batch-size from 8 to 4 to fit our gpus. see app. a. to allow evaluation during training, we sampled 10% of the training data and kept it for evaluation. this is in addition to the development data we use for evaluation at the end of the training. we carry out evaluation every 300 training steps for a small training set (smaller than 1k steps), and every 3k steps otherwise. we perform early stopping with patience of 10 evaluation rounds. the model is trained for a maximum of 3 epochs, and no less than one. if we early-stop during the first epoch @xcite , or reach a low correlation (< 0.1) on the last evaluation round, we reset the seed controlling initialization, batches, split of the training/evaluation data, etc. in order to improve our results, we train 3 random seeds and infer by an average ensemble. we carry out evaluation on the da test set. since this is the first work on prequel, there are no immediate candidates for baselines. an exception is discussed in §9.2 and app. §c.4. we take the negated length of the sentences as a baseline. we assume the longer a sentence is, the harder it is to translate. we experimented also with the perplexity score of gpt-2 (for english as the source language), resulting with low correlation (0.06). we discuss more correlated features in §7.5. in places where we compare ourselves to a qe system, we use transquest. also here we adjust the batch-size to 4 to fit our gpus. ## main results in ## determining whether to mt we expect that the standard test case for prequel would be to determine the chances of a sentence to be translated correctly to its target language. according to the flores guidelines, a translation that closely preserves the semantics of the source sentence gets a da score of 70-90. only a perfect translation gets a score of 91-100. therefore, we report the precision/recall curve for a threshold of 70, to find the 'good enough' sentences, following @xcite . in app. f we also report a threshold of 90 to find the perfect ones. we use the predictions of simples aug to estimate what sentences will get a da score above 70. figure 2 shows the precision/recall curves for en-de and et-en, for a da threshold of 70. for comparison, we plot the curves for a random model that uniformly predicts a random score between 0-100 for each sentence. for en-de, 76% of the sentences have a da score above 70, so we can maintain a relatively high precision simply by assigning random scores. still, our model outperforms it. for et-en, 43% of the sentences are above 70. 9 quality estimation ## conclusion we presented a new task, prequel, the task of predicting the quality of the output of mt systems based on the source sentence only. we developed a baseline model and reported its results, providing motivation for the task by showing that considerable information for predicting the quality scores is stored in the source sentence. we developed an automatic augmentation method, and used it to improve our results. we showed that the predictions made by a model that was trained with a specific system supervision do generalize to other state-ofthe-art systems. we analyzed our model by testing it on challenge sets and other languages, concluding that our model is aware of syntax, meaning, and the target language. motivated by these latter results, we suggested to use the prequel model as an analytic method, to confirm the effect of linguistic and semantic phenomena on the ability of a mt system to translate. future work will use this method to provide insights into the performance boundaries of current mt systems. other lines of work we intend to pursue include examining the advantages of a multilingual prequel model, and developing an advanced prequel model that selects the mt system that is most likely to generate the best translation for a given text.