Return to the Source: Assessing Machine Translation Suitability (2023)

Abstract: We approach the task of assessing the suitability of a source text for translation by transferring the knowledge from established MT evaluation metrics to a model able to predict MT quality a priori from the source text alone. To open the door to experiments in this regard, we depart from reference English-German parallel corpora to build a corpus of 14,253 source text-quality score tuples. The tuples include four state-of-the-art metrics: cushLEPOR, BERTScore, COMET, and TransQuest. With this new resource at hand, we fine-tune XLM-RoBERTa, both in a single-task and a multi-task setting, to predict these evaluation scores from the source text alone. Results for this methodology are promising, with the single-task model able to approximate well-established MT evaluation and quality estimation metrics - without looking at the actual machine translations - achieving low RMSE values in the [0.1-0.2] range and Pearson correlation scores up to 0.688.

## introduction there are many factors in play when assessing the suitability of a text for machine translation (mt). readability might account for part of the problem, but the metrics designed for its estimation aim at assessing the level of education necessary to understand a given text, from a monolingual perspective @xcite . as evidenced by @xcite , there is a clear-cut distinction between translatability, "the difficulty of a translation task", and readability, "the difficulty of a monolingual text". they argue that, although the two might overlap in some regards, a translation task cannot be solely defined based on monolingual features. their study is centred on human translation (ht), but given that mt and post-editing (pe) represent the strongest future trend for both industry and academia, according to the latest elis language industry report (european language industry @xcite , our work seeks to advance the discussion in the field of mt. in fact, although quality improvements over the last few years have indeed been significant, the translation world has expressed a need, time and time again, for new methods and technologies to properly assess its quality @xcite . most of the previous work in this regard has focused on the target translation; both in the reference-based machine translation evaluation (mte), where the machine-translated segment is compared against a human reference, and in the more recent quality estimation techniques (qe), where the machinetranslated segment is evaluated without any reference @xcite . this paper seeks a different perspective, switching the focus to the source text, to assess whether a given segment will produce a high quality machine translation. we define this task as machine translation suitability. existing mte and qe techniques either use a reference translation or an mt output, meaning they both require to first translate all the segments with mt system in order to obtain a quality evaluation. many such segments will inevitably not meet the desired quality and will be discarded, constituing a net loss. given that most commercial mt systems are paid by word, our approach would serve to reduce the costs of the overall system by avoiding to send certain segments to mt, thus creating a more efficient production pipeline. moreover, recent studies have also pointed towards a lower lexical variation of post-edited mt segments, as well as an overall lower quality of those segments with respect to translations from scratch @xcite , while others highlight the challenges of generating comprehensive guidelines for post-editors, especially regarding what constitutes an error in a given scenario and how to correctly provide quality assurance for such segments (nunziatini and marg, 2020). therefore, the presence of an additional evaluation step before generating the machine-translated segments would help avoid having to undergo an expensive pe step or reroute to human translation. lastly, applying such a model could reduce the pipeline's carbon footprint, because it would not need to compute a translation using large, resource heavy models. with the purpose of advancing research in this field, we thus formulate the following research question: rq: is it possible to accurately predict the mte or qe score of a translation from the source text alone? in order to give light to the rq, we compile an ad-hoc corpus pairing source segments with the evaluation scores of their automatic translations in the english-german language pair from one of the most prominent mt engines available: mod-ernmt foot_0 . we select two reference-based evaluation metrics and two quality estimation metrics: cush-lepor, bertscore, comet, and transquest, according to the state of the art @xcite . we frame the task as a regression problem and fine-tune our model to reproduce the evaluation score by looking at the source text alone. the experiments are conducted using the multilingual model xlm-roberta (xlm-r) @xcite foot_1 and approach the task in two different settings: single-task and multi-task. in the former, a model is fine-tuned on each evaluation score individually, whereas in the latter, a model is trained on all four scores to exploit the shared knowledge among the different metrics. by achieving low rmse values in the [0.1-0.2] range and pearson correlation scores up to 0.688, our results are promising and indicate that it is indeed possible to distil the knowledge acquired from different mt evaluation metrics into a model trained solely on the source text, thus confirming our rq. ## related work nowadays, the state of the art is divided between mte metrics, similar to bleu @xcite @xcite , which employ the source text, target text and a reference translation, and qe metrics which assess quality without looking at a reference @xcite . some of the most prominent reference-based metrics include cushlepor, an n-gram based metric whose parameters are automatically tuned using pre-trained language models @xcite , and bertscore, which exploits embedding similarity and has been shown to highly correlate with human judgments on sentence-level and system-level evaluation @xcite . being somewhat new, the field of qe achieved impressive results in the past few years by employing multilingual pre-trained representations from very large language models to generate their predictions. nevertheless, it instead appears to have no single metric being consistently deployed to production in either the industry or institutions, with the only exception being comet, which has consistently achieved top scores for three years in a row in the annual wmt qe shared task @xcite . both mte and qe metrics, though, depend on the underlying target translation produced by an mt engine and research specifically focused on the source text has been limited. @xcite aimed at developing a "translatability prediction system". it assigns a global difficulty score to a source text and identifies which passages are more problematic for translation. albeit promising, this work solely addressed human translation difficulty and no study tailored to mt has been published yet. smartlqa (smart linguistic quality assessment), aims at analysing the impact of the source text on mt @xcite . it handles the prediction of at-risk content prior to translation, identifying the most problematic linguistic aspects within the source text via linguistic features and readability tests, such as the flesch-kincaid met-ric @xcite . they conclude that poor source-text quality leads to poor target-text quality. to the best of our knowledge, no predictive model using these features has been proposed. additional work in this direction was carried out by @xcite , who use the source segment and mt training data to approximate translation quality without the target. their method is based on the assumption that a similarity can be found between the source segment to be translated and the underlying data seen by the mt system. they employ either a bag-of-word representation or the "all-mpnet-base-v2" sentence transformer model @xcite to encode both the source and the training segments and apply similarity metrics on their vectorial representations, also accounting for words unknown to the mt system. their technique achieves results comparable to qe metrics. similarly, @xcite shows how fuzzy matches retrieved from the training data can be highly informative for predicting sentence-level quality of a given mt model. another recent paper instead proposed a new task, called prequel: pre-quality-estimation learning @xcite , namely predicting the likelihood of an mt system to correctly translate a sentence in a given target language. they, too, entirely focus on the input text and their method also proposes to learn to predict quality evaluation metrics from the source text alone and for this they employ direct assessment (da) scores from the wmt shared task on qe @xcite . additionally, they use the open-source marian-mt @xcite rather than commercial systems. although we recognize that using quality da scores would lead to more reliable target scores, these are not available for commercial systems, as the authors also point out. while we share the same objective, our attempt bypasses the need for manual evaluation to understand whether a large transformer model would be able to predict state-of-the-art mte/qe scores, and instead uses a small pool of automatically scored data. additionally, they employ the monolingual roberta architecture, which limits their experiments to be carried out on english source texts @xcite . hence, we opt for the multilingual xlm-r to create a solid baseline which could be easily extended to multiple language pairs and directions. ## corpus in order to produce our corpus, we departed from a collection of parallel segments from opus (tiedemann, 2012), including europarl foot_2 , ubuntu foot_3 and news-commentary v16 foot_4 . we target the english-german language pair because it is especially prominent for both mte and qe @xcite . although these corpora have been already extensively used in the literature, their pre-processing is done automatically, without any type of manual corrections. to ensure their quality for our experiments, two additional filtering steps have been carried out on the translation units (tus), following @xcite . it involved the removal of both very long and very short segments from the corpora, set to a minimum length of 25 characters and a maximum length relative to each corpus and language. we removed outliers with respect to each subcorpus, since we do not deem them informative for modeling translation difficulty in a real use-case. the maximum allowed tu length is determined as: transquest. we used the @xmath0, as reported in @xcite ). 10 for our mt suitability experiments, the source text segments are paired with their respective quality scores by combining only the source text and the scores. our objective is to produce a model to predict the quality score from the source text alone. with such a model, it would be possible to know how well an mt engine would translate that segment in advance and thus how "suitable" would it be for machine translation. figure 1 represents a possible pipeline, including the rerouting step from source text to either mt, mt+pe or ht, depending on the expected quality -suitability-of the machine translation. we partition the corpus into two: 12,669 instances for training and 1584 instances for testing. since the original corpora used for this work are open-source and specifically designed for nmt training @xcite , it is likely that they have already been seen by modernmt during training. this would be problematic because an attempt at learning mt suitability using these corpora would not necessarily be applicable to unseen texts. hence, we compare the distributions of the training corpus to those of a new, smaller corpus, whose texts have surely not been seen by the system. if the scores' distribution of this secondary corpus were very similar to that of the training corpus, it would mean that there is no significant difference in the scores of unseen and already seen tus. to test this hypothesis, we performed a mann-whitney u test on all 4 independent variables @xcite between our corpus and a collection of texts from globalvoices for which we had guarantees of not having been used for the training of the mt model. appendix a contains all the details of the test. in summary, there was no significant difference (@xmath1.05) between the training and the globalvoices dataset for all metrics except for transquest. this gives confidence that both corpora belong to the same non-gaussian distribution, meaning there is no significant difference in the quality scores obtained by texts translated using our training corpus and a corpus containing texts not seen by the mt system. ## experiments we perform two sets of experiments: once in a single-task setting and once in a multi-task setting. the single-task experiment involves one training session per evaluation metric, thus resulting in four distinct models. in addition to attempting to learn each of the four metrics independently, we also experiment with multi-task learning @xcite to link the various label representations together instead of training separate models. this approach has been applied to multiple areas of nlp, ranging from the estimation of the check-worthiness of claims in political debates @xcite , to a demographic classifier based on features extracted from tweets @xcite and finetuning of transformer models to improve performance on the glue benchmark @xcite . appendix b includes details on the batch size and other model settings for the multitask approach, constrained by design decisions and the hardware at hand. figure 2 offers a representation of the model. we used xlm-roberta-base @xcite ) for our architecture, which has a total of 125 million parameters. 11 while it may be possible to achieve a higher performance with a monolingual englishonly model, we believe that this would not accurately reflect the potential performance on other languages, because high-quality transformer models are not available for all languages. furthermore, our choice is in line with the current trend in the wmt shared task on quality estimation, where xlm-r is one of the most commonly used transformer architectures @xcite . all the experiments used a learning rate of 2e -5 and employed the adamw optimiser. we explored an effective training batch @xmath2, 16, 32] and @xmath3, 5, 10], as suggested for xlm-r by a recent study on the performance of multilingual language models by @xcite . 12 additionally, for our use case, we used huber-loss as the loss function @xcite . 13 this loss combines the advantages of both the mseloss and the l1loss because it employs a squared term if the absolute element-wise error falls below a predefined δ and a δ-scaled l1loss otherwise (we use the default value for δ), making huberloss less sensitive to outliers. we use root mean square error (rmse) for the evaluation (lower values correspond to a better performance). since it is scale-dependent, and the distributions of the labels fall within different ranges, the rmse is not comparable across tasks. this makes it only informative with respect to the original distribution. in order to obtain a value which is not only comparable but also easily interpretable across tasks, all model predictions and gold labels are reshaped into the range [0, 1]. we 11 https://huggingface.co/xlm-roberta-base 12 https://github.com/junjiehu/xtreme-dev/issues/2 13 https://pytorch.org/docs/stable/generated/torch . nn.huberloss.html. also compute both pearson's and spearman's correlation coefficients @xcite between the predicted outputs and the original predictions, similarly to what is done in the ranking of wmt tasks, except that we use mte/qe scores as reference values instead of human evaluations @xcite . ## discussion the obtained results are promising. given that, on average, the reported rmse values of the best models lie in the [0.11, 0.17] range, whereas their correlation scores are in the [0.420, 0.688] range for pearson's r and in the [0.379, 0.652] range for spearman's ρ. this means that all single-task models are able to reproduce the mte/qe fairly accurately starting from the source text alone, which corroborates our rq. overall, the best performing batch size for the single-task model is 32, also thanks to its reduced training time, even though it is certainly more costly in terms of memory requirements. with regards to which approach is better suited to the problem, the answer is indeed more challenging. although the single-task model appears to be decidedly better than the multi-task model in 3 out of 4 target scores, there certainly is room for improvement for the multi-task model, given that it never showed a tendency to converge to the mean, contrary to the single-task model, and especially on bertscore, the knowledge transfer obtained by training on multiple metrics seemed to be beneficial. the results for all other metrics are overall stable, showing no noticeable sign of improvement past the 5-epoch margin (see the low error margins and the good correlation values shown in these experiments point towards the possibility to achieve an accurate estimate of the quality of mt based on the source text alone, without needing to even obtain a machine translated version of the given segment. additionally, given that these automatic metrics are not perfect themselves, future research should focus on testing this model on either human da provided by wmt @xcite , similarly to @xcite , or by assessing post-editing effort based on the scores produced, working towards the definition of thresholds to generate an actual implementation of the workflow sketched in figure 1 . nevertheless, it is also imperative to stress two limitations of this study. the corpus which was used in this study contains segments which have likely been seen by the mt system already during training. although a set of exploratory experiments has shown no significant difference between unseen and seen texts, this remains an aspect that requires further attention, since it would be possible to argue that to properly learn how difficult a text was for a given system, this had to never be seen by the system during training in the first place. we also need to consider the issue of sustainability. in recent years the carbon footprint of large language models has become increasingly impactful and longer training times have been disincentivised by the research community @xcite . the multi-task model used for this study took around 32 hours to train, much longer than the single-task model, which took a fifth of the time, further decreased to 2:30 hours when scaling to higher batch sizes. additionally, since it needs to load four distinct copies of the same xlm-r model, the total number of parameters used increases from 125 million to 500 million in training. this led to the experiments for the multitask model to be only carried out on a batch size of 2 and, given the significant improvements obtained by the single-task model both in training time and performance, a greater batch size could therefore not only improve the performance of the multi-task model but also reduce its carbon footprint. ## conclusions this work attempted to answer one main research question: is it possible to accurately predict the machine translation evaluation or quality estimation score from the source text alone? it was motivated by the increasing need to automatically assess the quality of machine translation in a way that is both dynamic and scalable, without the limitation of providing very expensive reference translations. while there exists a field entirely dedicated to reference-less metrics, namely quality estimation, this paper tried to explore innovative techniques that would focus entirely on the source text. such an approach offers an alternative that could further reduce the costs of machine translation by streamlining the post-editing process without the need to first generate every time the machine-translated version of all the segments, given that many will be inevitably discarded, which constitues a net loss. in fact, it might even be beneficial to avoid having these low-quality segments undergo post-editing, since recent studies have pointed towards lower lexical variation of post-edited machine translation segments, leading to an overall lower quality of the resulting translation @xcite . additionally, post-editing also leads to several challenges in liaising with the post-editors themselves, especially with respect to what constitutes an error in a given scenario and how to provide quality assurance, leading to increased costs (nunziatini and marg, 2020). in order to streamline these processes, reducing costs and improving efficiency, our proposed model can be integrated as part of a workflow which includes a machine translation suitability module to reroute a source text to mt, pe or human translation (ht) depending on the assessed level of suitability (see figure 1 ). the scripts and corpora used for the experiments are available for research purposes. 14 while further studies involving human evaluation are still needed, by obtaining an rmse score as low as 0.11 and good correlations of up to 0.688 with mte/qe metrics, we show a possible link between mt quality prediction and the source text. we also show that, while the multi-task model might be well-suited for this task, its performance is subpar when compared to the single-task model and there remain concerns regarding its computational cost and sustainability issues. nevertheless, the results point toward the possibility of obtaining accurate machine translation evaluations starting from the source text alone, paving the way for further research in the field of mt suitability. future research could improve many aspects touched by this work. exploring correlations with human da scores, research on source text translatability for humans or assessing post-editing effort based on the scores produced are all paramount aspects to investigate in order to correctly define the thresholds for the workflow proposed in figure 1 . moreover, since xlm-r is a multilingual model, an additional focus could be posed on extending the experiments to other language pairs, surveying significant differences among different language combinations and directions to further confirm the current findings. especially interesting would be to expand the analysis on the higher correlation between our metric and the qe metrics when compared to mte metrics, because it may shed further light on what state-of-the-art qe models are actually predicting. additionally, adding a pipeline for terminology recognition in the source text could offer valuable information for the final prediction, given how terminology is still a problematic aspect for many mt systems @xcite . lastly, two main aspects could be improved in order to surpass the current limitations: the training corpus and the training methodology, especially by scaling the current architecture to greater batch sizes. and segmented. this website was selected because one of the subcorpora from opus, the news subcorpus, contains some texts from globalvoices (tiedemann, 2012). 16 the mann-whitney u test assesses whether two independent populations belong to the same distribution. in order to perform the test, four assumptions are needed: (1) the dependent variable should be measured at the ordinal or continuous level (evaluation metrics are continuous), (2) the independent variable should consist of two categorical, independent groups (i.e., the corpus with "seen" texts and the corpus with "unseen" texts), ( 3 ) there is independence of observations (there is no inherent relationship among the various segments), and (4) the two variables are not normally distributed. our data adheres to these assumptions, as observed in