Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance (2022)

Abstract: Human-translated text displays distinct features from naturally written text in the same language. This phenomena, known as translationese, has been argued to confound the machine translation (MT) evaluation. Yet, we find that existing work on translationese neglects some important factors and the conclusions are mostly correlational but not causal. In this work, we collect CausalMT, a dataset where the MT training data are also labeled with the human translation directions. We inspect two critical factors, the train-test direction match (whether the human translation directions in the training and test sets are aligned), and data-model direction match (whether the model learns in the same direction as the human translation direction in the dataset). We show that these two factors have a large causal effect on the MT performance, in addition to the test-model direction mismatch highlighted by existing work on the impact of translationese. In light of our findings, we provide a set of suggestions for MT training and evaluation. Our code and data are at https://github.com/EdisonNi-hku/CausalMT

## introduction mt has long been concerned with the artifacts introduced by translationese, the human-translated text that is systematically different from naturally written text in the same language @xcite @xcite @xcite . for a translation system translating from language x to language y , there can be two types of test data: sentences that originated in language x and are human-translated into language y (denoted as x h -→y ), and sentences that originated in language y and humantranslated into language x (denoted as x h ← -y ). 2 real translation quality mt performance factor 1: translationese in the test set (test-model direction match) factor 2: train-test direction match factor 3: data-model direction match m o s t s tu d ie s o u r w o rk previous work: correlational studies on some of the factors (mostly factor 1). our work: a systematic causal analysis of the additional causal effects of factor 2 and factor 3. the main concern raised by this distinction of the two sets is whether the reported performance on a mixed test set truly reflects the actual translation quality. previous work in mt has shown that translationese is a confounder in evaluating translation quality @xcite @xcite . recent studies on causality have also brought to attention the importance of distinguishing the data-model direction match, namely whether the data collection direction is the same as or opposite to the model direction, also known as causal or anti-causal learning @xcite @xcite . if the dataset is collected by human annotators who see the input x and produce an output y , then learning an x-to-y model is causal learning, and learning a y -to-x model is anticausal learning. in this work, we study the artifacts in mt brought by translationese from the viewpoint of causality, specifically, the match between the data and model directions. we consider two factors of variation in mt: human translation direction (in both the training and the test set) and model translation direction. thus, we formulate the effect of translationese in the test set as the test-model direction match problem, and causal/anticausal learning as the data-model direction match problem. further, we identify the third important factor, the train-test direction match problem, namely, whether the training set and the test set are collected with the same human translation direction. we study the causal influences of these three factors on mt performance in figure 1 . previous work has mainly studied the artifacts introduced by the test-model direction match on mt performance @xcite @xcite , but little considers the other two factors, the train-test and data-model direction match @xcite . moreover, most analyses are based on correlation but not causation @xcite . in this paper, we use causal inference @xcite to re-investigate previous conclusions about the most studied relationship between the test-model direction match and mt performance. we calibrate the effect of test-model direction match by analyzing the causal effects of the two other factors, the train-test and data-model direction match. in our causal analyses, we use interventions to study the effect of train-test direction match, and covariate matching to study the effect of data-model direction match, by controlling for covariates such as sentence lengths and content. we build causalmt, a new dataset on five language pairs labeled with the human translation directions, and statistically verify that translationese tends to be simpler and more verbose, corroborating previous observations on translationese @xcite @xcite . then, we rigorously analyze causalmt, leading to the following new insights and contributions: c1. previous work claims that translationese in the test set inflates mt model performance and thus suggests removing the translationese-tooriginal half of the test set @xcite @xcite . our work shows that the translationese-to-original half of the test set does not necessarily inflate mt performance in all cases. in some cases, it can even be more challenging than the other half, depending on the human translation direction in the training corpus. hence, we suggest still reporting performance on both test sets, and also reporting the training data direction if available. c2. previous work @xcite claims that back-translation (bt) @xcite is usually more effective than self-training (st) @xcite . our work shows that bt is not necessarily better than st in all cases. this result also depends on how the pseudo-parallel corpus aligns with the human translation direction in the test set. we suggest choosing bt or st depending on this direction match. c3. previous work claims that bt's performance improvement is largely reflected on the translationese-to-original half of the test set, but the improvement is very small on the other half @xcite . our work shows that the improvement of bt can be larger on the other half of the test set as well, as long as the pseudo-parallel corpus aligns with the human translation direction in the test set. c4. our work shows that data-model direction match also has a large causal effect on the mt performance of up to 12.25 bleu points after adjusting for other covariates using the backdoor adjustment @xcite . ## causalmt dataset to investigate the effect of train-test direction match and data-model direction match, we need to collect translation data in different human translation directions. ## the overshadowing effect of train-test direction match the first analysis of this paper aims to calibrate the most studied relationship of the test-model direction match and mt performance by considering the additional effect of the train-test direction match. previous work observes that the translationeseto-original test set inflates the score. to evaluate a model with the x-to-y translation direction, traditionally, the test set is a mixture of two halves, one with the human translation direction x h -→y (aligned) and the other x h ← -y (unaligned, or translationese-to-original) @xcite . previous studies propose that the unaligned, translationese-to-original test set is easier to translate than the other aligned test set because translationese inputs are easy for the mt model to handle @xcite @xcite . the inflated test performance caused by translationese has long been speculated @xcite @xcite , and recent work has statistically verified the correlation @xcite . with the previous understanding, some works suggest removing the unaligned half of the test set @xcite @xcite , which was adopted by the 2019 wmt shared task @xcite , whereas others suggest keeping both but reporting the performance separately @xcite . the motivations from the two sides are that in the unaligned half, although the source text being translationese is an easy input to the model, its target text being naturally written text makes the evaluation more natural. this "inflation" depends on train-test direction match. we take a step back from the argument on whether the unaligned test set positively or negatively affects the mt performance evaluation. instead, we call attention to the fact that, beyond the test-model direction match, there can be other factors also playing a critical in the mt performance evaluation, i.e., the train-test direction match. for a given machine translation task to learn the x-to-y translation, there can be two questions: the question from previous work is whether we should use the test set aligned with the model translation direction (t1) or the test set unaligned with the model translation direction (t2) to evaluate the model fairly, whereas the question answered by our work is which training data should be used to achieve the best performance. our analysis aims to obtain causal conclusions on how intervening on the train-test direction match affects the mt performance. therefore, we control all other possible confounders. for each language pair, we control the total training data size hence, the two constructive suggestions for future work are that (1) it is important to still report on both test sets, and also the training data direction if available, and (2) parallel training data in the same direction with the test set is more helpful. monolingual data in the original language of the test set is more helpful. with the intuition that the train-test direction match is a crucial factor for mt performance, we also look into its implications on semi-supervised learning. given additional monolingual data, a common question in mt is what type of monolingual data to use, and the accompanying question, whether to use self-training (st) for the source language monolingual corpus @xcite or back-translation (bt) for the target language monolingual corpus @xcite @xcite . english-to-french (en-to-fr) translation test 1 (en, fr * ) test 2 (en * , fr) sup. on equal mix 16.16 16.65 + st (en, fr * * ) +2.04 (aligned) +1.74 + bt (en * * , fr) +1.91 +2.45 (aligned) french-to-english (fr-to-en) translation test 1 (fr, en * ) test 2 (fr * , en) sup. on equal mix 18.39 15.09 + st (fr, en * * ) +2.64 (aligned) +2.24 + bt (fr * * , en) +2.17 +3.26 (aligned) english-to-german (en-to-de) translation test 1 (en, de * ) test 2 (en * , de) sup. on equal mix 10.59 8.80 + st (en, de * * ) +1.92 (aligned) +1.60 + bt (en * * , de) +1.86 +2.25 (aligned) german-to-english (de-to-en) translation test 1 (de, en * ) test 2 (de * , en) sup. on equal mix 11.99 13.46 + st (de, en * * ) +2.28 (aligned) +1.25 + bt (de * * , en) +1.99 +3.72 (aligned) we reframe the question as "with abundant monolingual data from both languages, but limited computation resources, which data (together with the corresponding semi-supervised learning method) should we choose?" in previous work, bt is the most widely used technique @xcite @xcite , and is reported to outperform st @xcite . another line of previous work inspects the performance gain by bt. some argue that bt is helpful mostly on the test set aligned with the model @xcite @xcite , appendix a we re-inspect the two previous lines of work, and find (1) bt does not always outperform st, especially when st can make use of the monolingual data in the original language of the test set (to produce pseudo-aligned training data), and (2) the performance gain by bt is not always larger on the unaligned test set, but depends on the model direction, especially when bt generates pseudo-aligned training data with the test set. we implement bt by @xcite , and st by @xcite . to fairly compare the per-formance of st vs. bt, for each language pair x and y , we split half both training corpora into x to train st or bt, we use the second halves of the training data only as the monolingual corpora. for example, if the translation task is english-to-german translation, st generates a pseudo-parallel corpus with original english paired with machinetranslated pseudo-german, which we denote as (en, de * * ). for readability, we mark the machinetranslation direction with st and bt by * * and the human translation direction by * . our hypothesis is that the machine-translated text pairs (en, de * * ) will also show similar properties as the human-translated training data (en, de * ). specifically, the more the pseudo-training data is aligned with the test set, the higher performance the semi-supervised learning method will achieve. this is confirmed by the experiment results in ## causal effect of data-model direction match the second contribution of this work is to inspect how much another factor, the data-model direction match, causally affects the mt performance. formally, our research question is that, for a given translation task x-to-y , considering an equal mix of the test set, does the human translation direction of the training data still matter? if so, how large is the effect, and is it language-/task-dependent? this section will use causal inference to isolate the effect of data-model direction match from other possible confounders and discuss its effect in different languages and translation tasks. ## related work linguistic studies have long observed the distinct properties of translationese from text originally authored in the same language @xcite @xcite . recent work in mt identifies that the source side of the translationese-to-original portion of the test sets (i.e., test sets unaligned with the model direction) is easier @xcite , echoing with many previous observations @xcite @xcite and thus some suggest to exclude this portion from future test sets, in particular human evaluations @xcite @xcite . nevertheless, @xcite demonstrate that it is worth reporting automatic metric scores on both directions separately as both types of test sets evaluate different properties of translation quality. in follow-up work, @xcite introduce paraphrased test sets that overcome the problems of translationese for test sets, not solving the problem for the training data though. based on this speculated inflation of mt performance due to translationese in the test set, further work inspects what previous conclusions about the effectiveness of mt models should be recalibrated. some discover that models with bt mostly improve on the inflated test set but not the other more challenging portion @xcite @xcite , appendix a while for almost all test sets the original language of each example is known, the majority of training data does not contain this meta information. studies of the impact of translationese on training examples are thus mostly based on europarl where meta information is given. for instance, @xcite argue that the original side of each training sample is important when building a statistical machine translation (smt) system. @xcite ; @xcite ; @xcite build classifiers to automatically detect the direction of each training sample. @xcite use a cnn classifier to separate the training data at scale and bias the nmt model via tagging to generate translations that look like the original text. human evaluation demonstrates that this produces more accurate and natural translations. further, amponsah-kaakyire et al. (2021) investigate the impact of training samples generated with pivot ("relay") languages. our work differs from all previous work in that we conduct causal inference @xcite to contribute causal insights on how translationese affects mt. ## future work we list several directions for potential future work: ## conclusion in conclusion, this work study the causal effects of three important factors on mt performance: the test-model, train-test, and data-model direction match. we provide suggestions for future study in mt, such as using more training data in the aligned direction and paying attention to whether the nature of the translation task is causal or anticausal.