Translationese as a Language in “Multilingual” NMT (2020)

Abstract: Machine translation has an undesirable propensity to produce “translationese” artifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese and original (i.e. natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text? There is no data with original source and original target, so we train a sentence-level classifier to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model. Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both accuracy and fluency. Additionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality. We analyze these outputs using metrics measuring the degree of translationese, and present an analysis of the volatility of heuristic-based train-data tagging.

## introduction these differences between translated and original text mean that the direction in which parallel data (bitext) was translated is potentially important for machine translation (mt) systems. most *work done while at google research. translated original translated original target source src-orig data trg-orig data mt training bitext parallel data is either source-original (the source was translated into the target) or target-original (the target was translated into the source), though sometimes neither side is original because both were translated from a third language. figure 1 illustrates the four possible combinations of translated and original source and target data. recent work has examined the impact of translationese in mt evaluation, using the wmt evaluation campaign as the most prominent example. from 2014 through 2018, wmt test sets were constructed such that 50% of the sentence pairs are source-original (upper right quadrant of figure 1 ) and the rest are target-original (lower left quadrant). @xcite , @xcite , and @xcite have examined the effect of this testing setup on mt evaluation, and have all argued that target-original test data should not be included in future evaluation campaigns because the translationese source is too easy to translate. while target-original test data does have the downside of a translationese source side, recent work has also shown that human raters prefer mt output that is closer in distribution to original target text than translationese @xcite . this indicates that the target side of test data should also be original (upper left quadrant of figure 1 ); however, it is unclear how to produce high-quality test data (let alone training data) that is simultaneously sourceand target-original. because of this lack of original-to-original sentence pairs, we frame this as a zero-shot translation task, where translationese and original text are distinct languages or domains. we adapt techniques from zero-shot translation with multilingual models @xcite , where the training pairs are tagged with a reserved token corresponding to the domain of the target side: translationese or original text. tagging is helpful when the training set mixes data of different types by allowing the model to 1) see each pair's type in training to preserve distinct behaviors and avoid regressing to a mean/dominant prediction across data types, and 2) elicit different behavior in inference, i.e. providing a tag at test time yields predictions resembling a specific data type. we then investigate what happens when the input is an original sentence in the source language and the model's output is also biased to be original, a scenario never observed in training. tagging in this fashion is not trivial, as most mt training sets do not annotate which pairs are sourceoriginal and which are target-original foot_0 , so in order to distinguish them we train binary classifiers to distinguish original and translated target text. finally, we perform several analyses of tagging these "languages" and demonstrate that tagged back-translation @xcite can be framed as a simplified version of our method, and thereby improved by targeted decoding. our contributions are as follows: to train the classifier, we need target-language text annotated by whether it is original or translated. we use news crawl data from wmt foot_2 as targetoriginal data. it consists of news articles crawled from the internet, so we assume that most of them are not translations. getting translated data is trickier; most human-translated pairs where the original language is annotated are only present in test sets, which are generally small. to sidestep this, we choose to use machine translation as a proxy for human translationese, based on the assumption that they are similar. this allows us to create classifier training data using only unannotated monolingual data. we propose two ways of doing this: using forward translation (ft) or round-trip translation (rtt). both are illustrated in figure 2 . to generate ft data, we take source-language news crawl data and translate it into the target language using a machine translation model trained on wmt training bitext. we can then train a classifier to distinguish the generated text from monolingual target-language text. one potential problem with the ft data set is that the original and translated pairs may differ not only in the respects we care about (i.e. translationese), but also in content. taking @xmath0, one could imagine that certain topics are more commonly reported on in original english language news than in french, and vice versa, e.g. news about american or french politics, respectively. the words and phrases representing those topics could then act as signals to the classifier to distinguish the original language. to address this, we also experiment with rtt data. for this approach we take target-language monolingual data and round-trip translate it with two machine translation models (@xmath1, resulting in another target-language sentence that should contain the same content as the original sentence, alleviating the concern with ft data. here we hope that the noise introduced by round-trip translation will be similar enough to human translationese to be useful for our downstream task. in both settings, we use the trained binary classifier to detect and tag training bitext pairs where the classifier predicted that the target side is original. ## example output in ## conclusion we have demonstrated that translationese and original text can be treated as separate target languages in a "multilingual" model, distinguished by a classifier trained using only monolingual and synthetic data. the resulting model has improved performance in the ideal, zero-shot scenario of @xmath3, as measured by human evaluation of adequacy and fluency. however, this is associated with a drop in bleu score, indicating that better automatic evaluation is needed.