Like Chalk and Cheese? On the Effects of Translationese in MT Training (2021)

Abstract: We revisit the topic of translation direction in the data used for training neural machine translation systems and focusing on a real-world scenario with known translation direction and imbalances in translation direction: the Canadian Hansard. According to automatic metrics and we observe that using parallel data that was produced in the “matching” translation direction (Authentic source and translationese target) improves translation quality. In cases of data imbalance in terms of translation direction and we find that tagging of translation direction can close the performance gap. We perform a human evaluation that differs slightly from the automatic metrics and but nevertheless confirms that for this French-English dataset that is known to contain high-quality translations and authentic or tagged mixed source improves over translationese source for training.

## introduction prior work in statistical machine translation (smt) highlighted potential benefits of making use of information about the translation direction of training data @xcite . when text is translated, there is an authentic source (the language in which the text was originally produced), and its translation, which in contrast can be described as translationese. thus when considering translation direction in machine translation, training data can be described as consisting of authentic source, translationese source, or a mix. 1 backtranslated data produced by machine translation may be thought of as an extreme case of translationese source @xcite , but because the quality and types of errors that occur in machine translation are quite different from those that occur in human translation, it is worth examining translation direction of human translation separately from mt-based data augmentation. in figure 1 we show a fairly dramatic example of the kinds of translation quality differences that can occur when building mt systems using authentic source as opposed to translationese source. recent work in neural machine translation (nmt) has revisited this issue, motivating the automatic detection of (human) translationese by showing improved performance on several metrics when training translation direction matches the testing translation direction @xcite , examining domain and backtranslation along with the translation direction of test sets @xcite , and evaluating the treatment of predicted translation direction as separate languages in a multilingual-style nmt system through human and automatic metrics @xcite ). 1 for the purposes of this paper, we will set aside the situation where both sides of the text consist of translationese, translated from one or more other pivot languages. ## data we use parallel english-french (en-fr) text from the canadian hansard, house of commons. our corpus contains transcripts of debates from 1986 to 2016. earlier parts of this dataset are available from ldc @xcite , more recent transcripts are publicly available from the canadian parliament website. 3 this data is known to have high translation quality. it is annotated with direction of translation (the original language, fr or en, as spoken in the house is known); we omit all lines marked as unspecified. 4 the question of domain is always intertwined with the question of translation direction. here we hope to minimize that by confining our work to the parliamentary domain; we expect that the level of formality and style of parliamentary speech is relatively consistent, even across languages (certainly more so than it would be if compared between news data and parliamentary speech). nevertheless, we acknowledge that there will remain differences within this domain; i.e., members of parliament may speak more frequently about topics related to their own constituencies or about different topics over time. we also sample our data with an eye toward temporal aspects for this reason. the full dataset (from which we select our training, development, and test data) is unbalanced in terms of original language: 10,091,250 lines (68.5%) were originally spoken in english, while 3,699,822 lines (25.1%) were originally spoken in french (the remaining 933,996 lines, 6.3%, were labeled as unspecified). in order to run experiments on the proportion of source-side translationese used, we are limited by the size of the smaller sub-corpus, the authentic-fr language data. we sample data for validation and testing (2k and 8k lines, respectively), with authentic-en source data used for translation into fr and vice versa. the validation and testing data are randomly sampled sentences from recent data (nov. 1 to dec. 15, 2016), while training contains older data. this mimics a real-world scenario, where translators (potentially using computer aided translation) might post-edit or interactively translate new text using the output of machine translation systems build on older text. by drawing the test sentences from a separate portion of the hansard as the training data, we guarantee that test sentence performance is not inflated due to having included neighboring context in the training data; rather, the test data performance should be representative of realistic performance on new and previously unseen hansard data. for q3, we subsample authentic-en parallel text once, to match the authentic-fr training data in size, also attempting to match it in date distribution (which we expect may also serve as a proxy for matching topic distributions). 5 for the experiments that consider between 0% and 100% source side translationese, we then subsample this authentic-en subsample and the authentic-fr data. we preprocess the data using open-source normalization and tokenization scripts from portagetextprocessing. foot_4 specifically, we applied clean-utf8-text.pl (removing control characters, standardization, etc.), followed by fix-slashes.pl (heuristically adding whitespace around slashes), and tokenization with utokenize.pl -noss -@xmath0. we then train joint 32k byte-pair encoding (bpe) subword vocabularies on the training data (sennrich et al., 2016), 7 and apply them to train, development, and test. ## models we build transformer @xcite models using sockeye-1.18.115 @xcite , with 6 layers, 8 attention heads, network size of 512 units, and feedforward size of 2048 units. we have changed the default gradient clipping type to absolute, used source-target soft-max weight tying, an initial learning rate of 0.0002, batches of ∼8192 tokens/words, maximum sentence length of 200 tokens, optimizing for bleu, checkpoint intervals of 4000, and early stopping after 32 checkpoints without improvement. decoding used beam size 5. training used 4 nvidia v100 gpus. ## conclusion we have shown that in a moderate-resource setting with high-quality translations in training data, training on authentic-source data or tagged mixed-source data is preferred over training on translationese-source or mixed (untagged) source data, by both automatic metrics and human judgments. this is in contrast with the findings of @xcite , who found that bleu scores could be "gamed" to produce higher scores with translationese-like output, while being judged to be worse by human annotators. this raises questions for future work, such as whether translationese training effects may vary depending on the quality of the parallel text, the proportion of translationese data, and the size of the training data, or whether differences in experimental setup and human annotation may also come into play. future work could examine these issues across a wider range of language pairs and domains, as well as directly comparing known translation direction with automatically predicted translation direction.