Translation Artifacts in Cross-lingual Transfer Learning (2020)

Abstract: Both human and machine translation play a central role in cross-lingual transfer learning: many multilingual datasets have been created through professional translation services, and using machine translation to translate either the test set or the training set is a widely used transfer technique. In this paper, we show that such translation process can introduce subtle artifacts that have a notable impact in existing cross-lingual models. For instance, in natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them, which current models are highly sensitive to. We show that some previous findings in cross-lingual transfer learning need to be reconsidered in the light of this phenomenon. Based on the gained insights, we also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively.

## introduction while most nlp resources are english-specific, there have been several recent efforts to build multilingual benchmarks. one possibility is to collect and annotate data in multiple languages separately @xcite , but most existing datasets have been created through translation @xcite . this approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models. the resulting multilingual datasets are generally used for evaluation only, relying on existing english datasets for training. closely related to that, cross-lingual transfer learning aims to leverage large datasets available in one language-typically english-to build multilingual models that can generalize to other languages. previous work has explored 3 main approaches to that end: machine translating the test set into english and using a monolingual english model (translate-test), machine translating the training set into each target language and training the models on their respective languages (translate-train), or using english data to finetune a multilingual model that is then transferred to the rest of languages (zero-shot). the dataset creation and transfer procedures described above result in a mixture of original, foot_0 human translated and machine translated data when dealing with cross-lingual models. in fact, the type of text a system is trained on does not typically match the type of text it is exposed to at test time: translate-test systems are trained on original data and evaluated on machine translated test sets, zero-shot systems are trained on original data and evaluated on human translated test sets, and translate-train systems are trained on machine translated data and evaluated on human translated test sets. despite overlooked to date, we show that such mismatch has a notable impact in the performance of existing cross-lingual models. by using back-translation @xcite to paraphrase each training instance, we obtain another english version of the training set that better resembles the test set, obtaining substantial improvements for the translate-test and zero-shot approaches in cross-lingual natural language inference (nli). while improvements brought by machine translation have previously been attributed to data augmentation @xcite , we reject this hypothesis and show that the phenomenon is only present in translated test sets, but not in original ones. instead, our analysis reveals that this behavior is caused by subtle artifacts arising from the translation process itself. in particular, we show that translating different parts of each instance separately (e.g., the premise and the hypothesis in nli) can alter superficial patterns in the data (e.g., the degree of lexical overlap between them), which severely affects the generalization ability of current models. based on the gained insights, we improve the state-of-the-art in xnli, and show that some previous findings need to be reconsidered in the light of this phenomenon. ## related work cross-lingual transfer learning. current crosslingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages. some authors leverage parallel data to that end @xcite , but training a model akin to bert @xcite on the combination of monolingual corpora in multiple languages is also effective @xcite . closely related to our work, @xcite showed that replacing segments of the training data with their translation during fine-tuning is helpful. however, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide. ## experimental design our goal is to analyze the effect of both human and machine translation in cross-lingual models. for that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate english paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data. we next describe the models used in our experiments ( §3.1), the specific training variants explored ( §3.2), and the evaluation procedure followed ( §3.3). ## nli experiments we next discuss our main results in the xnli development set ( §4.1, §4.2), run additional experiments to better understand the behavior of our different variants ( §4.3, §4.4, §4.5), and compare our results to previous work in the xnli test set ( §4.6). ## qa experiments so as to understand whether our previous findings apply to other tasks besides nli, we run additional experiments on qa. as shown in model en fr es de el bg ru tr ar vi th zh hi sw ur avg fine-tune an english model and machine translate the test set into english (translate-test) ## discussion our analysis prompts to reconsider previous findings in cross-lingual transfer learning as follows: the cross-lingual transfer gap on xnli was overestimated. given the parallel nature of xnli, accuracy differences across languages are commonly interpreted as the loss of performance when generalizing from english to the rest of languages. however, our work shows that there is another factor that can have a much larger impact: the loss of performance when generalizing from original to translated data. our results suggest that the real cross-lingual generalization ability of xlm-r is considerably better than what the accuracy numbers in xnli reflect. ## conclusions in this paper, we have shown that both human and machine translation can alter superficial patterns in data, which requires reconsidering previous findings in cross-lingual transfer learning. based on the gained insights, we have improved the state-of-theart in xnli for the translate-test and zero-shot approaches by a substantial margin. finally, we have shown that the phenomenon is not specific to nli but also affects qa, although it is less pronounced there thanks to the translation procedure used in the corresponding benchmarks. so as to facilitate similar studies in the future, we release our nli dataset, 13 which, unlike previous benchmarks, was annotated in a non-english language and human translated into english.