Translate-Train Embracing Translationese Artifacts (2022)

Abstract: Translate-train is a general training approach to multilingual tasks. The key idea is to use the translator of the target language to generate training data to mitigate the gap between the source and target languages. However, its performance is often hampered by the artifacts in the translated texts (translationese). We discover that such artifacts have common patterns in different languages and can be modeled by deep learning, and subsequently propose an approach to conduct translate-train using Translationese Embracing the effect of Artifacts (TEA). TEA learns to mitigate such effect on the training data of a source language (whose original and translationese are both available), and applies the learned module to facilitate the inference on the target language. Extensive experiments on the multilingual QA dataset TyDiQA demonstrate that TEA outperforms strong baselines.

## introduction cross-lingual transfer has drawn wide attention in recent years @xcite . it has great potentials to be applied in advanced industries and real applications such as for improving dialog and advertisement systems in multilingual countries @xcite . it aims to reuse nlp models trained on a source language for the task of a target language. the most intuitive method is transfer learning by leveraging pre-trained multilingual language models (lms) such as mbert @xcite and xlm-r @xcite . these pre-trained lms encode different languages into a joint space of multilingual representations @xcite , and they perform well especially for zero-shot cross-lingual tasks @xcite ). another method orthogonal to this is called translatetrain @xcite . it translates training data from the source language into to tackle the issue with translationese artifacts, inspired by domain mapping techniques @xcite , we explore the idea of projecting originals figure 2 : the xlm-r @xcite classification results of translationese for different languages on tydiqa @xcite . we use the classifiers trained with only english pairs (originals and translationese), or randomly initialized (without any training). more details of the experiments are given in appendix b. and translationese into a common embedding space to close their gap. since only the originals of the source language are available under the translatetrain setting, whether this idea is feasible depends on whether the projection function is learnable from the source language and transferable to other languages. therefore, we first conduct experiments to investigate if translationese artifacts, or patterns of differences between orginals and translationese, are recognizable and transferable across languages by deep learning models. specifically, we train a binary classifier to distinguish english originals from english translationese. we then test the effectiveness of this binary classifier on other languages. our intuition is that (1) if the model converges, it suggests that the patterns of translationese artifacts can be potentially learned to some extent, and 2) if the trained model recognizes the translationese of other languages, it means the model can likely transfer the learned patterns across different languages. our results in figure 2 validate both: 1) the model converges well and achieves 97% accuracy on english, the training language. (2) it also performs reasonably well on other languages (77% ∼ 91%). based on the above intuition and validation, we propose a translationese embracing artifacts (tea) method that projects originals and translationese into a common space to mitigate the translationese artifacts. tea explicitly learns a mapping function from originals to translationese using originals and translationese of the source lan-guage (english in our experiments), where learning is through minimizing the distance between the mapped representation of originals and of the corresponding translationese. tea then applies this mapping function to the originals of the target language during the testing stage. for evaluation, we conduct experiments on multilingual qa using the tydiqa dataset @xcite foot_1 . our results show that tea outperforms translate-train baselines and sota translationese mitigation methods designed for machine translation @xcite . ## related work the effect of translationese has been widely studied in translation tasks @xcite @xcite @xcite . some works focus on mitigating or controlling the effect of translationese, e.g., tagged training @xcite @xcite , which are adopted as baselines in our paper. in the field of cross-lingual transfer, there are very few works about translationese. @xcite is the only attempt for translate-test and zero-shot learning. in contrast, we focus on translate-train and aim to mitigate the artifacts in translationese. our research is also related to domain adaptation (da) that aims to transfer the knowledge from a source domain to target domains. our original-totranslationese projection function can be seen as something similar to projecting source domain and target domain data into a common space, which has been used before for domain adaptation @xcite . ## our approach (tea) let x represent the input text and y represent the output label. x denotes the domain (i.e., all possible values) of x and y is the set of labels. the input x comes from different languages, and it can be either originals or translationese during training. specifically, we use x src, orig to denote the domain of source language originals, and define x trgt, orig and x trgt, trans in a similar way (where trgt refers to the target language and trans refers to translationese). we further use back-translation @xcite to generate source language trans-lationese (i.e., the source language originals are first translated into a pivot language and then translated back into the source language), denoted as x src, trans , for the purpose of learning a mapping function to project originals and translationese into the same space. we now present our tea method. our ultimate goal is to learn a mapping function f ∶ x trgt, @xmath0, which takes target language originals as input. however, we only have d src, @xmath1, orig ×y and d trgt, @xmath2, trans ×y during training. the challenge is that an f learned from either d src, orig or d trgt, trans may not work effectively on x trgt, orig because of the differences between the source and the target languages and between originals and translationese. to mitigate the differences between the source and the target languages, we rely on pretrained multilingual language models, as many existing works do. as for the differences between originals and translationese, based on the idea discussed in section 1, we propose to mitigate the translationese artifacts of the target language using an original-to-translationese mapping function, and because of the lack of target originals, we propose to learn the original-to-translationese mapping function from the source language. to concretely illustrate our idea, we break down the mapping from x to y into the following steps foot_2 : multilingual projection (mp): first, input x is projected into a language-agnostic multilingual space by using a pre-trained multilingual lm. we use x ml to denote the projected multilingual space, and f mp is a multilingual projection (i.e., the multilingual lm) that maps an input x in any language into x ml . original-to-translationese projection (otp): suppose x ml consists of two subspaces: x @xmath3, orig ⋃ x ml, trans , where x ml, orig and x ml, trans denote the multilingual representations of any originals and translationese, respectively. to close the gap between originals and translationese, we define an original-to-translationese projection function f otp ∶ x ml, @xmath4, trans to convert the vector representation of a piece of originals to its corresponding representation of translationese. language-agnostic qa (qa): the last step is a language-agnostic classifier for the qa task itself. we use f qa ∶ x ml, @xmath5. given an input x, depending on whether it is from originals or translationese, we use different here • represents the composition of two functions, i.e., f • g(x) = f (g(x)), and * denotes source language or target languages. more concretely, for (x, y) ∈ d src, orig , we use x src, orig as discussed in section 1, we make use of the source language originals and translationese to learn f otp . specifically, for (x, y) ∈ d src, orig , we use x ′ ∈ x src, trans to represent its corresponding translationese, i.e., generated by backtranslation @xcite through a pivot language. let {(x, x ′ )} ∈ d src, pairs denotes all the pairs of originals and translationese in the source language. then, we minimize the distance between f otp (f mp (x)) and f mp (x ′ ) to optimize f otp . in summary, the loss function consists of the following three components: ## experiments dataset. we conduct experiments on ty-diqa @xcite . specifically, we evaluate our approach on the gold-passage subtask of ty-diqa, which includes 9 languages. we set english as source language and others as target languages, and report the results on target languages. during training, we utilize translated training data in all target languages for joint training. we use exact match (em) and f1 scores as evaluation metrics. implementation. translations of english training data for target languages are from xtreme (hu baselines. we compare our model with the following baselines: (1) standard translate-train (stt) @xcite . (2) fil-ter @xcite is an advanced translatetrain method fully utilizing the parallel data. ( 3 ) tagging (tag) @xcite , which distinguishes originals and translationese by adding a tag for machine translation. (4) two-stage training (tst) @xcite , which is another approach to address the gap between translationese and originals for machine translation. it first uses the combination of them for training followed by another round of training only on originals. (5) gradient reversal layer (grl) @xcite , which is a general da method. main results. ablation studies. we conduct in-depth ablation studies to analyze tea. specifically, we explore the following settings: (1) since we use 11% more data in tea (unlabeled x src, trans ) compared to stt, here we add labeled x src, trans in stt. (2) since we use additional 0.38% parameters (otp) in our method compared to stt, here we add the same otp module in stt. (3) we replace the original-to-translationese projection (otp) by translationese-to-original projection (top). (4) we replace the self-attention layer in otp with a multi-layer perceptron (mlp). (5) we replace the cosine distance function in loss with mean square function. the results are summarized in ## conclusions we aim to mitigate the translationese artifacts when training translate-train models. after varifying the transferability of the translationese patterns across languages, we propose the tea that mitigates artifacts using a source language and to facilitate the prediction on unseen target languages. our approach is simple and generic. moreover, our results on multilingual qa show its effectiveness.