Synergy with Translation Artifacts for Training and Inference in Multilingual Tasks (2022)

Abstract: Translation has played a crucial role in improving the performance on multilingual tasks: (1) to generate the target language data from the source language data for training and (2) to generate the source language data from the target language data for inference. However, prior works have not considered the use of both translations simultaneously. This paper shows that combining them can synergize the results on various multilingual sentence classification tasks. We empirically find that translation artifacts stylized by translators are the main factor of the performance gain. Based on this analysis, we adopt two training methods, SupCon and MixUp, considering translation artifacts. Furthermore, we propose a cross-lingual fine-tuning algorithm called MUSC, which uses SupCon and MixUp jointly and improves the performance. Our code is available at https://github.com/jongwooko/MUSC.

## introduction large-scale pre-trained multilingual language models @xcite @xcite @xcite have shown promising transferability in zeroshot cross-lingual transfer (zsxlt), where pretrained language models (plms) are fine-tuned using a labeled task-specific dataset from a richresource source language (e.g., english or spanish) and then evaluated on zero-resource target languages. multilingual plms yield a universal representation space across different languages, thereby improving multilingual task performance @xcite . recent work has enhanced cross-lingual transferability by reducing the discrepancies between languages based on translation approaches during fine-tuning @xcite @xcite . our paper focuses on when translated datasets are available for cross-lingual transfer (xlt). in this paper, we combine translate-train and translate-test using a pre-trained multilingual bert, to improve the performance. next, we identify that fine-tuning using the translated target dataset is required to improve the performance on the translated source dataset due to translation artifacts even if the languages for training and inference are different. finally, to consider translation artifacts during fine-tuning, we adopt two training methods, supervised contrastive learning (supcon; @xcite and mixup @xcite and propose musc, which combines them and improves the performance for multilingual sentence classification tasks. @xcite . for marc and mldoc, the original target dataset for training t trn is additionally given. we use the given translated datasets t mt trn for pawsx and xnli. however, for marc and ml-doc, the translated datasets are not given. therefore, we use an m2m_100_418m translator @xcite from the open-source library easynmt foot_1 to create the translated datasets. t mt trn is translated from s trn (i.e., s @xmath0, and t bt trn is backtranslated from t trn (i.e., t @xmath1; @xcite . similarly, for inference, s mt tst is translated from t tst . the notations used in this paper are listed in we use the pre-trained cased multilingual bert @xcite from huggingface transformers @xcite and use accuracy as a metric. detailed information for fine-tuning is provided in appendix b. in this section, we demonstrate that the two baselines, translate-train and translate-test, are easily combined to improve performance, which we call it translate-all. from to explain the changes in inferences via test time augmentation, we describe the predicted probability values on the correct label when the models are evaluated on t tst and s mt tst , as depicted in figure 1 . the green and orange dots represent the benefits and losses via the ensemble, respectively. the improved performance through the ensemble means that the number of green samples is greater than the number of orange samples in figure 1 . to analyze where the performance gain comes from, we focus on the green samples. the green samples are concentrated around the right down corner, which implies that wrong predictions on t tst can be right predictions with high confidence on s mt tst . in fact, this phenomenon is the opposite of what we expected; the samples are expected to be concentrated around the @xmath2, because the semantic meaning between t tst and s mt tst is similar even though the languages are different. this implies that semantic meaning is not the main factor explaining the performance gain of the ensemble. ## translation artifacts for training to find the main factor of performance gain, we hypothesize that matching the types of text (i.e., original or translated) between training and inference is important even if the languages used for training and inference are different, by expanding on @xcite . for the analysis, we use marc and mldoc because they provide t trn , which has no artifacts. ## conclusion in this paper, we showed that translate-train and translate-test are easily synergized from the test time augmentation perspective and found that the improved performance is based on translation artifacts. based on our analysis, we propose musc, which is supervised contrastive learning with mixture sentences, to enhance the generalizability on translation artifacts. our work highlighted the role of translation artifacts for xlt.