Revisiting Machine Translation for Cross-lingual Classification (2023)

Abstract: Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate-test), or translating the training set into the target languages and finetuning a multilingual model (translate-train). However, most research in the area focuses on the multilingual models rather than the MT component. We show that, by using a stronger MT system and mitigating the mismatch between training on original text and running inference on machine translated text, translate-test can do substantially better than previously assumed. The optimal approach, however, is highly task dependent, as we identify various sources of cross-lingual transfer gap that affect different tasks and approaches differently. Our work calls into question the dominance of multilingual models for cross-lingual classification, and prompts to pay more attention to MT-based baselines.

## introduction recent work in cross-lingual learning has pivoted around multilingual models, which are typically pretrained on unlabeled corpora in multiple languages using some form of language modeling objective @xcite . when finetuned on downstream data in a single languagetypically english-these models are able to generalize to the rest of the languages thanks to the aligned representations learned at pretraining time, an approach known as zero-shot transfer. the so called translate-train is an extension of this method that augments the downstream training data by translating it to all target languages through mt. a third approach, translate-test, uses mt to translate the test data into english, and runs inference using an english-only model. the conventional wisdom is that translate-train tends to bring modest improvements over zero-shot, while translate-test has been largely overlooked and is often not even considered as a baseline @xcite . in this context, most recent research in multilingual nlp has focused on pretraining stronger multilingual models @xcite @xcite , and/or designing better fine-tuning techniques to elicit cross-lingual transfer @xcite @xcite . however, little attention has been paid to the mt component despite its central role in both translate-train and translate-test. most authors use the official translations that some multilingual benchmarks come with, and it is unclear the extent to which better results could be obtained by using stronger mt systems or developing better integration techniques. ## experimental setup pre-trained models. we use roberta large @xcite and xlm-r large @xcite as our primary english-only and multilingual models, respectively. xlm-r can be considered the multilingual version of roberta, foot_0 as they are both encoder-only masked language models trained with the same codebase. so as to understand the benefits from using a stronger english-only model, we also experiment with debertav3 large @xcite . all the 3 models have 304m backbone parameters, although they differ in the size of their vocabulary. ## main results we next present our main results revisiting mt for cross-lingual classification, both for translate-test ( §3.1) and translate-train ( §3.2). finally, we put everything together and reconsider the state-of-theart in the light of our findings ( §3.3). ## analyzing the variance across tasks as we have just seen, the optimal cross-lingual learning approach is highly task dependent. in this section, we try to characterize the specific factors that explain this different behavior. to that end, we build on the concept of cross-lingual transfer gap, which is defined as the difference in performance between the source language that we have training data in (typically english) and the target language that we are evaluating on @xcite . while prior work has used this as an absolute metric to compare the cross-lingual generalization capabilities of different multilingual models, we argue that such a transfer gap can be attributed to different sources depending on the approach used, which we try to quantify empirically. in what follows, we identify the specific sources of transfer gap that each approach is sensitive to ( §4.1), propose a methodology to estimate their impact using a monolingual dataset ( §4.2), and present the estimates that we obtain for various tasks and target languages ( §4.3). ## related work while using mt for cross-lingual classification is a long standing idea @xcite @xcite , there is relatively little work focusing on it in the era of language model pretraining. ponti et al. (2021) improve translate-test by treating the translations as a latent variable, which allows them to finetune the mt model for the end task through minimum risk training and combine multiple translations at inference. our approach is simpler, but obtains substantially better results on xnli and paws-x. artetxe et al. (2020) explore a simpler variant of our training data adaptation ( §3.1.2), but their focus is on translation artifacts and our numbers are considerably stronger. oh et al. (2022) show that translate-train and translate-test are complementary and better results can be obtained by combining them. isbister et al. (2021) report that translate-test outperforms monolingual models fine-tuned on the target language, but their work is limited to sentiment analysis in scandinavian languages. while we focus on classification tasks, there is also a considerable body of work exploring mt for cross-lingual sequence labeling, which has the additional challenge of projecting the labels @xcite @xcite 6 conclusions contrary to the conventional wisdom in the area, we have shown that translate-test can outperform both zero-shot and translate-train in most classification tasks. while most research in crosslingual learning pivots around multilingual models, these results evidence that using an englishonly model through mt is a strong-and often superior-alternative that should not be overlooked. however, there is no one method that is optimal across the board, as not all tasks are equally sensitive to the different sources of cross-lingual transfer gap. using a new approach to quantify such sources of transfer gap, we find evidence that complex tasks like commonsense reasoning are more sensitive to representation quality, making them more suitable for translate-test, whereas shallower tasks like sentiment analysis work better with multilingual models. in the future, we would like to extend our study to other types of nlp problems like generation and sequence labelling, and study how the different approaches work at scale.