To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages (2024)

Abstract: Perfect machine translation (MT) would render cross-lingual transfer (XLT) by means of multilingual language models (mLMs) superfluous. Given, on the one hand, the large body of work on improving XLT with mLMs and, on the other hand, recent advances in massively multilingual MT, in this work, we systematically evaluate existing and propose new translation-based XLT approaches for transfer to low-resource languages. We show that all translation-based approaches dramatically outperform zero-shot XLT with mLMs—with the combination of round-trip translation of the source-language training data and the translation of the target-language test instances at inference—being generally the most effective. We next show that one can obtain further empirical gains by adding reliable translations to other high-resource languages to the training data. Moreover, we propose an effective translation-based XLT strategy even for languages not supported by the MT system. Finally, we show that model selection for XLT based on target-language validation data obtained with MT outperforms model selection based on the source-language data. We believe our findings warrant a broader inclusion of more robust translation-based baselines in XLT research.

## introduction multilingual language models (mlms) like mbert @xcite , xlm-r @xcite , or mt5 @xcite have become the backbone of multilingual nlp. their multilingual pretraining and the consequent ability to encode texts from a wide range of languages make them suitable for cross-lingual transfer (xlt) for downstream nlp tasks: fine-tuned on available task-specific data in high-resource languages, they can be used to make predictions for languages that lack task-specific (training) data. their effectiveness as vehicles of both zero-shot (no taskspecific training instances in the target language, zs-xlt) and few-shot xlt (few training instances in the target language, fs-xlt) has been documented for a plethora of tasks and languages @xcite @xcite . cross-lingual transfer with mlms, however, yields poor performance for low-resource target languages that are (i) un(der)represented in the pretraining corpora, especially if they are additionally (ii) linguistically distant from the source language @xcite @xcite . recent years have witnessed a large body of work that focused on improving xlt, in particular for low-resource target languages. first, a multitude of new multilingual benchmarks have been introduced, aiming to either evaluate xlt with mlms on sets of linguistically diverse languages @xcite @xcite or on groups of related low-resource languages from underrepresented language families (i.e., families without any high-resource language) and/or geographies @xcite @xcite @xcite @xcite . second, a diverse set of methodological proposals have been introduced, ranging from (i) attempts to better align mlms' representation subspaces of languages @xcite @xcite those that increase the representational capacity for underrepresented languages, typically via additional post-hoc language-specific language modeling training @xcite @xcite to (iii) various fs-xlt proposals that seek to maximally exploit small sets of task-specific target language instances @xcite @xcite . much of the above work rendered translationbased xlt strategies-in which an mt model is employed to either translate the source-language training data into the target language before training (referred to as translate-train) or translate the target-language instances to the source language before inference (translate-test)-competitive w.r.t. mlm-based transfer @xcite @xcite . sporadically, however, mt has been leveraged for more elaborate translation-based strategies-e.g., translating source-language training data to multiple (related) target languages @xcite , combining the translated target-language training data with the original source-language training data @xcite , or using monolingual english lms instead of mlms for translate-test @xcite -complicating the selection of translation-based baselines in xlt research. in fact, the most recent evidence @xcite suggests that the potential of translation-based xlt has been underestimated due to the selection of suboptimal translation strategies. what is more, much of the work on low-resource xlt completely disregards translation-based baselines, arguing a priori, without empirical confirmation, that (1) due to the lack of parallel data, mt models for low-resource languages exhibit poor performance, which directly caps the potential of translation-based xlt and/or (2) their evaluation encompasses target languages that are unsupported by @xcite mt systems. two recent developments, however, warrant a systematic (re-)evaluation of translation-based xlt for low-resource languages: (i) the availability of open massively multilingual mt models that not only support an increasingly large set of languages @xcite @xcite @xcite , but also yield meaningful translations even for the smallest of those languages; and (ii) recent proposals of novel translation-based xlt strategies that have been largely uninvestigated in xlt to truly low-resource languages @xcite @xcite . contributions. in this work, we contribute to the body of translation-based xlt in light of these recent advances, focusing explicitly on low-resource target languages: 1) we offer a systematic comparison of different translation-based xlt strategies on three established benchmarks for sequence-and token-level classification, encompassing in total 40 different low-resource languages; 2) motivated by the success of multi-source training @xcite and ensembling @xcite , as well as the high quality of mt between high-resource languages, we propose two novel strategies that integrate translations from the source data to three diverse high-resource languages (turkish, russian, and chinese); we find that integrating translations to other high-resource languages substantially improves performance for sequence-level classification tasks; 3) we propose a simple and effective translation-based xlt approach for languages not covered by the mt models in which we translate from/to the linguistically closest supported language, demonstrating substantial gains over zs-xlt with mlms; 4) we introduce a translation-based model selection in which the optimal model checkpoint is selected based on performance on the validation data automatically translated to the target language; we show that this results in better performance than model selection based on source-language validation data. 5) finally, we run several ablations, offering insights into the impact of lower-level design decisionssuch as the mt decoding strategy or joint vs. sequential fine-tuning-on translation-based xlt. ## translation-based strategies most of the existing xlt work evaluates only the most straightforward translate-train (t-train) and translate-test (t-test) baselines. the former assumes the translation of the training data, available in some high-resource language (almost always english), to the target language in which inference is performed. the latter trains on the clean sourcelanguage data but, at inference time, translates the target language instances to the source language before making predictions. more recent works @xcite propose a combination of the two, which we dub roundtrip-traintest (rtt), where the source-language training data is round-trip translated (i.e., to the target language and then back) so that the translated test data at inference time better matches the training distribution, reflecting the idiosyncrasies of the same mt model. in what follows, we describe the variants of t-train, t-test, and rtt that we evaluate. figure 1 concisely illustrates all mt-based approaches under evaluation. ## experimental setup machine translation. for translation, we leverage the state-of-the-art massively multilingual nllb model with 3.3b parameters @xcite . building on prior work @xcite , we ablate over decoding strategies, including greedy decoding, nucleus sampling with top-@xmath0.8, and beam search with beam size 5. in our final evaluation, translations are generated using beam search. evaluation tasks and datasets.. following prior work on low-resource xlt @xcite @xcite , we evaluate on sequence-and token-level classification tasks covering languages un(der)represented in the pretraining corpus of our base models. in all experiments, english is the source language. foot_0 ## main results and discussion t-train vs. t-test. we first assess the widely used t-train and t-test baselines. these simple translation-based xlt strategies outperform zs-xlt dramatically: from 6.2% on masakha (t-test with xlm-r) up to 18.9% on amnli (m-trg+src), rendering them as unavoidable baselines for any xlt effort. keeping the original clean source language data in the training mix is beneficial: trg+src and m-trg+src consistently outperform trg and m-trg, respectively. for sequencelevel classification tasks (amnli and nusax), training on the concatenation of the clean source data and the source data translated to a set of related target languages (m-trg+src) yields the best results. for ner on masakha, trg+src maximizes xlt performance. we further observe that the optimal t-train (trg+src) strategy significantly outperforms (+6.2%) the best t-test approach. our t-test results also demonstrate that in low-resource xlt, mlms yield comparable performance to monolingual lms: this contradicts the recent t-test finding for high-resource languages of @xcite . ## further findings decoding strategy. previous work examined the impact of various decoding strategies on downstream performance, particularly in the context of back-translation @xcite and sequence-level classification @xcite . they found nucleus sampling consistently superior to beam search and greedy decoding. however, our results in importance of the mt model. in the mt landscape, the translation quality of the industrial-grade models is often considered superior to that of their publicly available counterparts. because of this, we ablate the impact of the used mt model on translation-based xlt by generating translations through google translate (gt)-a representative example of an industrial mt model. we evaluate t-train (i.e., trg+src), t-test (i.e., src), and rtt (i.e., rt+src) with gt. our results in we hypothesize that our observation stems from the increased translation quality which is of larger importance for t-test and rtt than for t-train @xcite . furthermore, our ablation confirms that rtt remains the most competitive translation-based xlt method for sequence-level classification tasks. unfortunately, gt does not support the exact same set of languages as nllb. we thus carry out the ablation on the following languages supported by both mt systems: for amnli: aymara, guarani, and quechua; for nusax: javanese and sundanese; and for masakha: bambara, éwé, hausa, igbo, kinyarwanda, chishona, kiswahili, akan/twi, isixhosa, yorùbá, isizulu. ## related work translation-based transfer. translation-based xlt has been adopted early @xcite @xcite yet remains a competitive baseline to date @xcite @xcite . prior work evaluated training on the translated data of a single target language @xcite , on the concatenation of all target languages @xcite , and have integrated the source language either by sequentially training first on the source followed by the translated target language data @xcite or by jointly training on the concatenation of both @xcite . while earlier approaches focus primarily on the translation of the training data (t-train), more recent work evaluated the translation of test data as well @xcite (t-test). finally, both approaches can be combined by training the model on round-trip translated noisy source data (i.e., translating source data to the target language and back to the source) and evaluating it on target language test data translated to the source language @xcite @xcite . previous studies have either focused on improving one of these paradigms or utilized them as baselines. in contrast, we provide a comparative empirical evaluation of existing translation-based approaches to xlt, testing them explicitly against zs-xlt for low-resource languages. label projection. translation-based transfer for token-level tasks necessitates label projection, which is achieved through either alignmentbased or @xcite @xcite ) markerbased approaches @xcite @xcite . the former maps each token in the source sequence to a token in the translated target sequence, with recent neural word aligners utilizing contextualized embeddings of mlms to produce the alignment @xcite . marker-based alignment, in contrast, entails marking labeled tokens in the sequence prior to translation, often by enclosing them in xml or html tags, and preserving them throughout the translation process. subsequently, the labels can be recovered from the markers. while alignment-based methods are prone to issues like error propagation, translation shift @xcite , and non-contiguous alignments @xcite , marker-based projection compromises translation performance by introducing artificial tokens and is susceptible to vanishing markers, particularly with non-industrial, publicly available translation models @xcite . in xlt for ner (masakha), we leveraged a state-of-the-art alignment-based model @xcite . ## conclusion we reviewed the field of translation-based crosslingual transfer (xlt) to low-resource languages through a comparative evaluation of various approaches-derived from translate-train (t-train), translate-test (t-test), and roundtriptrain-test (rtt)-on three established benchmarks encompassing 40 languages. we demonstrated that translation-based xlt substantially outperforms zero-shot xlt no matter the task. furthermore, irrespective of the translation-based strategy, including the clean source language data in the training yielded consistent improvements. for sequencelevel tasks, training on the source language data round-trip translated through a set of related target languages and evaluating, at inference, the target language instances translated back to the source language performed best (rtt). in contrast, for tokenlevel tasks, training on the translations to a single target language showed the best results (t-train). additionally, we proposed novel translation-based xlt strategies for t-train and rtt by including translations to a set of typologically diverse highresource languages. further, we successfully proposed translation-based strategies for languages unsupported by the mt model and showcased the effectiveness of using automatically translated validation data for model selection. our empirical comparison and its findings warrant broader inclusion of more competitive translation-based xlt approaches as standard baselines in all research efforts set to improve xlt with mlms.