Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity (2023)

Abstract: The scarcity of data in many of the world’s languages necessitates the transfer of knowledge from other, resource-rich languages. However, the level of scarcity varies significantly across multiple dimensions, including: i) the amount of task-specific data available in the source and target languages; ii) the amount of monolingual and parallel data available for both languages; and iii) the extent to which they are supported by pretrained multilingual and translation models. Prior work has largely treated these dimensions and the various techniques for dealing with them separately; in this paper, we offer a more integrated view by exploring how to deploy the arsenal of cross-lingual transfer tools across a range of scenarios, especially the most challenging, low-resource ones. To this end, we run experiments on the AmericasNLI and NusaX benchmarks over 20 languages, simulating a range of few-shot settings. The best configuration in our experiments employed parameter-efficient language and task adaptation of massively multilingual Transformers, trained simultaneously on source language data and both machine-translated and natural data for multiple target languages. In addition, we show that pre-trained translation models can be easily adapted to unseen languages, thus extending the range of our hybrid technique and translation-based transfer more broadly. Beyond new insights into the mechanisms of cross-lingual transfer, we hope our work will provide practitioners with a toolbox to integrate multiple techniques for different real-world scenarios. Our code is available at https://github.com/parovicm/unified-xlt.

## introduction while training data is available for a wide range of nlp tasks in a handful of high-resource languages, the vast majority of the world's languages with their billions of speakers do not fit into this * equal contribution. category @xcite . the lack of labelled data makes it difficult or impossible to directly train effective nlp systems for these languages. for this reason, researchers have looked for ways to harness data from one or more high-resource "source" languages to compensate for a shortage of data in low-resource "target" languages in a process known as "cross-lingual transfer" (xlt). different techniques have been developed to deal with the various dimensions of resource scarcity, which encompass not just data availability, but also the degree of support by pretrained models. these research threads have generally been investigated somewhat independently. in this paper, we attempt to unify several of the most prominent threads of xlt research into a single framework. specifically, we synthesise findings from zero-shot (zs) xlt with massively multilingual transformers (mmts), few-shot (fs) xlt, xlt for low-resource languages and xlt through machine translation (mt) to formulate a practical, general-purpose approach to cross-lingual transfer, with a focus on low-resource scenarios. massively multilingual transformers (mmts), transformer-based architectures @xcite pretrained with an unsupervised objective such as masked language modelling (mlm) on text from a large number of languages, are perhaps the most fundamental tool for contemporary xlt. prominent examples of mmts include mbert @xcite , xlm-r @xcite and mdeberta @xcite . in addition to providing broad language coverage, mmts have been shown to learn representations which have a degree of cross-lingual alignment, even though they do not receive any explicit cross-lingual signal during training @xcite . this allows an mmt fine-tuned for a specific task in a given "source" language to perform the same task in another "target" language with a level of performance generally much better than random chance @xcite , despite never having seen a single example of the task in the target language; this is known as "zero-shot" cross-lingual transfer. while mmts typically cover around 100 languages, this is still only a small fraction of the world's estimated 7,000 languages. @xcite and @xcite have shown that an effective strategy for zs-xlt for target languages not covered by the mmt is to learn a parameterefficient fine-tuning (peft) to specialise the mmt to that "unseen" language. the resulting language module can be composed with a task module, typically yielding much better performance than the mmt could achieve without language adaptation. in contrast to zs-xlt, often more realistic is the few-shot case (fs-xlt), where a small number of gold-standard target language examples are available during training. though it may be expensive to annotate target language data, especially for lowresource languages where native speakers are hard to access, prior work has shown that using even a small amount during training can yield significant gains in performance @xcite . while early approaches to fs-xlt involved fine-tuning first on the source language data, then separately on the few target language shots @xcite , recent work has shown that it is more effective to jointly train on both at once @xcite . another tool often employed for cross-lingual transfer is machine translation (mt). mt approaches can generally be categorised as translatetrain or translate-test @xcite . to confine the scope of our work, we consider only the translate-train approach, which has so far been predominant, although @xcite have made a strong case for considering translate-test further in future work. we consider two translate-train variants: ttrain-single, where a model is trained for each target language using only its own translated data; and ttrain-all, where one model covering all target languages is trained on their translated data and the source language data simultaneously. in this work, we consider how best to employ the above techniques in response to cross-lingual transfer scenarios with varying levels of data scarcity. we thus explore several promising directions of integrating the zero-shot, few-shot and translatetrain techniques across a range of resource levels in order to delve deeper into: (i) to what extent these techniques and the different data sources that they exploit are complementary, (ii) what is the most effective way of combining different data sources in order to maximise the performance, and (iii) how much each of the available sources of data contributes to the overall performance. we aim to equip practitioners with a recipe for how to use the available data resources in the most effective way. we experiment on the americasnli natural language inference (nli) dataset for american languages @xcite , and the nusax sentiment analysis (sa) dataset for indonesian languages @xcite . we find that combining language adaptation, few-shot learning and translation can be highly effective, yielding average performance gains of 14-24 points over the zero-shot baseline without language adaptation. ## results and discussion main results. the results of our primary configurations on nli and sa are presented in with ablations shown in ## related work parameter-efficient fine-tuning methods. parameter-efficient fine-tuning methods have emerged from a necessity to reduce compute and memory requirements of fine-tuning when dealing with large models. these methods can generally be grouped into those that modify the subset of parameters of the pretrained llm @xcite @xcite and those that introduce a completely fresh set of parameters to be updated @xcite @xcite allowing for different interactions with the pretrained model. they have been adopted for cross-lingual transfer as they are preferable when dozens of different fine-tunings for different languages and tasks need to be learned, stored and combined @xcite @xcite . for a comprehensive overview of parameter-efficient fine-tuning, we refer the reader to @xcite . few-shot cross-lingual transfer. @xcite and later @xcite demonstrate the effectiveness of few-shot over the zero-shot cross-lingual transfer, showing that continued training of a source-trained model on a small number of labelled examples in the target language significantly increases performance (target-adapting). @xcite trade-off efficiency for performance by replacing the sequential fine-tuning procedure of @xcite with joint training on source and target language, showing it also improves training stability and robustness. they additionally show that first fine-tuning on multiple target languages provides extra performance gains. we adopt their joint training procedure, combining it further with parameter-efficient fine-tuning methods and language adaptation. xu and murray (2022) also exploit a joint source-and-target training procedure, further extending it to all target languages simultaneously instead of having language-specific models, which becomes particularly attractive when dealing with a large number of target languages. they also introduce stochastic gradient surgery to circumvent the issue of conflicting gradients among languages. jundi and lapesa (2022) compare few-shot and translation-based approaches, trying to gain an insight into which approach is better and under which circumstances. we consider these approaches in combination rather than in competition and find that the use of few-shot data can enhance performance even when a machine translation of the full source language dataset is available. however, their work complements ours by proposing a way to identify the examples which may be most profitable for humans to translate into target language "shots." @xcite study few-shot cross-lingual transfer on languages unseen by mmts using the nusax dataset. they analyse the effectiveness of several few-shot strategies focusing on selecting languages for transfer and different learning dynamics exhibited by different types of mmts. machine translation for cross-lingual transfer. the translate-train and translate-test approaches are common baselines for cross-lingual transfer @xcite . a number of enhancements have been proposed. @xcite showed that translate-test performance could be improved by training on backtranslated rather than the original source language data to better model translation artefacts encountered at inference time. @xcite note that translation-based approaches suffer from an error accumulation over the phases of the pipeline. they re-interpret this pipeline as a single model with an intermediate "latent translation" between the target text and its classification label, permitting the translation model to be fine-tuned according to a feedback signal from the task loss. @xcite show that the translate-train and translatetest approaches can be combined synergistically. @xcite show that translate-test is more favourable relative to translate-train than previously thought when better translation and monolingual models are used, and when measures are taken to correct the mt-induced mismatch between the data encountered at train and inference time. while we only consider applying translation-based crosslingual transfer to classification tasks, prior work has considered its application to sequence labelling tasks as well @xcite @xcite . for simplicity, we employ only continued pretraining on the standard mt task when adapting nllb to unseen languages. @xcite enhance nmt model adaptation with additional tasks: denoising autoencoding, which exploits monolingual target language data; back-translation; and adversarial training which encourages the encoder to output language-agnostic features. ## conclusions and future work we have investigated how to combine several crosslingual transfer techniques which are applicable across several dimensions of resource scarcity into a single framework. we find that parameterefficient language adaptation, few-shot learning and translate-train are complementary when employed in a multi-source training setup with fewshot upsampling. however, our training setup supports the use of any subset of these techniques depending on the availability of the necessary data and models. we remark on the significance of the finding that gold-standard few-shot target data can improve performance even when the entirety of the training data is translated into the target language by a high-quality nmt model. we also observe that languages not natively supported by an nmt model can benefit from translate-train through a simple adaptation procedure even with a small amount of parallel data.