Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022)

Abstract: A large body of recent work highlights the fallacies of zero-shot cross-lingual transfer (ZS-XLT) with large multilingual language models. Namely, their performance varies substantially for different target languages and is the weakest where needed the most: for low-resource languages distant to the source language. One remedy is few-shot transfer (FS-XLT), where leveraging only a few task-annotated instances in the target language(s) may yield sizable performance gains. However, FS-XLT also succumbs to large variation, as models easily overfit to the small datasets. In this work, we present a systematic study focused on a spectrum of FS-XLT fine-tuning regimes, analyzing key properties such as effectiveness, (in)stability, and modularity. We conduct extensive experiments on both higher-level (NLI, paraphrasing) and lower-level tasks (NER, POS), presenting new FS-XLT strategies that yield both improved and more stable FS-XLT across the board. Our findings challenge established FS-XLT methods: e.g., we propose to replace sequential fine-tuning with joint fine-tuning on source and target language instances, offering consistent gains with different number of shots (including resource-rich scenarios). We also show that further gains can be achieved with multi-stage FS-XLT training in which joint multilingual fine-tuning precedes the bilingual source-target specialization.

## introduction and motivation successful fine-tuning of mainstream pre-trained language models @xcite @xcite for various nlp tasks requires a sizeable set of labeled task-specific instances. while such abundant task data are available for many tasks in english and a few highresource languages, annotated examples are much scarcer for low-resource languages @xcite . a large body of recent work thus focused on zero-shot cross-lingual transfer (zs-xlt), for which no labeled instances are available in the tar-get language @xcite . catalyzed by pretrained massively multilingual transformers (mmt) such as mbert @xcite , xlm-r @xcite , or mt5 @xcite , zs-xlt has achieved impressive results on a wide variety of tasks @xcite . the mmt-driven zs-xlt, however, exhibits dramatic performance drops when transferring to low-resource languages and/or languages distant from the source language @xcite @xcite . in contrast, recent work highlights that language models are excellent few-shot learners @xcite : they adapt well to new tasks or languages when exposed to only on a handful of labeled instances. for cross-lingual transfer in particular, sequential few-shot transfer (fs-xlt) -in which large(r)scale fine-tuning in the source language is followed by the secondary fine-tuning on a few target language instances -has been rendered particularly effective, with massive performance gains reported for some tasks with as little as 10 target language instances @xcite . however, the effectiveness of sequential fs-xlt crucially depends on the shot selection @xcite . even more concerning, as we show in §3, is the sensitivity of fs-xlt to hyperparameter values, most notably the duration (number of epochs) of few-shot target language training: such fluctuations are problematic for true few-shot learning @xcite , where target language validation data, to be leveraged for model selection, does not exist. contributions. in this work, we shed new light on fs-xlt and seek to remedy the above pitfalls of current fs-xlt method. we depart from the established sequential fs-xlt paradigm and propose new training regimes for fs-xlt, comparing them across the dimensions of effectiveness, stability, and modularity. concretely, we propose training regimes that jointly exploit source and target language instances, and allow to model their interaction. 1) we demonstrate, both for higher-level semantic tasks (e.g., nli) and lower-level tokenlevel tasks (ner, pos tagging), that joint source and target language training 'feeds two birds with one scone': (i) it consistently improves fs-xlt performance, even in setups with a larger number of target-language shots (e.g., @xmath0, and (ii) makes the training procedure much more stable and robust, allowing for a reliable selection of the model checkpoint in true few-shot transfer setups without a target-language validation set. 2) we find that preceding the joint bilingual fine-tuning with a multilingual training step, in which we combine the shots from multiple target languages, brings further performance gains. we also show that such multistage training regime improves the computational efficiency in multilingual fs-xlt setups, i.e., when the model transfer to multiple target languages is required. 3) finally, we validate that benefits of the new fs-xlt training regimes are not limited to english as the source language. our work fundamentally challenges the status quo in fs-xlt and introduces and compares training paradigms that enable more effective, more efficient, and much more robust few-shot cross-lingual transfer. concurrent (closely related) effort. the concurrent work of @xcite similarly demonstrates the utility of joint multilingual fs-xlt: although their joint fine-tuning approach differs from ours -they employ gradient surgery @xcite , an approach that harmonizes competing gradients originating from instances of different languages in a training batch -it yields the same two main benefits: (1) improved target language performance and (2) more stable training that facilitates models selection (i.e., alleviates the need for target-language validation data). ## background and related work mmts like mbert and xlm(-r) @xcite have become the main vehicles of cross-lingual transfer. pretrained on multilingual corpora covering 100+ languages, mmts conceptually enable zero-shot cross-lingual transfer (zs-xlt) between any two languages seen in pretraining @xcite or even to unseen languages @xcite . the (extent of) zs-xlt success depends on the quality and alignment of the representation subspaces of individual languages @xcite @xcite . accordingly, zs-xlt with mmts tends to be ineffective in transfers to target languages that are (i) linguistically distant from the source language and especially those (ii) un(der)represented in mmt's pretraining @xcite @xcite . one line of work boosts zs-xlt by improving semantic alignment between the representation subspaces of individual languages, exploiting to this end available word or sentence translations @xcite @xcite . another, complementary line of work improves zs-xlt through increasing the mmt's capacity for individual languages @xcite @xcite . it attempts to remedy for the "curse of multilinguality" @xcite ) -an effect where, for a fixed model capacity, the quality of representations of individual languages at some point starts degrading with the addition of more languages. unlike the above efforts, which improve the mmts' representation space in a task-agnostic fashion, fs-xlt assumes a handful of labeled taskspecific examples in the target language(s) @xcite @xcite . @xcite propose sequential fs-xlt: fine-tuning on few target-language instances follows the initial fine-tuning on sizable source language data. they show that fs-xlt brings the largest gains exactly where zs-xlt fails the most: for target languages distant from the source and underrepresented in pretraining. in follow-up work, @xcite demonstrate that fs-xlt is highly sensitive to the choice of shots. both studies show the effectiveness of fewshot transfer to be subject to nature of the task: lower-level syntactic and token-level tasks (e.g., pos-tagging, ner) benefit much more from few annotated target language instances than higherlevel semantic tasks (e.g., nli). the evaluation protocols of both @xcite and @xcite , however, do not reflect a true few-shot setup: they assume that substantial validation data in the target language exists and utilize it to guide model selection (hyperparameter optimization and early stopping). as such, these works overestimate the effectiveness of true fs-xlt: while focused only on monolingual setups, @xcite demonstrate that model selection criteria based on training data alone yield in this work, we rethink fs-xlt and propose novel fs-xlt paradigms that jointly leverage both (sizable) source and (few-shot) target language data in multi-task fashion or via mix-up @xcite , and demonstrate their effectiveness as well as robustness in realistic (i.e., true) fs-xlt setups. ## methodology issues with current fs-xlt methods. figure 1 illustrates the main issues of current fs-xlt techniques, adopting the established sequential approach @xcite @xcite . in this experiment, we adapt models fine-tuned on sizable english taskspecific data with {10, 50, 100} target-language shots to americasnli @xcite and wikiann (ner) @xcite (see §4). we execute three fs-xlt runs for each target language with different randomly selected shots and examine the test performance over time, displaying the mean and deviation (±1σ) across all languages and runs for different training duration (i.e., for {1, . . . , 50} epochs of target language training). the gray horizontal line denotes the optimal performance (average across all languages and runs) in the presence of a target language validation set (i.e., 'not-true' few-shot learning): for each run, we select the checkpoint that yields the best validation performance. individual runs are denoted with colored dots, each color indicating one target language. each dot is vertically aligned with the epoch/checkpoint of the respective run (x-axis) that yields the best validation performance. the figure reveals the instability of sequential fs-xlt. 1) the optimal epoch/checkpoint varies across all dimensions of analysis: number of shots, tasks, and languages. besides the expected result that, on average, with more shots we benefit from longer training, foot_0 no discernible pattern emerges. 2) the optimal training duration substantially varies even across different runs of the same language, that is, for different random selections of n shots (and even for larger number of shots, @xmath1, cf. figure 2 later in §5.1). these observations render sequential fs-xlt highly unreliable for the true fs-xlt setups without target validation data. new fs-xlt training methods. motivated by these empirical insights, we explore new fs-xlt paradigms, aiming to increase robustness and effectiveness in true fs-xlt setups. our hypothesis is that combining abundant source-language task examples with scarce target examples in a joint fashion will 1) prevent the models to overfit to source-language features (see figure 1 ), 2) also prevent overfitting to an (extremely) small set of target-language shots @xcite , and 3) result in the models that are better calibrated for a particular source-target transfer direction. the fs-xlt methods should model the interaction between source and target examples, rather than performing source-language fine-tuning which is fully agnostic of the target language (and vice versa). the first approach, dubbed 'macro-averaging fs-xlt' (macro), conducts bilingual or multilingual fine-tuning in a joint (i.e., multi-task) setup. in particular, we compute the total loss @xmath2, where l s and l t are monolingual losses associated with the examples from the source language s and the target language t , respectively. δ is a standard interpolation hyper-parameter that adjusts the relative weight between the two losses. the two individual losses operate over the dedicated mini-batches b @xmath3, y s i } @xmath4,...,n and b @xmath5, t t j } @xmath6,...,m , which are sampled from the respective source and target language datasets d s and d t . n and m in combination determine the size |b| of the entire mini-batch, as well as the relative share of samples for each language within the mini-batch. the generalization of the bilingual macro fs-xlt method (macro-bi) to its multilingual variant (macro-multi) is straightforward: each multilingual batch b would simply comprise examples from more than 2 languages, and the joint loss will span more than 2 language-specific losses. the second paradigm is based on the standard mix-up technique @xcite . it has been proven beneficial for improving task performance and robustness in monolingual tasks; here, we extend it to the cross-lingual fs-xlt scenario. this method, termed mix-up, linearly interpolates between pairs of annotated instances from the source and the target language as follows: ## experimental setup tasks and languages. following prior studies focused on fs-xlt @xcite , we evaluate all the methods in a representative set of tasks that require varying degrees of semantic and syntactic understanding for successful cross-lingual transfer. natural language inference (nli). nli experiments are conducted on americasnli (amnli) @xcite : it encompasses indigenous target languages from the americas, with data carefully translated from the spanish xnli dataset @xcite . 3 unless stated otherwise, the source is english, and we transfer to the following 7 target languages with sizable nli data available: aymara (aym), bribri (bzd), guarani (gn), quechua (qu), raramuri (tar), shipibo-konibo (shp), wixarika (hch). for nli, we jointly embed the hypothesis-premise sentence-pair, obtain the [cls] token and feed it into the classifier. paraphrasing. the paraphrasing task is conducted on the paws-x dataset @xcite , spanning parallel evaluation data for 6 high-resource languages: german (de), spanish (es), french (fr), korean (ko), japanese (ja), and chinese (zh). we train classifiers in the same fashion as classifiers for nli, now only with paraphrase pairs. named entity recognition (ner). we use the wikiann dataset of @xcite , and evaluate cross-lingual transfer between english and the following 13 languages: arabic (ar), afrikaans (af), german (de), japanese (ja), quechuan (qu), russian (ru), kinyarwanda (rw), swahili (sw), tamil (ta), urdu (ur), vietnamese (ur), yoruba (yo), mandarin (zh). for ner, we feed output representations of each token into the classifier. part-of-speech tagging (pos). we use the pos tags of the ud treebanks (zeman et al., 2020) and transfer from english to the following 12 target languages: afrikaans (af), arabic (ar), basque (eu), chinese (zh), german (de), hindi (hi), hungarian (hu), indonesian (id), japanese (ja), russian (ru), tamil (ta), urdu (ur). the model architecture is identical to ner experiments. data sampling and shots. for amnli and paws-x, we subsample training and validation subsets from the provided validation splits. 4 wikiann and the universal dependencies treebank comprise sufficiently large training and validation splits; we subsample shots from the training data. we follow @xcite and train models with @xmath7, 50, 100, 250, 500} targetlanguage shots, fixed by task and language. 5 training details. the main mmt is the base variant of xlm-r from the transformers library @xcite with mixed precision. for all tasks, we train models with adamw @xcite with the learning rate fixed to 2e -5 and weight decay of 0.05. all models apply 10% dropout to the output representations prior to the classification layer at training time. the maximal input sequence length is set to 256 subwords for amnli and paws-x, and 512 for ner and pos. foot_5 zs-xlt and source-target variants are trained for 10 epochs with the linear warm-up rate of 0.1 and linear decay. 7 we fine-tune target regimes for 50 epochs with a constant learning rate. we train in mini-batches of size 32: the source-target regimes balance instances from source and target languages -for macro-bi, we sample 16 instances per language choose the language-balanced loss (δ = 0.5); mix-up interpolates between 32 pairs of instances between the languages, resulting with 32 'mixed' bilingual examples. for mix-up, we keep α fixed to 0.4. 8 we run all experiments over three (fixed) random seeds. further details on reproducibility are provided in appendix a.1. evaluation details. we measure performance with accuracy on amnli and paws-x. for wikiann and pos, we report the token-level f 1 score. we report both performance of final/last (l) and oracle (o) checkpoints to provide appropriate bounds on expected and ideal transfer performance. foot_8 ## results and discussion the main results are listed in ## conclusion recent work demonstrated large benefits of fewshot cross-lingual transfer (fs-xlt) with multilingual language models, where a handful of annotated examples in the target language exist, over its zero-shot counterpart (zs-xlt). however, as we have proven in this paper, prior work overestimated fs-xlt performance, relying on an unrealistic assumption of having a dedicated validation set in the target language to guide model selection. in this work, we have performed an extensive comparative study of a wide variety of fs-xlt approaches, challenging the status quo in fs-xlt. our detailed analyses have rendered established fs-xlt largely unstable and performing sub-par in true fs-xlt setups without the target validation data. we have thus proposed novel fs-xlt fine-tuning regimes that take into account interaction between sourcelanguage and target-language data instances, yielding improved, more stable, and more predictable fs-xlt performance across different tasks, languages, and numbers of target-language shots. we hope that our study will inspire better fs-xlt training and evaluation practices in future work, and guide new developments for true fs-xlt setups.