Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging (2023)

Abstract: Massively multilingual language models have displayed strong performance in zero-shot (ZS-XLT) and few-shot (FS-XLT) cross-lingual transfer setups, where models fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s). However, current work typically overestimates model performance as fine-tuned models are frequently evaluated at model checkpoints that generalize best to validation instances in the target languages. This effectively violates the main assumptions of ‘true’ ZS-XLT and FS-XLT. Such XLT setups require robust methods that do not depend on labeled target language data for validation and model selection. In this work, aiming to improve the robustness of ‘true’ ZS-XLT and FS-XLT, we propose a simple and effective method that averages different checkpoints (i.e., model snapshots) during task fine-tuning. We conduct exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks (NLI, extractive QA) and lower-level token classification tasks (NER, POS). The results indicate that averaging model checkpoints yields systematic and consistent performance gains across diverse target languages in all tasks. Importantly, it simultaneously substantially desensitizes XLT to varying hyperparameter choices in the absence of target language validation. We also show that checkpoint averaging benefits performance when further combined with run averaging (i.e., averaging the parameters of models fine-tuned over independent runs).

## introduction and motivation massively multilingual transformers (mmt) such as mbert @xcite and xlm-r @xcite have become the main driver of multilingual nlp research. when fine-tuned on sizable task data in a high-resource source language, typically english, mmts demonstrate crosslingual transfer capabilities @xcite in zero-shot (zs-xlt; without any task-annotated instances in the target language) and few-shot (fs-xlt; only a few task-annotated instances/shots available in the target language) transfer setups @xcite . however, recent work has shown that both cross-lingual transfer (xlt) paradigms are subject to large variation in xlt performance, especially if the target language is typologically distant to the source @xcite @xcite . the protocols for model selection in previous xlt work vary broadly, which exacerbates the comparison of reported xlt results. some studies (i) do not sufficiently discuss their protocol @xcite , while others (ii) tune hyperparameters on the english development splits @xcite , or even (iii) perform model selection on the targetlanguage validation sets @xcite @xcite . assuming the availability of sufficiently large target-language validation sets for hyperparameter-tuning and model selection is unrealistic and violates the assumption of a true zs-xlt and fs-xlt setup @xcite . on the other hand, model selection on english validation data often does not correlate well with target-language performance @xcite . furthermore, benchmarking new and emerging xlt approaches with existing methods is even more challenging when the code or models from prior work are not publicly available (e.g., @xcite . 1 we therefore seek methods that reliably improve zs-xlt and fs-xlt irrespective of the underlying model and the transfer paradigm, are easy to implement, inexpensive to evaluate, robust to varying hyperparameters, and applicable to true xlt setups where the existence of any target-language validation data cannot be assumed nor guaranteed. in this work, we propose a simple and effective method of checkpoint averaging (ca) that satisfies all the desiderata above. the principal idea is to save model snapshots at periodic intervals during fine-tuning and then average the weights of the multiple single-run snapshots (i.e., checkpoints) prior to xlt evaluation. a similar procedure has been successfully adopted, for instance, in computer vision @xcite , other nlp domains such as machine translation @xcite , and speech processing @xcite ; however, it has not investigated nor adequately leveraged in xlt, notorious for its sensitivity to different choices of shots and hyperparameters. averaging model weights can be extended to merging last or multiple model snapshots from multiple model runs in a straightforward manner. as we show later, within-run snapshot averaging performs comparable, or even better in individual experiments, than the computationally more expensive ensembling of last snapshots of multiple models (i.e., from different training runs). contributions. (1) to the best of our knowledge, we are the first to extensively benchmark and analyze ca for both zs-xlt and fs-xlt; we do this on a range of higher-level semantic (nli, extractive qa) and lower-level token classification tasks (ner, pos). ca yields two benefits in true xlt setups, coming for 'free' (i.e., at no additional computation cost): the transfer performance (i) improves consistently, and (ii) it becomes much less sensitive to varying hyperparameters. (2) we shed more light on averaging models across runs (i.e., ensembling). we first confirm that standard plain ensembling (i.e., averaging the models across multiple runs) does not improve over single runs for natural language understanding tasks @xcite . we then illustrate that sizable gains from run averaging (ra) are unlocked only once models are constrained a priori to converge to more structurally similar sets of parameters. we also show that averaging the averaged checkpoints as opposed to averaging only the final models further benefits performance. further, (3) for multilingual fs-xlt, we benchmark ca against the established gradient surgery method (gs), which aims to better align gradients between languages in a batch during training for improved fs-xlt @xcite . we demonstrate that the intricate and hyperparameter-conditioned gs performs subpar to the simple ca. finally, (4) we validate that benefits of ca, ra, and their combinations extend to a variety of experimental settings for xlt, across a large number of different languages. prior work on zs-xlt thus typically aims at better aligning the language-specific subspaces for xlt. for instance, modular approaches such as adapters @xcite and sparse subnetworks @xcite extend mmt to new languages by assigning a small number of languagespecific parameters (i.e., modules) that can be combined with the base mmt. another strand of work utilizes signals from word translations or parallel data aiming to tie cross-lingual representations of languages of interest closer together @xcite @xcite . research on fs-xlt empirically validated that using even a handful of labeled instances in the target language along with source-language instances can considerably improve xlt beyond zs-xlt @xcite @xcite . fs-xlt can be stabilized and improved with (i) joint training on source-and target-language data @xcite or (ii) the so-called gradient surgery approach (gs) which 'de-conflicts' gradients between instances belonging to different languages within a training batch @xcite . in general, the methods that aim to boost xlt suffer from issues such as incurring large computational costs @xcite , require additional task-annotated data @xcite , and other external data (e.g., parallel data), which limits their wider portability to a multitude of possible tasks, domains, and languages @xcite . ## methodology motivated by the success of weight averaging discussed in §2, we hypothesize that the approach might also prove effective for xlt: weight averaging should 'denoisify' idiosyncratic variation in weights of different model snapshots, which should in turn stabilize training and improve transfer. in particular, we propose checkpoint averaging (ca) and run averaging (ra) of model snapshots for zs-xlt and fs-xlt. for ca, we first initialize the model with the parameters of the pretrained mmt: we refer to this set of parameters as θ 0 . we then fine-tune the mmt for t steps on the task data. we store the model weights k times at a regular interval of t k training steps. before inference, we then re-initialize the model with the averaged weights 1 k k @xmath0, and then use the averaged parameter set θ for inference. run averaging (ra) denotes the straightforward extension of ca to average model snapshots taken at checkpoints across r independent training runs. for ra, we put forth and evaluate two different variants. first, we can average only the model snapshots taken at the last checkpoint of each individual run. the parameters at inference for this variant, termed ra-last are then computed as here, θ i k denotes the final (i.e., k-th) model snapshot at the end of run i, @xmath1, . . . , r. the second variant, termed ra-ca, combines ca with ra: we average all k model snapshots per run over all r independent runs. effectively, we average over all k • r different model snapshots. the final set of model parameters used for inference is then computed as 1 r r @xmath2. ## results and discussion the full results for each task, dataset, and language are available in appendix a.2. in what follows, we analyse results top-down, by type of transfer, between single runs and ensembling, along metrics, and finally datasets. ## conclusion it is hard to meaningfully compare prior work on xlt: experimental setups are opaque and models are (often unreportedly) selected based on performance on english development data or even targetlanguage instances. on the one hand, selecting models based on target-language performance violates the 'zero-shot' assumption of zs-xlt and overestimates performance in both zs-xlt and fs-xlt. model selection on source-language data, on the other hand, has been proven unreliable @xcite . further, reproducing existing work on xlt is unwieldy: even if code and models are available, replication incurs a significant overhead in terms of integration efforts and computing resources. in this work, we propose to average checkpoints (ca) stored periodically in training as a simple, computationally cheap, and effective baseline for xlt that remedies for all of the above. we show that (1) ca consistently improves both zs-xlt and fs-xlt over model selection based on source-language data xlt baselines and (2) brings stability in performance across different runs. further, we propose a curriculum training that involves freezing of classifier's parameters, allowing ca benefits to propagate to ensembling, i.e., averaging of models from independent runs. we hope that future works adopts ca as a competitive and robust baseline. this would lead to more transparency and fairness in xlt evaluation, leading to more trustworthy results.