One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023)

Abstract: Multilingual language models enable zero-shot cross-lingual transfer (ZS-XLT): fine-tuned on sizable source-language task data, they perform the task in target languages without labeled instances. The effectiveness of ZS-XLT hinges on the linguistic proximity between languages and the amount of pretraining data for a language. Because of this, model selection based on source-language validation is unreliable: it picks model snapshots with suboptimal target-language performance. As a remedy, some work optimizes ZS-XLT by extensively tuning hyperparameters: the follow-up work then routinely struggles to replicate the original results. Other work searches over narrower hyperparameter grids, reporting substantially lower performance. In this work, we therefore propose an unsupervised evaluation protocol for ZS-XLT that decouples performance maximization from hyperparameter tuning. As a robust and more transparent alternative to extensive hyperparameter tuning, we propose to accumulatively average snapshots from different runs into a single model. We run broad ZS-XLT experiments on both higher-level semantic tasks (NLI, extractive QA) and a lower-level token classification task (NER) and find that conventional model selection based on source-language validation quickly plateaus to suboptimal ZS-XLT performance. On the other hand, our accumulative run-by-run averaging of models trained with different hyperparameters boosts ZS-XLT performance and closely correlates with “oracle” ZS-XLT, i.e., model selection based on target-language validation performance.

## introduction and motivation massively multilingual transformers (mmts) like xlm-{r,v} @xcite or mt5 @xcite are pretrained via language modeling on vast corpora encompassing 100+ languages. mmt fine-tuned on labeled task data in a source language can transfer cross-lingually zero-shot, i.e. without further annotations, to target languages @xcite . however, pretraining corpora size and linguistic distance between the source and target language dictate the quality of xlt @xcite . this is why model selection based on source-language validation data unreliably correlates with zs-xlt and selects checkpoints that yield suboptimal target-language performance @xcite . worse yet, there is no "best practice" for replicating zs-xlt results of prior work. some works, as our results suggest (cf. §4), (1) exhaust extraordinarily large hyperparameter grids and (2) monitor target-language performance for the best transfer (i.e., violating "true" zs-xlt) to outperform baselines @xcite . other works rerun baselines with little to no hyperparameter tuning @xcite : the re-evaluation then often trails original results by non-negligible margins. 1 as a remedy, @xcite propose to evaluate zs-xlt on the snapshot that generalizes best to validation data in the target language ("oracle" zs-xlt): as such, oracle zs-xlt stabilizes evaluation and denotes ideal transfer performance. nonetheless, oracle zs-xlt overstates the performance of true zs-xlt, for which no target-language instances are available @xcite . if they are, targetlanguage annotations are always better levered for training than for validation @xcite . this calls for an evaluation protocol that (1) maximizes "true" zs-xlt results and (2) makes them easily reproducible, regardless of the extent of hyperparameter tuning. in this work, we find that model averaging fulfills both criteria. weights averaging has proven effective in, e.g., mt @xcite and recently nlu @xcite . @xcite enable model averaging for sizable gains in xlt. they first fine-tune an mmt on labeled source-language data and then re-train models (i.e., more runs) by copying and freezing the task head of the initially fine-tuned model: this aligns snapshots and enables weight averaging across runs. 2 contributions. in this work, we propose an evaluation protocol that decouples maximizing zs-xlt performance from hyperparameter tuning. the key idea is to accumulatively average snapshots of runs with different hyperparameters: this improves performance over model selection based on sourcelanguage validation performance. we run exhaustive experiments on higher-level (nli, extractive qa) and lower-level (ner) nlu tasks on a broad grid of hyperparameters and show, examining the cross-section of all runs, that model selection based on source-language validation almost exclusively picks snapshots suboptimal for zs-xlt. we also confirm that conventional hyperparameter tuning on source-language validation prematurely settles for models that maximize source-language performance at the expense of zs-xlt. crucially, we show that accumulative model averaging performs on par or better than the best snapshot picked by source-language validation already from the second (i.e. first averaged-in) run and then consistently improves zs-xlt with more runs. we additionally show that this accumulative model averaging closely correlates with oracle zs-xlt without requiring any source-or target-language validation data to maximize transfer performance. ## accumulative run averaging prior work conducts model selection for zs-xlt by extensive hyperparameter tuning using either source-or target-language validation data. whereas the latter violates true zs-xlt @xcite , the former overfits to source-language performance @xcite . the recent success of snapshot averaging in xlt @xcite motivates our research question: can (accumulative) averaging of models trained during hyperparameter search outperform -with fewer overall training runs -the zs-xlt performance of the "optimal" model selected based on source-language validation performance? we benchmark model selection based on source-language validation against accumulative model averaging as follows. we iteratively sample models (i.e., runs with different hyperparameters (i.e., pairs of learning rates and batch sizes) from the pool of runs containing n runs per hyperparameter configuration (cf. appendix §a.2). we repeat this procedure 10 times and report mean zs-xlt performance. standard model selection picks the model {arg max i val(θ i ) | 1 ≤ @xmath0, capturing the "true" ("oracle") zs-xlt performance. "accumulative averaging", in contrast, naively averages (i.e. without any supervision) all models of r runs to a single model 1 r r @xmath1. ## experimental setup tasks and languages. we select for our evaluation two higher-level semantic tasks (nli and and extractive qa) and one lower-level structured prediction task (ner). for each task, we fine-tune the mmt on the provided english training splits. 3 natural language inference (nli). we evaluate nli on xnli @xcite and in-dicxnli @xcite which together cover 25 typologically diverse languages. extractive qa (tydiqa-goldp). tydiqa-goldp comprises questions that are answered by a span of text in the provided gold passage and covers 9 diverse languages @xcite . named entity recognition (ner). we evaluate ner on 25 languages from wikiann @xcite , 10 languages from masakhaner @xcite , and 9 languages from masakhaner 2.0 @xcite . ## conclusion inconsistent hyperparameter tuning and model selection protocols exacerbate replicating previous results on zs-xlt. in this focused study, we devise a zs-xlt evaluation protocol that addresses previous shortcomings and feeds two birds with one scone. we show that accumulatively averaging snapshots -rather than selecting models based on source-language validation performance -both improves and stabilizes zs-xlt. conventional model selection strategies prematurely settle for models that maximize source-language validation performance and discard runs that generalize better in zs-xlt. accumulative model averaging both incorporates snapshots that transfer well and irons out models that perform badly. we find that model averaging correlates closely with "oracle" zs-xlt, which assumes models selection on target-language validation instances. we hope future work adopts model averaging to promote fair and reproducible zs-xlt that puts models on equal footing.