Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training (2023)

Abstract: Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen. Despite learning a small subset of parameters, this approach is not compute-efficient, as training the new embeddings requires a full forward and backward pass over the entire model. We propose mini-model adaptation, a compute-efficient alternative that builds a shallow mini-model from a fraction of a large model’s parameters. New language-specific embeddings can then be efficiently trained over the mini-model and plugged into the aligned large model for rapid cross-lingual transfer. We explore two approaches to learn mini-models: MINIJOINT, which jointly pretrains the primary model and the mini-model using a single transformer with a secondary MLM head at a middle layer; and MINIPOST, where we start from a regular pretrained model, build a mini-model by extracting and freezing a few layers, and learn a small number of parameters on top. Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the performance of the standard approach using up to 2.3x less compute on average.

## introduction recent work on multilingual nlp has focused on pretraining (masked) language models on unlabeled corpora in multiple languages @xcite @xcite . the resulting models can then be finetuned using labeled downstream data in a single language (typically english), and zero-shot transferred to the rest of the languages. while effective, existing models rarely cover more than a few dozen languages, and pretraining new models from scratch to support additional languages can be prohibitively expensive. figure 1 : average speedup of mini-model adaptation over @xcite . a speedup of x means that our approach needs x times less compute to achieve the same performance. see §4.2 for more details. motivated by this, a recent line of work has explored pretraining an initial model in a few languages, and expanding it to new languages posthoc in a continual learning fashion @xcite . more concretely, @xcite showed that it is possible to expand an english masked language model (mlm) to new languages by freezing the transformer body and learning a new embedding layer using the original mlm objective. recent work has reported improved results by using a better initialization scheme @xcite , or learning additional languagespecific parameters through adapters @xcite . all these approaches are parameter-efficient, as they only learn a small number of parameters for each language, while the rest remain frozen. however, learning such parameters is not computeefficient, as it requires a full forward and backward pass over the entire model, including the frozen transformer body. we introduce mini-model adaptation, a new approach to extend mlms to new languages that is both parameter-and compute-efficient. minimodels are shallow models that are aligned with a standard bl_base/bl_small @xcite step 1: larger parent model. thanks to this, one can efficiently train a new embedding layer for a new language over the mini-model, and plug it directly into the parent for strong cross-lingual performance. as shown in figure 2 , we explore two approaches to learn mini-models, depending on whether we start from an existing primary model and learn a mini-model posthoc (minipost), or we jointly learn the primary model and the mini-model from scratch (minijoint). in minipost, we extract the bottom layers from the existing mlm, and learn a small number of parameters on top to make it a usable small mlm itself. in the minijoint variant, we pretrain an mlm from scratch including a secondary head at a middle layer. both heads are optimized jointly, creating a complete, wellaligned mlm contained within a larger mlm. we evaluate our approach on natural language inference (xnli), question answering (mlqa) and paraphrase identification (paws-x). as shown in figure 1 , mini-model adaptation can match the performance of the standard method from artetxe et al. (2020) using 1.6x and 2.3x less compute for minipost and minijoint, respectively (averaged over tasks), and @xmath0. all in all, our work shows that it is possible to adapt language models to new tasks (in this case, new languages) using smaller aligned models for training. while we focus on the problem of crosslingual lifelong learning to validate this idea, we believe that this new paradigm opens exciting opportunities to make finetuning large language models more affordable. ## experimental settings languages and data. following common practice, we use english as the source language (l src ), and experiment with 14 other languages as the target (l trg ). we use cc-100 @xcite as our training corpus, which is a filtered version of commoncrawl. we report the full list of languages along with their corpus size and linguistic details in models. we use the roberta base @xcite architecture throughout from fairseq @xcite . embeddings are tied. as said in §2, we compare 4 systems: 2 variants of artetxe et al. models are trained for 125,000 steps with a global batch size of 2048, sequence length of 512, and learning rate of 7e-4 with 10,000 warmup updates and linear decay, both for the original pretraining ( step 1 evaluation. we evaluate on 3 tasks: natural language inference in xnli @xcite , question answering in mlqa @xcite , and adversarial paraphrase identification in paws-x @xcite . we also report xquad @xcite ) results in §a.2. in all cases, the model is finetuned using the corresponding training data in english (step 3), and zero-shot transferred into the rest of languages (step 4). we perform 5 independent finetuning runs with different random seeds, and report average results. during finetuning, we use a peak learning rate of 1e-5 for xnli and paws-x, and 3e-5 for mlqa and xquad. each uses a warmup ratio of 0.06 and linear decay, and is finetuned for 3 epochs. ## conclusion and future work our work shows that it is possible to extend pretrained models to new languages using only a fraction of their parameters. we achieve this by learning a new embedding layer over a shallow minimodel aligned with the primary model. we explore two approaches to learn mini-models: minijoint augments a transformer with a second mlm head during pretraining, adapting with an average 2.3x speedup over the standard method from @xcite , and minipost builds a mini-model by extracting a small number of layers from a pretrained model, providing an average 1.6x speedup. our analysis reveals that shallower mini-models converge faster but plateau at lower performance. as such, one might explore combining multiple mini-models of different depths, using the shallowest at the beginning of cross-lingual adaptation, and then deeper ones as training progresses. one could add multiple mlm heads to a minijoint model and train all simultaneously to facilitate this. we would also like to explore applications of mini-model adaptation beyond the multilingual scenario. in particular, by adapting rapidly on models significantly smaller than the base model used for inference, minijoint/minipost might be used to finetune large lms on modest hardware. this could allow for a new paradigm whereby one shares a small model for adaptation while keeping a large aligned model private behind an api. clients could then learn parameters for their task on the small model, which are later plugged into the large model for better performance. shortly after us, @xcite proposed offsite-tuning, an adaptation method similar to ours but motivated by privacy.