Distilling Efficient Language-Specific Models for Cross-Lingual Transfer (2023)

Abstract: Massively multilingual Transformers (MMTs), such as mBERT and XLM-R, are widely used for cross-lingual transfer learning. While these are pretrained to represent hundreds of languages, end users of NLP systems are often interested only in individual languages. For such purposes, the MMTs’ language coverage makes them unnecessarily expensive to deploy in terms of model size, inference time, energy, and hardware cost. We thus propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMTs for cross-lingual transfer. This is achieved by distilling the MMT *bilingually*, i.e., using data from only the source and target language of interest. Specifically, we use a two-phase distillation approach, termed BiStil: (i) the first phase distils a general bilingual model from the MMT, while (ii) the second, task-specific phase sparsely fine-tunes the bilingual “student” model using a task-tuned variant of the original MMT as its “teacher”. We evaluate this distillation technique in zero-shot cross-lingual transfer across a number of standard cross-lingual benchmarks. The key results indicate that the distilled models exhibit minimal degradation in target language performance relative to the base MMT despite being significantly smaller and faster. Furthermore, we find that they outperform multilingually distilled models such as DistilmBERT and MiniLMv2 while having a very modest training budget in comparison, even on a per-language basis. We also show that bilingual models distilled from MMTs greatly outperform bilingual models trained from scratch.

## introduction massively multilingual transformers (mmts), pretrained on unlabelled data from hundreds of languages, are a highly effective tool for cross-lingual transfer @xcite @xcite . however, they suffer from several limitations as a result of their ample language coverage. firstly, aiming to represent many languages within their parameter budget and dealing with the training signals from different languages might result in negative interference. this is known as the "curse of multilinguality" @xcite , which impairs the mmt's transfer capabilities @xcite . secondly, in practice people are often interested in using or researching nlp systems in just a single language. this makes the mmts unnecessarily costly in terms of storage, memory, and compute and thus hard to deploy. this especially impacts communities which speak low-resource languages, which are more likely to have limited access to computational resources @xcite . in this work, we address the question: can we increase the time-efficiency and space-efficiency of mmts while retaining their performance in crosslingual transfer? knowledge distillation @xcite is a family of general methods to achieve the first goal by producing smaller, faster models @xcite and has also been applied to mmts specifically. however, when the distilled mmt is required to cover the same number of languages as the original model, whose capacity is already thinly stretched over hundreds of languages, the "curse of multilinguality" asserts itself, resulting in a significant loss in performance @xcite . as a consequence, to achieve the best possible performance with reduced capacity, we depart from the practice of retaining all the languages from the original mmt in the distilled model. instead, we argue, we should cover only two languages, namely the source language and the target language of interest. in fact, distilling just one language would fall short of the second goal stated above, namely facilitating cross-lingual transfer, as a monolingually distilled model would be unable to learn from a distinct source language during task-specific finetuning. maintaining cross-lingual transfer capabilities, however, is crucial due to the paucity of labelled task data in many of the world's languages in most tasks @xcite . in particular, we propose a method for bilingual distillation of mmts, termed bistillation, inspired by the two-phase recipe of @xcite . we start from a "student" model, initialized by discarding a subset of layers of the original "teacher" mmt, as well as the irrelevant part of its vocabulary. in the first, "general" phase of distillation, unlabelled data is used to align the the hidden representations and attention distributions of the student with those of the teacher. in the second, taskspecific phase, the student is fine-tuned for the task of interest through guidance from a task-adapted variant of the teacher. rather than fully fine-tuning the student during this second phase, we instead use the parameter-efficient lottery-ticket sparse fine-tuning (lt-sft) method of @xcite . parameter-efficient task fine-tuning enables a system to support multiple tasks with the same distilled compact model, without unnecessarily creating full model copies per each task. we evaluate our efficient "bistilled" models on a range of downstream tasks from several bench-marks for multilingual nlp, including dependency parsing from universal dependencies (ud; zeman et al., 2020), named entity recognition from masakhaner @xcite , natural language inference from americasnli @xcite , and qa from xquad @xcite . we evaluate the model performance as well as its space efficiency (measured in terms of parameter count) and time efficiency (measured in terms of flops and inference time). we compare it against highly relevant baselines: bilingual models pretrained from scratch and two existing multilingual distilled models, distilmbert @xcite and minilmv2 @xcite . we find that while our bilingually distilled models are twice or thrice smaller and faster than the original mmt, their performance is only slightly degraded, as illustrated in figure 1 . our method outperforms the baselines by sizable margins, showing the advantages of (i) bilingual as opposed to multilingual distillation, and (ii) distilling models from mmts rather than training them from scratch. we hope that our endeavour will benefit end-users of multilingual models, and potential users under-served by currently available technologies, by making nlp systems more accessible. the code and models are publicly available at https://github.com/alanansell/bistil . ## bistillation: methodology overview. we are interested in providing nlp capabilities with limited computational resources in a specific target language t which lacks training data in the tasks of interest. a common paradigm in previous work @xcite is to use cross-lingual transfer with an mmt in conjunction with parameter-efficient task and language adaptation to support multiple tasks without adding a large number of additional parameters per task, see §2.2. our goal in this work is to replace the highly general mmt, plus optional language adaptation, with a target language-specific model which maintains the benefits of cross-lingual transfer. an obvious first attempt would be to simply distil the mmt into a smaller model using only text in the target language. however, this monolingual distillation approach is insufficient, as during task finetuning, the monolingually distilled student model no longer "understands" the source language. indeed, our preliminary experiments confirmed the intuition that this approach is inadequate. this problem can be overcome through bilingual distillation, where text from both the source and target language is used to train the student model. 1 therefore, our aim is to devise a method for deriving from an mmt m a smaller model m ′ s,t,τ to perform a given task τ in the target language t given only training data in the source language s. our approach is inspired by the two-stage distillation paradigm of @xcite . in the first, "general" phase, a bilingual student model m ′ s,t is distilled from m using the same unsupervised task (e.g., masked language modeling) that was used for m 's pretraining. in the second, "task-specific" phase, m ′ s,t,τ is produced by fine-tuning m ′ s,t using m τ as its teacher, where m τ is derived from m by fine-tuning it for task τ . the following sections explain the details of these phases. ## experimental setup we largely adopt the evaluation framework of @xcite for direct comparability with their lt-sft method, which they apply to undistilled mmts, and which we apply for task-specific fine-tuning of bilingually distilled mmts. specifically, we evaluate zero-shot cross-lingual transfer performance on four representative tasks: dependency parsing, named entity recognition, natural language inference, and qa. while the prior work focused only on low-resource languages, our method is also highly relevant to high-resource languages: the xquad qa task @xcite provides additional insight into high-resource target language performance. we experiment with three different mmts as shown in ## results and discussion til to achieve the base mmt's depth of knowledge of the target language during its relatively short distillation training time. bistil-tf, @xmath0.7 em points, despite minilmv2 receiving 320 times more training than each bidistil model, or roughly 6 times more per language 7 . furthermore, bistil-tf, @xmath1, with a 6.1 las gap on dp and 2.9 f1 gap on ner. bistil, @xmath2, once again, are trained for vastly less time 8 . training bilingual models from scratch performs poorly, lagging behind the other methods 7 minilmv2 is trained for 1m steps with a batch size of 256 and max sequence length of 512; bidistil for 200k steps with a batch size of 8 and max sequence length of 256. interestingly, when we evaluate the scratch models on their english dp performance, we obtain an average uas/las score of 81.8/77.1, which is much more competitive in relative terms with the bistil-tf, @xmath3.0/88.2 than the corresponding comparison in average target language dp scores of 29.9/11.0 to 55.5/36.5. this suggests that an even larger factor in scratch's weakness than its poor monolingual performance is a lack of alignment between its representations of the source and target languages, for ner, nli and qa, the efficiency results conform quite closely to the intuitive expectation that a model's inference time should scale linearly with its number of layers; that is, bidistil with @xmath4. for dp, we observe a seemingly sublinear scaling which is caused by the very large biaffine parsing head, consisting of ∼23m parameters. the significant cost of applying the model head contributes equally to all models regardless of their degree of distillation. despite having a moderate lrf of 2, minilmv2 exhibits impressive speed as a result of the fact that it additionally has a smaller hidden dimension than its teacher (see we argue that bidistil accomplishes its aim by achieving two-to three-fold reductions in inference time and model size without sacrificing much in the way of raw performance. its superior performance relative to multilingually distilled models despite its comparatively very modest training budget supports the assertion that specializing multilingual models for a specific transfer pair during distillation helps to avoid performance degradation resulting from the curse of multilinguality. ## related work one strand of prior work focuses on parameterefficient adaptation of pretrained mmts, i.e. adaptation by adding/modifying a small subset of parameters. adapters @xcite have been used extensively for this purpose @xcite , with the mad-x framework of @xcite becoming a starting point for several further developments @xcite @xcite , where a notable theme is adapting mmts to unseen languages @xcite . @xcite propose composable sparse fine-tunings as an alternative to adapters. @xcite create a modular mmt from scratch, where some parameters are shared among all languages and others are languagespecific. this allows the model to dedicate considerable capacity to every language without each language-specific model becoming overly large; thus it is quite similar in its aims to this work. a variety of approaches have been proposed for general distillation of pretrained language models. the simplest form uses only soft target probabilities predicted by the teacher model as the training signal for the student @xcite . other approaches try to align the hidden states and self-attention distributions of the student and teacher @xcite and/or finer-grained aspects of the self-attention mechanism @xcite . @xcite initialize the student's embedding matrix with a factorization of the teacher's for better performance when their hidden dimensions differ. of these, sanh et al. (2019); wang et al. (2020, 2021a @xcite adapt pretrained mmts to specific transfer pairs with adapters; this approach is similar to ours in spirit, but it is aimed towards improving performance rather than efficiency. @xcite learn to transfer full monolingual models across languages. the only work prior we are aware of which creates purely bilingual models for cross-lingual transfer is that of @xcite . this approach starts with a monolingual pretrained source language model, initializes target language embeddings via an alignment procedure, and then continues training the model with the added target embeddings on both languages. ## conclusions while mmts are an effective tool for cross-lingual transfer, their broad language coverage makes them unnecessarily costly to deploy in the frequentlyencountered situation where capability is required in only a single, often low-resource, language. we have proposed bistillation, a method of training more efficient models suited to this scenario which works by distilling an mmt using only the source-target language pair of interest. we show that this approach produces models that offer an excellent trade off between target language performance, efficiency, and model compactness. the 'bistilled' models exhibit only a slight decrease in performance relative to their base mmts whilst achieving considerable reduction in both model size and inference time. their results also compare favorably to those of multilingually distilled mmts despite receiving substantially less training even on a per-language basis.