When the Student Becomes the Master: Learning Better and Smaller Monolingual Models from m BERT (2022)

Abstract: In this research, we present pilot experiments to distil monolingual models from a jointly trained model for 102 languages (mBERT). We demonstrate that it is possible for the target language to outperform the original model, even with a basic distillation setup. We evaluate our methodology for 6 languages with varying amounts of resources and language families.

## introduction the introduction of the transformer architecture, which aims to solve sequence-to-sequence tasks while also handling long-range dependencies relying on self-attention @xcite , has caused a huge improvement on the state-of-theart for a wide range of natural language processing tasks. despite the clear gain in performance, these pre-trained language models are extremely data-and computation-hungry and their sizes keep growing at an incredible speed. for example, roberta @xcite is trained on a text corpus of 160 gb, and the recent collaboration of microsoft and nvidia resulted in a language model containing 530 billion parameters that was trained on 270 billion tokens @xcite . besides the ethical questions on sustainability raised by the nlp community, these huge language models also pose a lot of operational challenges, as they require massive amounts of training data, computation power and storage capacity. more importantly, the limits on training data also introduce limits on the languages these models can work with. consequently, english has been the default language newer transformers have been trained on. more recently, much attention has been devoted to multilingual aspects, especially with the advent of joint models like mbert @xcite and xlm @xcite . however, the percentage of data used to train joint models for low-resourced languages when compared to english is orders of magnitudes lesser and results in poorer representations for lower-resourced languages @xcite . moreover, when language representations for a specific low-resourced language are needed, there are no available monolingual models for a lot of these languages, and the entire large jointlytrained model needs to be loaded. more recently, a lot of attention is paid to sustainability for transformers, resulting in various approaches like pruning, weight sharing and distillation to reduce the size of large models. previous research has shown good results for task-specific distillation, distillation from larger english bert models and distillation from mbert into a smaller multilingual model (see section 2). in this paper, we attempt to combine these two research directions and distil smaller monolingual transformers from mbert. our hypothesis was that distilling would help to improve the representations for a target language, as the model can focus its prediction power on the target language, instead of attempting to accommodate 101 other languages. to the best of our knowledge, this is the first research presenting results for distilling monolingual student models from mbert. not only are we able to successfully distil smaller monolingual models from mbert, we also demonstrate that these smaller models outperform mbert for the distilled language. we experiment with student models for well-resourced (dutch, french), middle-resourced (hindi, hebrew) and low-resourced (swahili, slovenian) languages from very diverse language families and attempt to understand how a student model can outperform a teacher model in a distillation setup. ## related research while models that generate deep contextualized representations like elmo @xcite , bert @xcite and @xcite have pushed the state-of-the-art for downstream tasks in english, the improvements for medium-and low-resourced languages have not been as significant. trained models on large monolingual corpora are abundant for english and other western european languages, but they are extremely scarce for under-resourced languages (having small wikipedias to train language models on). although joint models trained for multiple languages like mbert @xcite and xlm @xcite have been an excellent alternative to monolingual models, and even perform better than monolingual models due to additional supervision, these large joint models also come with drawbacks. @xcite show that a majority of languages in mbert are underrepresented and have poor performance on downstream tasks. this can be attributed to multiple reasons, like minimal monolingual data compared to english and other western european languages, a small percentage share when it comes to mbert's shared vocabulary and the sub-word tokenization not being suitable for all scripts. however, mbert still performs better than monolingual models for these languages, since the monolingual models are trained on very limited pre-training data compared to mbert. therefore, with monolingual transformers either not existing or performing worse than mbert, one is forced to use mbert for monolingual tasks for a particular low-resourced language, and thus deploying representations for 101 other languages. @xcite propose a simple alternative by loading specific sections of mbert's vocabulary for particular languages, causing no loss in performance while decreasing the size of the deployed model considerably. various methodologies have been proposed to improve the sustainability of transformers. distillation methods have shown to be very successful, but also low-rank approximation @xcite , weight sharing @xcite , pruning @xcite and quantization @xcite propose excellent alternatives. while the central idea behind distillation is to reduce the model size as much as feasible while keeping the performance as close to the original model as possible, task-specific distillation @xcite is a variant which focuses on a certain subset of the model's capabilities. the broader concept behind both methodologies is similar: a student model is initialised with significantly lesser parameters than the original teacher model. the student is trained with the loss for the objective at hand (masked language modeling (mlm) for generic distillation and specific tasks for task-based distillation), while also forcing the predictions to be identical to the teacher model. the underlying hypothesis being that once a larger model has learned the nuances of the data, a much smaller model can simply mimic the predictions with significantly lesser parameters. @xcite further introduced the softmax temperature to emphasize learning from the entire distribution. a seminal work in the distilation of bert-like transformers was performed by @xcite , who report a considerable model size reduction (40%) while retaining 97% performance on downstream tasks. in this paper, we present pilot experiments for a new type of knowledge distillation that aims to extract representations for a particular language from a multilingual model. while there have been more advanced distillation approaches proposed, such as patient knowledge distillation @xcite , this research is a first attempt at languagebased distillation, where the basic distillation setup is validated as a proof of concept. we demonstrate that even when using basic distillation, it is possible to obtain monolingual models 3 times smaller than mbert, while also performing consistently better on the distilled language. in addition, we show promising results for languages with a large variety in terms of available resources and belonging to different language families. ## distillation methodology and experimental setup we use the basic, proven distillation technique by @xcite , which uses mbert as a teacher for all setups, and a 6-layer bert architecture without the pooler and token-type embeddings as the student. we also use the 3 loss functions proposed, which are formally represented in the equations below. the distillation loss (1) ensures the similarity between the teacher output distribution t i and the student output distribution s i using negative log-likelihood. student's output distribution while y i is the ground truth output distribution. the three losses are then aggregated with a weighted sum ( 4 ). an important factor here is the unlabelled monolingual data used for distillation. while it is often the case for distillation systems to pre-train the student model on the entire corpus to ensure optimal transfer, we want to focus on the capabilities of a single target language, and therefore only distil for iϵl where l is the target language in question. a vital thing to note here is that l mlm and l distillation can be slightly contradictory. while both enforce the same obejctive, l mlm uses the ground truth while l distillation uses the teacher's predictions. therefore, if the teacher predictions are often dissimilar to the ground-truth, the two losses might interfere with each other's progress. we explore this further in section 4. we also follow in the footsteps of the previous work for the student initialisation. it was noted by @xcite that initialising the student model from the teacher model by skipping alternate layers greatly speeds up the learning process and improves the student model considerably. our final step was to reduce the vocabulary of the student model to only the target language l. since the student vocabulary was initialized from the teacher mbert, which contains sub-words from over a 100 languages, this majorly contributes to the large vocabulary, and therefore directly to the model size and inference speed. we use the approach suggested by @xcite , which selects a subset of the vocabulary of a large model and eliminates the unnecessary parameters from the embedding layer and tokenizer, thus significantly bringing down model parameters. we replicate the distillation experiments for a set of 6 languages, using the entire available wikipedia for each language. since the data size varies considerably (ranging between 0.1 million pages for swahili to 12 million pages for french), we train for 20,000 steps to ensure all models are trained to a similar extent. the student models for dutch and french might therefore be improved by further training since they have more available data, but our objective here was to focus on the viability of the approach for low-resourced languages. experiments with the high-resourced languages (dutch, french) are only added for the sake of comparison. we use the values of 5.0 for α 1 , 2.0 for α 2 and 1.0 for α 3 , respectively. a starting learning rate of 5e -4 was used, with a batch size of 8 per device, for 4 tesla a100 gpus, and the distillation takes approximately 48 hrs per language. post-distillation, we reduce the model's vocabulary as described above. next, we test the monolingual student models on two different downstream tasks for each language. for the fine-tuning, we add a classification layer to the distilled model and train with an lr of 5e -5 for 5 epochs with 500 warm up steps before a linear lr decay. we include tasks requiring both semantic understanding (sentiment analysis, news classification) as well as syntactic knowledge (pos-tagging, ner). an overview of the evaluated tasks and datasets per language is presented in ## results and discussion the results for all experiments are summarized in to conclude, we demonstrate that it is possible to distil better, smaller and faster monolingual models from mbert, using a very basic distillation setup. the results show that the eliquare monolingual models consistently outperform mbert which has 3 times more parameters, and distilmbert which has almost 2 times more parameters and uses orders of magnitude more data for the distillation process. while the eliquare models may not be useful for the high-resourced languages due to the availability of large monolingual models, they present a major step towards having small monolingual models for a number of low-resourced languages. in future research, we will investigate how to further improve the student from the teacher by diminishing the impact of the distillation loss. in addition, we will run the same set of experiments with xlm-r to investigate whether the same distillation approach can be applied to other joint multilingual models as well. finally, we will also explore the impact of distilling multiple typologically similar languages from mbert in parallel. and the distillation loss for the other 5 languages, and consistently observe the findings presented in section 4. the mlm loss continues to have a much larger mean value than the distillation loss, while also having issues converging with various spikes in the loss, while the training for the distillation loss is much more stable. this is line with our hypothesis that the large α 1 values prioritize the distillation over the language modelling objective, thus not allowing the model to further improve from the teacher.