You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models (2023)

Abstract: Multilingual models have been widely used for the cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their under-representation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacher-student knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers’ knowledge into a single multilingual student. Our method outperforms standard training methods in low-resource languages and retains performance on high-resource languages while using the same amount of data. If applied widely, our approach can increase the representation of low-resource languages in NLP systems.

## introduction while multilingual language models have been gaining popularity, largely thanks to their crosslingual transfer ability, their performance has been shown to be skewed toward languages with abundant data @xcite . introducing language models that better incorporate diverse and low-resource languages can increase accessibility to nlp technologies in these languages and help improve cross-lingual transfer @xcite . in this work, we address two research questions. first, we ask if we can we improve performance on low-resource languages without hurting it on high-resource ones? second, does a better trade-off between high-and low-resource languages improve cross-lingual transfer? to answer these two questions, we distill multiple monolingual teacher models optimized for various languages into a single multilingual student * equal contribution. the order was decided by a coin toss. figure 1 : we train a student for multilingual language modeling using a collection of teachers optimized for each of the target languages and multilingual data subsampled to the data size of the lowest resource language. our approach achieves a better trade-off in performance between high-and low-resource languages. model, using a small balanced multilingual dataset (figure 1 ). our experiments show that this allows taking advantage of data in high-resource languages while avoiding under-fitting low-resource languages. we compare two alternatives for the masked lm loss functions: the original loss used for masked language modeling, i.e., hard labeling and soft labeling as defined in @xcite : where y denotes tokens in the model's vocabulary v . please note that l sof t is also equivalent to a kl-divergence between oracle and predicted distributions. in the following sections, we will explain how soft labeling allows us to distill multiple teachers into a single multilingual student while accounting for balanced performance in high-and lowresource languages. ## teacher-student distillation for multilingual language models we train a multilingual student using the maskedlanguage modeling objective and a collection of monolingual teachers optimized for each student's language. all models share one multilingual vocabulary. sharing vocabulary was necessary to apply our soft labeling loss, which requires that the student's and teacher's probability space (in the case of language models: vocabularies) are the same. 2 to avoid under-fitting low-resource languages, we naively balance the students' training data by truncating data in all target languages to the data size of the lowest resource language. to make the most out of high-resource languages, we rely on soft labeling. for a mask in a given language, we use the high-resource language-specific teacher's distribution over the mask and use it as the oracle ml in equation 2 as a soft label. our intuition is that this allows the student to gain the broader teachers' knowledge in its language and thus compensate for the sub-sampled data size. figure 1 provides a visual scheme for this approach. formally, given a set of languages @xmath0, l 2 , ..., l k }, their corresponding teachers t l 1 , t l 2 , ..., t l k , and their data @xmath1, d 2 , ..., d k } we teach the student model using the k teachers (which are trained for each of the languages). for student training, we truncate the data size of all languages in d to the smallest dataset size (min size [characters] shared script diverse script 100m english russian 100m german german 50m spanish korean 30m hungarian greek 20m vietnamese hindi 10m turkish telugu 10m basque urdu data selection and processing. we collect pretraining data from wikipedia, foot_0 aiming to capture a diverse set of high and low-resource languages, as summarized in throughout our experiments, we compare 7 languages that share the latin script versus 7 languages with varying scripts, as the script was found to be an essential factor for multilingual performance @xcite @xcite . we include german in both sets (as one of 7 languages), to compare its performance in both settings. ## experiments we validate our method using two experiments. first, we ascertain that our method indeed improves performance for low-resource languages while maintaining performance for high-resource languages. this is done by comparing the performance of our approach in masked language modeling with two multilingual baselines. second, we show that our method is competitive for downstream tasks and cross-lingual transfer by probing the pre-trained models for pos and ner tagging. multilingual modeling. we evaluate masked language modeling performance on monolingual test sets by measuring mean reciprocal rank (mrr). since the performance of multilingual models is often compared to the performances of monolingual baselines, we report the average performance difference between a multilingual model and the monolingual models trained on the same set of respective languages. downstream probing. we use the models trained in the previous experiment and train a probe, 5 keeping the base model parameters frozen, to predict part-of-speech tagging (pos) and name entity recognition (ner), as provided respectively by universal dependencies @xcite and the xtreme benchmark @xcite . 6 we chose those two tasks because they commonly appear in nlp pipelines @xcite . we measure the models' performance in two cases: when the training and test datasets are in the same language (denoted in-lang) and when a probe trained for a language l 1 is tested on another one l 2 (denoted zero-shot). as noted by @xcite , zero-shot evaluation is a good measure of a model's cross-lingual transfer. we use probing because it offers a good insight into the representation learned by the model @xcite . baselines. we compare the students' performance to multilingual models trained with hard labels, on the same data and languages as the student and its teachers. one such model was trained on all the available data in each language to examine 5 6 see section d.2 in the appendix for more information. the extent of under-fitting low-resource languages, denoted hl. additionally, to measure how much our student gains from its teacher's knowledge, we train another model on the corpora constrained to the size of the least resourceful language using the standard hard labels, denoted hl balanced. experimental setup each teacher is a monolingual model trained with hard labels. the teachers are trained on the entire training corpus available in their language. in a student model, we distill the knowledge of multiple monolingual teachers into a multilingual student using soft labels, as described above. the distillation into the student is performed on groups of shared and diverse script languages. the data is constrained to 10 million characters for each language. all our models are trained using default bert hyper-parameters detailed in section 3. ## results we report the experimental results on our test sets, in three language sets grouped by the amount of data available in pre-training, i.e., low-resource, high-resource, and all data. we address our research questions in light of the results: our method offers a good trade-off between performance on high-and low-resource languages. figure 2 shows the trend of language modeling scores (mrr) when changing from low-to highresource set. in low-resource setting, our model outperforms hl and achieves similar results to hl balanced. for high-resource languages, our approach closely trails hl and is better than hl balanced, which was trained on the same data as our student model. it indicates that the student model effectively acquires knowledge from the teachers' distributions. our model achieves the best results overall when calculated over all languages. ## related work recent work utilized knowledge distillation in training nlp models. however, to the best of our knowledge, we are the first to do this in low-resource, balanced data settings. contrary to the approaches of @xcite ; @xcite , we do not scale down student models but constraint training datasets. sun et al. (2020) finally, pfeiffer et al. (2020) introduce crosslingual adapters, compact components that allow adapting a given model pre-trained for a task in a different desired language. ## conclusions we train multilingual language models aimed at balancing the models' performance for languages with uneven data sizes. we outperform standard models for low-resource languages while maintaining performance on high-resource languages. noticeably, our method gives better results overall than the naive data sub-sampling. lastly, our model is a good representation learner for downstream tasks, outperforming baselines for two probing tasks. taken together, our results suggest a new direction for multilingual modeling that accounts for a more even performance across low-and highresource languages and improves cross-lingual transfer.