Efficient Multilingual Language Model Compression through Vocabulary Trimming (2023)

Abstract: Multilingual language models (LMs) have become a powerful tool in NLP, especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabulary covering tokens in different languages. Instead, monolingual LMs can be trained in a target language with the language-specific vocabulary only. In this paper, we propose vocabulary-trimming (VT), a method to reduce a multilingual LM vocabulary to a target language by deleting potentially irrelevant tokens from its vocabulary. In theory, VT can compress any existing multilingual LM to any language covered by the original model. In our experiments, we show that VT can retain the original performance of the multilingual LM, while being considerably smaller in size than the original multilingual LM. The evaluation is performed over four NLP tasks (two generative and two classification tasks) among four widely used multilingual LMs in seven languages. The results show that this methodology can keep the best of both monolingual and multilingual worlds by keeping a small size as monolingual models without the need for specifically retraining them, and can even help limit potentially harmful social biases.

## introduction multilingual language model (lm) pre-training @xcite @xcite has been shown to be an efficient mechanism to store information from many languages into a single model, without the need for training multiple language-specific models. moreover, it has been proven reliable for cross-lingual tasks @xcite and can provide competitive performance in most settings, generally similar to its monolingual counterparts @xcite , while being generally less affected by culturally-dependant biases @xcite . similarly to monolingual models, multilingual lms can be used for zero/fewshot learning @xcite ) by increasing the model size and, more frequently, can be specialized to different tasks by fine-tuning to specific data. in practice, however, there are a few practical issues when training multilingual lm such as the curse of multilinguality @xcite , a trade-off between the number of languages and individual performance in a single language, or the multilingual vocabulary construction, which requires a careful design for better generalization @xcite @xcite . besides such generalization concerns, multilingual lms usually consist of larger parameters than their monolingual counterparts due to the need for a large vocabulary covering multiple languages. this becomes an important issue in practice when the resources to host models are limited. for instance, while using the same configuration (i.e., same number of layers and hidden units), the parameter size of t5 small @xcite and mt5 small @xcite are 140m and 300m, respectively. this is only due to their difference in vocabulary size, with t5 being 50k and mt5, 250k. in fact, the embedding matrix stemming from the lm vocabulary can occupy a large portion of the parameter space. for instance, the ratio of the embedding matrix to the full model's parameter size in multilingual lms can be higher than 80% as t5 (see figure 1 ). in this paper, we propose a simple vocabulary trimming (vt) method to remove tokens from the vocabulary of multilingual lms that may be irrelevant to the target language. foot_0 this is achieved by automatically identifying language-specific tokens from an underlying text corpus. we consider two vt strategies of pre-ft vt (vt before finetuning) and post-ft vt (vt after fine-tuning) and analyse them by varying the final vocabulary size. we conduct experiments on two generation tasks, question answering (qa) and question generation (qg), and two classification tasks, sentiment analysis and natural language inference (nli), across seven different languages. the experimental results show that both pre and post fine-tuning vt can reduce the model size while retaining the original performance in generation tasks (qa and qg), and particularly in classification tasks (sentiment and nli) where the results are close to being identical despite the significant reduction in vocabulary size. in all tasks, the original performance can be generally maintained with less than 40% of the full model parameters for all languages. finally, even though pre-trained lms have reported impressive performance on various nlp downstream tasks @xcite @xcite , such lms also demonstrate worrying levels of social biases in certain situations @xcite @xcite . one natural question that arises is whether vt can have an influence on the bias level in multilingual lms, including fine-tuned models. for this purpose, we evaluate social bias in multilingual lms after applying vt with different settings and compare it against its monolingual counterpart. experimental results show that the monolingual lm tends to contain more bias than its multilingual versions. moreover, compared to the original multilingual lm, the bias level has no significant change after applying vt. these results suggest that a monolingual lm can be induced by applying vt to its corresponding multilingual lm, thereby obtaining a less biased monolingual lm compared to its original monolingual counterpart. ## related work several studies have explored the possibility to modify or adapt the vocabulary of lms. for instance, @xcite al. (2020) 2020 ) to efficiently enlarge the vocabulary size along with the model size scaling. ostendorff and rehm (2023) used a multi-stage fine-tuning to obtain a lm in the target language from other lm in the source language. these prior works modify existing mono/multi-lingual lms to include new languages, i.e. augmenting the multilinguality of the lms. in contrast, our study focuses on compressing multilingual lms into the target language to effectively achieve smaller monolingual lms, i.e. reducing the multilingual representation of the lms while retaining the capability in a specific target language. the work of @xcite is the most relevant to our study as, to the best of our knowledge, they introduced the idea of vt for the first time. however, their analysis is limited to nli with pre-fine-tuning vt with mbert @xcite only, as well as a fixed vocabulary size after vt. in contrast, our study compares two vt strategies, before and after fine-tuning, and show how this latter strategy, not considered in @xcite , can be a more effective compression technique in some settings. furthermore, we extend the experiments to generation tasks as well as classification tasks with more recent lms such as mbart @xcite and mt5 @xcite , and provide an exhaustive analysis on the effect of vt. ## vocabulary trimming to perform vocabulary trimming (vt), we first need a multilingual lm as an input. the idea is to tailor model to a particular target language l, which in principle belong to the same set of languages used to trained the input multilingual lm. 2 for the target language l, vt first identifies languagespecific tokens on a language-specific corpus c l , and remove all the tokens along with their embeddings except for those appeared in c l as described in figure 2 . in our analysis ( § 5), we also consider to keep the top-n most frequent tokens in c l to further reduce the model size by removing less frequent tokens. we consider two vt strategies: (1) before fine-tuning and (2) after fine-tuning. the difference between these two strategies is whether to perform vt before or after fine-tuning, as shown in figure 3 . both vts have advantages and drawbacks: while pre-ft vt can reduce the time of fine-tuning as the trimmed lm is smaller than the original lm, post-ft vt only need a finetuned multilingual lm -this way, post-ft vt can be used as a postprocessing step and no additional language-specific training is required. finally, we release a simple lm vocabulary trimming starting package to apply our proposed technique to any input multilingual transformerbased lm, along with all the models and code needed to reproduce our experiments, at https: //github.com/asahi417/lm-vocab-trimmer. ## evaluation in this section, we present our experimental results to test the reliability of our proposed vt methodology in nlp tasks. ## vocabulary size analysis in our main experimental results ( § 4.2), all the unique tokens that appeared in the monolingual corpus were kept, which resulted in a low compression ratio for some languages such as english and russian. in this analysis, we constrain the number of vocabulary and choose the top-n vocabulary at vt in terms of the frequency in the corpus (see § 3). for qa and qg, we compare mt5 small results with n from [5k, 10k, 15k, 30k, 60k, 90k, 120k], which correspond to an overall parameter size of [49m, 54m, 59m, 74m, 105m, 136m, 166m], respectively. for sentiment analysis and nli, we experiment with xlm-r base with n from [5k, 10k, 15k, 30k, 60k], which correspond to [89m, 93m, 97m, 109m, 132m] of parameter size, respectively foot_2 . ## discussion vocabulary trimming before and after finetuning. according to the results, pre-ft vt appears to be generally more effective in classification tasks (see section 4.2.2). for generation tasks, both pre/post-ft vt robustly retain the original performance while being able to considerably reduce the model size (see section 4.2.1). as a guideline to choose the type of vt in such a case, post-ft vt should be more suitable if one already has a finetuned model, as no additional training is needed for this case. moreover, post-ft is more robust as a compression mechanism as the performance is largely maintained with respect to that of the original multilingual lm. on the other hand, if one needs to fine-tune a model from scratch and the computation resources are limited, we recommend exploring pre-ft vt, as fine-tuning on a trimmed lm should be more efficient due to its smaller vocabulary and parameters and, in some cases, can lead to better overall results. however, this process has to be done carefully as the set of optimal parameters could differ from the original multilingual lm fine-tuning process. ## conclusion in this paper, we proposed vocabulary-trimming (vt), a method to reduce the vocabulary of a multilingual lm to a vocabulary specific to any given target language. vt can induce a monolingual lm in the target language by leveraging an existing multilingual lm. the main advantage of this filtering step is the reduced size, as well as avoiding having to train monolingual lms from scratch, which would be computationally demanding. our experiments show how vt can retain the high performance of the original multilingual lm, while largely reducing the model size. for all languages evaluated, a 35% compression rate proves sufficient to keep the original performance of the larger mt5 multilingual lm in both qa and qg, with a similar 39% in nli and 55% in sentiment analysis with xlm-r. interestingly, in some cases, the compressed lm can even achieve better results than the original larger model when trimmed before fine-tuning. since the main goal of the paper was to compress a multilingual lm while keeping its original performance, we leave the analysis of this behaviour for future work.