Load What You Need: Smaller Versions of Mutililingual BERT (2020)

Abstract: Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to extract smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.

## introduction while transformer based models are getting larger, there is a growing difficulty to meet production requirements when deploying them. reducing the size of these models is therefore an important step towards a democratization of transformers in real industrial environments. in addition to the model architecture, the vocabulary size may have a huge impact on the total number of parameters. however, in the case of multilingual transformers, we need to increase the model vocabulary as we include more languages. for instance, the cased version of multilingual bert (mbert) has a vocabulary of 119k entries, while the english version (bert base ) has only a 30k tokens vocabulary @xcite . therefore, even if both models share the same architecture, mbert has 178 million parameters, while bert base has only 110 million. as a matter of fact, mbert allocates more than 51% of its parameters to the embeddings layer. still, reducing the vocabulary size may induce an important drop of the model performance on downstream tasks. indeed, @xcite obtained more than 3% improvement in overall accuracy on xnli by increasing the vocab size from 128k to 512k. here, we propose to reduce the vocabulary size by reducing the number of languages the model handles. indeed, most of multilingual transformers have been learned on more than 100 languages. however, in several real world applications, we need to handle a lower number of languages. in this paper, we suggest to extract smaller multilingual transformers that handle fewer languages. here, we evaluate smaller versions of mbert on the xnli data set @xcite , but we believe that this method may be applied to other multilingual transformers and other nlp tasks. we compared our models with the original mbert and the hugging face multilingual dis-tilbert (distilmbert) @xcite . to our knowledge this is the first detailed evaluation of distilmbert on the xnli data set. the obtained results confirm that unlike distilmbert, our strategy reduces the model size without decreasing the average accuracy. the aim of this work is to draw the attention of the community on this simple yet efficient way of reducing the size of multilingual transformers. ## methods in order to generate smaller versions of mbert, we have (i) identified the vocabulary of each language, and then (ii) rebuilt the embedding layer to generate the corresponding models. ## results and discussion in this section, we present the results of the original mbert, its available distilled version (distilm-bert) and our extracted mbert versions on the xnli data set. we also discuss the obtained results and show a few limitations of the proposed method. ## conclusion multilingual transformers have several advantages such as their capacity to do zero shot cross-lingual transfer. therefore, most of multilingual transformers have been learnt to handle an important number of languages (around 100 languages). however, handling more languages requires to increase the vocabulary capacity and therefore the model size. in this paper, we evaluated a simple method to break multilingual transformers into smaller models according to the targeted languages. we evaluated smaller versions of mbert on the xnli data set and showed that they reduced the number of parameters without decreasing the average accuracy. as a future work, it would be interesting to evaluate this method on more models and tasks. indeed, we are planning to reduce more recent multilingual transformers that showed better results than mbert such as xlm-r @xcite . we hope that these models will facilitate the deployment of multilingual transformers in real world applications.