Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training (2021)

Abstract: Compared to monolingual models, cross-lingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. To this end, we propose an algorithm VoCap to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VoCap benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github.com/bozheng-hit/VoCapXLM .

## introduction pretrained cross-lingual language models (conneau and lample, 2019; @xcite @xcite have recently shown great success in improving cross-lingual transferability. these models encode texts from different languages into universal representations with a shared multilingual vocabulary and a shared transformer encoder @xcite . by pretraining cross-lingual language models on the largescale multilingual corpus, the models achieve stateof-the-art performance on various downstream tasks, e.g., cross-lingual question answering and cross-lingual sentence classification. although the transformer architecture used in most pretrained monolingual and cross-lingual language models are almost identical, the vocabularies are quite different. the vocabulary sizes in existing pretrained monolingual language models typically range from 30k to 60k subword units @xcite @xcite . meanwhile, state-of-the-art pretrained cross-lingual language models use the shared multilingual vocabulary of 250k subword units to represent more than 100 languages @xcite @xcite . although some subword units are shared across languages, no more than 2.5k language-specific subword units on average are allocated for each language, which is still relatively small. besides, the multilingual vocabulary is trained on the combined multilingual corpus with subword segmentation algorithms like bpe @xcite and unigram language model @xcite . during vocabulary construction, these algorithms tend to select more subword units shared across languages with common scripts like @xcite , but have a lower chance to select language-specific subword units. it is hard to determine how much vocabulary capacity a particular language requires and whether the shared multilingual vocabulary has allocated enough vocabulary capacity to represent the language. in this paper, we propose vocap, an algorithm to allocate large vocabulary for cross-lingual language model by separately evaluating the required vocabulary capacity of each language. first, we use the average log probability (alp) to evaluate the ability of a vocabulary to represent a particular language. we find that alp is highly correlated to the downstream task performance, and we use it as an indicator to allocate language-specific vocabulary capacity. in addition, the language-specific pre-training corpus size should also be considered since the pretrained model can only learn limited knowledge from low-resource languages where the pre-training data is scarce. therefore, allocating too much vocabulary capacity for low-resource lan-guages is inefficient. vocap leverages both alp and pre-training corpus size to evaluate the required vocabulary capacity of each language. we finally allocate a multilingual vocabulary with 500k subword units with vocap and show it can significantly improve the model performance. however, increasing the vocabulary size has two practical drawbacks: slow pre-training speed and heavy model size. to address the pre-training speed issue, we propose k-nn-based target sampling, an approximate algorithm to improve the computing efficiency in the expensive softmax caused by the large vocabulary. we pre-train the model with a small subset of the entire vocabulary constructed with k nearest neighbors of the target words in current mini-batch data, evaluated with the inner product of subword embeddings. as for the model size, we halve the embedding dimension and draw a different conclusion from @xcite that increasing vocabulary from 250k to 500k with a fixed capacity model can also improve the performance. our contributions are summarized as follows: ## vocap: language-specific vocabulary capacity allocation we attribute the main factors that affect the performance of a particular language in a crosslingual language model to language-specific pretraining corpus size and vocabulary capacity. while previous work adjusts pre-training corpus size with an exponentially smoothed sampling distribution @xcite , few existing works have explored the effect of the language-specific vocabulary capacity in pretrained cross-lingual language models. in this section, we first investigate the correlation between the language-specific vocabulary capacity and downstream task performance through experiments. then we introduce our proposed multilingual vocabulary allocation algorithm vocap. ## accelerate large-vocabulary language model pre-training although extending the multilingual vocabulary benefits cross-lingual language models, pretraining with such large vocabularies brings two practical issues: slow pre-training speed and heavy model size. to tackle the issues, we first introduce our k-nn-based target sampling in section 3.1, which is a softmax approximation algorithm to improve computing efficiency. then we describe how we reallocate the model parameters to keep the model size fixed in section 3.2. ## conclusion in this paper, we study pre-training cross-lingual language models with large vocabulary capacity. first, we propose vocap to construct large multilingual vocabulary in cross-lingual language models. we conduct a quantitative analysis to show that average log probability is an valid indicator of vocabulary capacity for a particular language, which also correlates with downstream task performance on the language. vocap uses the languagespecific average log probability and pre-training corpus size to allocate appropriate vocabulary capacity for each language in the multilingual vocabulary. moreover, we propose k-nn-based target sampling to accelerate pre-training with the allocated large multilingual vocabulary by approximating the expensive softmax. we also show that reducing the embedding dimension is an effective way to keep the improvement brought by the large vocabulary without increasing the number of model parameters. the experiments demonstrate the effectiveness of the proposed vocabulary construction method as well as the acceleration methods.