XLM - V : Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models (2023)

Abstract: Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2% and 5.8% absolute on MasakhaNER and Americas NLI, respectively.

## introduction while multilingual language models have increased in parameter count and depth over time, vocabulary size has largely remained unchanged: mbart (680m parameters; @xcite , @xcite , xlm-r xxl (10.7b parameters; @xcite , mt5 xxl (13b parameters; @xcite ; and bloom @xcite all share the same 250k token vocabulary size as xlm-r base @xcite , a 250m parameter model. for models like mt5 and xlm-r, this 250k vocabulary is shared across 100+ languages. discounting shared tokens, this results in an average of 2,500 unique tokens per language, calling into question the vocabulary's ability to represent the diverse selection of languages that it was intended to model. for example, there are 8,105 characters in the table of general standard chinese characters and over 100,000 unique characters in total; the number of commonly used chinese words (consisting of multiple characters) is even larger @xcite . in fact, prior work has already shown that this vocabulary bottleneck hinders the performance of multilingual models on question answering and sequence labeling where in-depth token-level and sequence-level understanding is essential @xcite . in this paper, we construct a large multilingual vocabulary by attending to two core principles: (1) vocabularies can be improved by de-emphasizing token sharing between languages with little lexical overlap and (2) proper vocabulary capacity allocation for individual languages is crucial for ensuring that diverse languages are well-represented. then, we show that our new vocabulary exhibits favorable characteristics including the ability to frequently output semantically meaningful tokenizations while reducing over-tokenization for lowresource languages. finally, we present xlm-v, the first multilingual language model with a one million token vocabulary trained on 2.5tb of data from common crawl @xcite . our main contributions are as follows: ## analysis 6.1 the zipf ceiling we explored training models with vocabulary sizes greater than 1m tokens but found that these models perform comparatively worse on downstream tasks. we visualize the diminishing utility of increasing the vocabulary size in figure 3 . specifically, we create vocabularies with 500k, 1m, 1.5m, and 2m tokens using our methodology. then, we use these vocabularies to tokenize the flores-200 dataset. for vocabulary sizes of 500k, 1m, and 2m, we find that 99% of the content is covered by just 140,337, 197,817, and 243,832 unique tokens, respectively. ## conclusion in this paper, we presented xlm-v, a multilingual language model with a 1m token vocabulary. we showed that our model outperforms xlm-r, has outsized gains on tasks in low-resource languages, results in semantically meaningful tokenizations, reduces average sequence length, and serves as an efficient form of conditional compute. in the future, we would like to further investigate the zipf ceiling discussed in section 6 by increasing the vocabulary beyond 2m tokens while also using more data. another possible direction for future work is to explore larger multilingual vocabularies for autoregressive language models. finally, further exploration with different clustering methods such as hierarchical clustering may prove both interesting and effective.