Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages (2023)

Abstract: Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers.Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training.

## introduction multilingual language models perform surprisingly well in a variety of nlp tasks for diverse languages @xcite @xcite . it has been observed that the representation of the input sequence has a significant effect on their effectiveness @xcite . in the widely used transformer @xcite models achieving state-of-the-art results through diverse tasks, a large fraction of parameters are allocated in the input encoding layer. 2 the popular language-independent approach to represent the input texts is to learn a vocabulary of frequently appearing strings that may consist of words or parts of words @xcite @xcite . 1 the code is available at: github.com/tomlimi/ entangled_in_scripts. in this work, we focus on the characteristics of subword tokenization methods in a multilingual setting. our main contribution is the introduction of the methods for measuring whether tokenizers effectively represent meaningful language-specific tokens in the vocabulary (vocabulary allocation) and whether the units they learn are shared across languages (vocabulary overlap). we posit the following questions: we address this question by training small language models utilizing different tokenization methods. we evaluate the models on masked word prediction and a diverse set of downstream tasks: pos, ner tagging, dependency tree labeling, nli, and cross-lingual sentence retrieval. the proposed evaluation scheme offers a good prediction of language models' performance. notably, we show that the system results significantly improve when tokenizers allocate more vocabulary units for specific languages. our investigation shows that this aspect has a bigger influence than the vocabulary overlap for word-level tasks (see figure 1 ). to the best of our knowledge, the interactions between multilingual vocabulary allocation and vocabulary overlap have not been investigated in past research. ## multilingual subword tokenization the majority of the currently deployed models use subword tokenization as a way to pre-process the input texts. the input is represented as a sequence of units from a finite vocabulary, which can be translated into numeric representation by an input embedding layer. the benefits of subword tokenization are the ability to obtain numeric representation for meaningful words frequently used in the resources and handling less frequent words by splitting them into subwords. the latter property mitigates the problem of out-of-vocabulary (oov) words by breaking them down into smaller parts (sub-words) already present in the vocabulary. it is crucial in handling multilingual texts, especially in languages with large vocabularies and complex morphology. in the following section, we describe two widely used algorithms of subword tokenization: 2.1 background: subword tokenization byte-pair encoding bpe: @xcite is a subword tokenization method that iteratively replaces the most frequent pair of vocabulary units in the input text with a single unit. the process starts with taking unique characters of the training text as the initial vocabulary. subsequently, we take the most frequent pair of vocabulary units, merge the pair, and add it as a new unit to the vocabulary. this process is repeated until a pre-set vocabulary size n is reached. unigram lm: @xcite is the method of obtaining subword vocabulary that was first introduced as the underlying tokenizer of senten-cepiece algorithm @xcite . the prerequisite is obtaining an extensive vocabulary, e.g., consisting of all strings present in data with at most, a predefined number of characters. the expectation-maximization algorithm is used to estimate the probability of vocabulary units. after em convergence, the portion of units with the lowest contribution to the likelihood of the training corpus is removed from the vocabulary. the procedure is repeated until the pre-set vocabulary size is obtained. @xcite observed that subword tokenizers trained on monolingual data outperform multilingual ones. the latter can overrepresent the subwords specific to languages constituting a large portion of the training corpora (e.g., english). moreover, their vocabulary is less likely to contain morphemes important in modeling low-resource languages and instead prioritizes less meaningful character sequences appearing across languages. ## measuring tokenizer properties this section presents our in-depth analytical approach to evaluate different aspects of multilingual tokenization. we introduce non-parametric measures that describe the key properties of multilingual tokenizers: quality of vocabulary representation for particular languages and lexical overlap across languages. we base our analysis on the empirical probability distribution of vocabulary units @xmath0: function f (v, c l ) is the number of occurrences of a vocabulary unit v in monolingual training corpus c l . ## evaluating language modeling and downstream tasks in this section, we present the tasks and measures for evaluation of multilingual language models trained with different tokenizers. ## experiments and results we train four tokenizers for the smaller set of diverse 6 languages (en, es, tr, el, zh, ar) using existing methods: unigram, bpe, and our methods for monolingual tokenizer merging: noover-lap, tokmix. using these tokenizers, we then train four models foot_2 following the settings of xlm- in section 5.1, we analyze the distribution of learned vocabulary units and compute vocabulary allocation and vocabulary overlap measures described in section 3. then in section 5.2, we evaluate the models' performance measures introduced in section 4 and compare them with the measures for tokenizers. subsequently, we repeat the analysis for the broader set of 20 diverse languages (including six mentioned earlier and: he, ka, ur, hi, mr, th, ta, te, bg, ru, sw, vi, fr, de) with three tokenization methods used in three pre-trained models. in this setting, we do not use nooverlap tokenizer, which cannot be trained effectively due to the necessity of constraining vocabulary for each language to n @xmath1, 000. ## conclusions we introduced a new framework for the evaluation of multilingual subword tokenizers. we show that vocabulary allocation is a crucial aspect affecting the results of many downstream tasks. specifically, we have observed the following trends: 1. including longer and more diverse vocabulary units (higher vocabulary allocation) improves inlanguage results and cross-lingual transfers for word-level tasks; 2. vocabulary overlap is beneficial for cross-lingual transfer in sentence-level tasks; 3. among languages with the same script, vocabulary overlap improves transfer for ner and deteriorates it for pos and dependency labeling. our conclusions are in line with the observation of @xcite that there is no "silver bullet solution" tokenizer suiting all purposes. we release the code for measuring tokenizer properties: github.com/tomlimi/entangled_ in_scripts. we believe that it will be a useful evaluation tool for the developers of models who can get a better insight into the tokenization method before computationally expensive model training.