Improving Multilingual Models with Language-Clustered Vocabularies (2020)

Abstract: State-of-the-art multilingual models depend on vocabularies that cover all of the languages the model will expect to see at inference time, but the standard methods for generating those vocabularies are not ideal for massively multilingual applications. In this work, we introduce a novel procedure for multilingual vocabulary generation that combines the separately trained vocabularies of several automatically derived language clusters, thus balancing the trade-off between cross-lingual subword sharing and language-specific vocabularies. Our experiments show improvements across languages on key multilingual benchmark tasks TyDi QA (+2.9 F1), XNLI (+2.1%), and WikiAnn NER (+2.8 F1) and factor of 8 reduction in out-of-vocabulary rate, all without increasing the size of the model or data.

## introduction multilingual models such as mbert @xcite , xlm @xcite , and xlm-r @xcite have built on the advances of deep contextualized language modeling by pretraining on texts from many languages at once. one trait common to all of these models is the use of a single vocabulary containing subwords from all languages, used to segment the input text before transforming it into a sequence of embeddings. @xcite showed that increasing the vocabulary size can produce quality gains, but unlike similar monolingual models, the vocabulary embedding matrix in each of these multilingual models constitutes a significant fraction of its total parameters; for example, 47% of xlm-r's parameters are in its embedding matrix. therefore, scaling up a model's vocabulary size requires the construction of inductive biases that * work done as a member of the google ai residency program. the multilingual subword vocabularies used by the state-of-the-art models are generated by algorithms such as wordpiece @xcite , sentencepiece @xcite ), foot_0 or byte pair encoding (bpe) @xcite . given a desired vocabulary size, these algorithms select an inventory of subwords that compactly represents the training corpora, which means preferring subwords that occur frequently, and, by extension for multilingual models, occur frequently across languages. because these algorithms look at overall subword frequencies in the combined multilingual corpus, they may learn suboptimal decompositions for low-resource languages that happen to have character combinations that resemble subwords of high-resources languages. 2 additionally, subwords in common scripts like latin and cyrillic have a higher chance of selection since their counts are combined across a large number of languages @xcite . by attempting to optimize for the best overall vocabulary across all languages without regard for the differences among those languages, the joint procedure over-emphasizes cross-lingual subword sharing-even across languages with little lexical overlap-which is at odds with the finding of k et al. (2020) that subword sharing is not the principal reason for the effectiveness of multilingual models. it also under-emphasizes the need for all languages-particularly low-resource languages or those written in scripts used by few languages-to contribute subwords that are most effective for their own representations. in this paper, we propose a novel approach to multilingual vocabulary generation that seeks to balance the trade-off between optimizing for crosslingual subword sharing and the need for robust representation of individual languages. at a high level, we: 1) automatically group languages into clusters based on the distributional similarities of their individual subword vocabularies, 2) apply the sentencepiece algorithm separately to the data for each individual cluster, and finally 3) combine all cluster-vocabularies together to form a single unified multilingual model vocabulary. we evaluate our approach on three distinct downstream tasks: tydi qa, xnli, and wikiann ner. our experimental results show that our method improves model performance on all three tasks, and achieves a state-of-the-art result for tydi qa and zero-shot cross-lingual ner. crucially, our method improves performance without any changes to the model, and since it does not depend on the model architecture, it can be applied to any model that uses a vocabulary. we begin by defining a vector representation for each language (figure 1 ). for each language l in the set of languages l, we generate an l-specific vocabulary v l by running the sentencepiece algorithm on l's corpus. then we take the union of the resulting vocabulary to form the global vocabulary, v @xmath0. next, for each language, we form a binary vector v l of dimension v l , where each component of v l corresponds to a subword in v l , i.e., with each language encoded as a vector v l , we can apply standard clustering algorithms to assign each l to some c j in the set of clusters c. for our experiments, we used k-means with cosine distance. preliminary experiments ( note that our method is a generalization of the conventional approach, which has |c| = 1. ## intrinsic analysis in this section, we directly examine the vocabularies produced by the standard recipe (joint) and our clustering-based method (cluster) in order to assess the degree to which each approach is able to capture and balance the inductive biases introduced in §1: a multilingual vocabulary should encourage subword sharing across languages when appropriate, but each language should also have the freedom to contribute the subwords that are most effective for its own representation. for all of our experiments, we generate vocabularies from the wikipedia articles of 104 languages with 906m sentences. each multilingual vocabulary is 488k subwords. the cluster assignments and sizes generated by cluster are found in in order to quantify the extent of subword sharing between languages, we look at each language as a distribution over the vocabulary. in particular, we apply the multilingual vocabulary v 's segmentation to each language l's monolingual corpus in order to count the frequency of each sub- name clusters |v c | c 1 af, sq, hy, az, bn, bs, my, ceb, hr, en, fi, ka, el, he hu, id, ga, jv, lv, lt, ms, min, pl, pa, sco, hbs, sl su, sw, tl, th, tr, uz, vi, cy, yo 200,306 c 2 ar, bpy, fa, azb, ur, lah 40,218 c 3 an, ast, eu, ca, gl, io, it, la, lmo, oc, pms pt, ro, scn, es, war 80,764 c 4 ba, be, bg, ce, cv, kk, ky, mk, mn, ru, sr, tg, tt, uk 82,163 c 5 bar, br, fr, de, ht, nds, lb, mg, vo 51,653 c 6 zh-hans, zh-hant, ja, ko 25,528 c 7 cs, da, nl, et, is, nb, nn, sk, sv, fy 57,681 c 8 gu, hi, kn, ml, mr, ne, new, ta, te 61,683 ## experiments in order to demonstrate the effectiveness of our approach across languages and downstream tasks, we evaluate our method on three distinct datasets: • tydi qa @xcite ): question answering in 10 languages. results in the principal goal of this work is to investigate the effect of improved vocabulary composition on multilingual models. we make a good-faith effort to control all other variables, e.g., hyperparameters, training/evaluation procedures. in particular, we keep the number of languages constant, since perlanguage model capacity is known to affect the performance of multilingual models as shown in @xcite . in addition, we keep the number of parameters constant foot_2 , including the vocabulary size, since the performance of transformerbased @xcite models is strongly correlated with number of parameters @xcite @xcite @xcite . with carefully controlled experiments, we are confident that any improvement in the results is solely attributable to vocabulary composition. in other words, our vocabularies improve models without increasing the model size, compute or data. we pretrain using the masked language modeling (mlm) task on the raw text without applying any language-specific pre-processing. 4 at a high level, the experimental results demonstrate that our cluster approach to multilingual vocabulary generation improves over the standard joint recipe, increasing the macro average performance across languages for all datasets and tasks. for tydi qa, we see improvements on all languages. for xnli, cluster perform better or equally well in all languages except french. the results on individual languages are more mixed for ner, with cluster providing large gains on lowresource or rare-script languages, and small losses on high-resource languages. ## conclusion