Clustering Monolingual Vocabularies to Improve Cross-Lingual Generalization (2021)

Abstract: Multilingual language models exhibit better performance for some languages than for others (Singh et al., 2019), and many languages do not seem to benefit from multilingual sharing at all, presumably as a result of poor multilingual segmentation (Pyysal o et al., 2020). This work explores the idea of learning multilingual language models based on clustering of monolingual segments. We show significant improvements over standard multilingual segmentation and training across nine languages on a question answering task, both in a small model regime and for a model of the size of BERT-base.

## introduction since its release in 2018, multilingual bert (mbert) has been extensively analyzed. for instance, mbert has been shown to facilitate zero-shot cross-lingual transfer, but transfer performance is significantly worse for typologically distant languages @xcite . this is because cross-lingual generalization is only partial, with some language-specific partitioning of the representational space @xcite @xcite . moreover, mbert relies on a single, multilingual tokenizer to create its vocabulary @xcite ), a subword tokenizer based on an algorithm similar to bpe @xcite . the multilingual tokenizer lets subwords be shared across different languages, but the approach penalizes outlier languages, whose words end up being segmented according to statistics derived from high-resource languages @xcite . in this study, we propose a novel method for building a multilingual vocabulary that is more fair to outlier languages: the core idea is to train mbert-like language models on multilingual clusters of monolingual tokens derived from monolingual tokenizations into vocabularies of equal size. we argue this approach brings us the best of two worlds: monolingual (dedicated) tokenization and multilingual input representations. contributions our main contribution is devising a novel way of learning multilingual vocabularies that are more fair to outlier languages. our approach relies on clusters, and we analyze these clusters in depth. next, we induce a multilingual mbert-like language model from cluster representations of input sentences and evaluate this language model on the downstream task of crosslingual question answering. we show that the model trained on our cluster representations improves over our baseline, trained with standard mbert tokenization, across nine typologically diverse languages. we observe the same improvements across two different language model sizes. @xcite showed how training monolingual bert models instead of a single multilingual one can improve performance for some languages even if the amount of training data decreases. a possible cause of this phenomenon could be identified in the better quality of tokenization that can be reached when a dedicated monolingual vocabulary is used. this thesis is also supported by the findings of @xcite on the importance of vocabulary size to a multilingual language model's performance. ## multilingual vocabulary induction the key idea of our method is to build a single multilingual model from the output of dedicated, monolingual tokenizers, i.e., language-specific subword vocabularies, by clustering the obtained subwords by semantic similarity. a single model is then trained on the desired number of clusters, learning an embedding for each cluster. this allows for implicitly using a very large vocabulary without increasing model complexity, hopefully leading to better segmentation for outlier languages and better generalization across languages. our approach involves three steps: these steps are described in turn below. an overview of the entire process is given in figure 2 . ## model training once the clustering was completed, a bert vocabulary was built extending the set of cluster ids (cids) with bert special tokens, to give a vocabulary size of 30397, similar to the one of the original bert model. we called our model icebert, an acronym for interlingual-clusters enhanced bert. ## results as shown in in the paper we have compared clustering of monolingual vocabularies to multilingual segmentation, i.e., bpe on concatenated corpora. other alternatives have been proposed in the literature, including multilevel tokenization @xcite and tokenization-free, character-based modeling @xcite . it is beyond the scope of this paper to systematically compare all these different approaches. the idea of using sequences of token group ids to represent input sentences was previously explored in @xcite , who used finite state automata instead of clusters. ## conclusion this study aimed to improve multilingual language models by training them on clusters of monolingual segments. the proposed approach yielded good quality clusters able to group semantically similar words and subwords across languages. our clustering strategy led to improvements over standard segmentation and training methods on the majority of the included languages, when evaluated on a question answering task.