Wine is not v i n. On the Compatibility of Tokenizations across Languages (2021)

Abstract: The size of the vocabulary is a central design choice in large pretrained language models, with respect to both performance and memory requirements. Typically, subword tokenization algorithms such as byte pair encoding and WordPiece are used. In this work, we investigate the compatibility of tokenizations for multilingual static and contextualized embedding spaces and propose a measure that reflects the compatibility of tokenizations across languages. Our goal is to prevent incompatible tokenizations, e.g., “wine” (word-level) in English vs. “v i n” (character-level) in French, which make it hard to learn good multilingual semantic representations. We show that our compatibility measure allows the system designer to create vocabularies across languages that are compatible – a desideratum that so far has been neglected in multilingual models.

## introduction pretrained language models @xcite @xcite have become the de-facto standard in natural language processing (nlp). due to memory constraints and to overcome sparsity, subword tokenization models such as byte pair encoding @xcite or wordpiece tokenization @xcite are most commonly used. multilingual pretrained language models (plms) such as mbert @xcite are mostly trained in an unsupervised fashion without any cross-lingual supervision. they rely on the fact that the underlying distributions across languages can be well aligned @xcite . however, there is little work that investigates the compatibility of tokenizations across languages. we argue that this is an important research question for several reasons. i) recent work suggests that the tokenization in multilingual models influences * equal contribution. the figure confirms our intuitions about vocabulary size compatibility: for english/russian, similar vocabulary sizes are compatible, but for chinese, we only find compatibility for larger english vocabulary sizes. this is because a chinese logogram by itself is usually a meaning bearing unit, so that most subword (or sub-root) elements in english have no analog in chinese. the multilinguality of these models @xcite . this corresponds to a strong intuition: in case of heavily diverging tokenizations, e.g., word tokenization in english and character tokenization in french it will be hard to train common cross-lingual representations. ii) given a limited vocabulary budget, e.g., from memory constraints, a researcher is naturally interested in how to distribute the budget across multiple languages. iii) currently, a balance of the tokenization is heuristically achieved through up-and downsampling (e.g., @xcite . with a better tokenization strategy, one might be able to achieve performance improvements. research in multilinguality has so far focused, among other directions, on analyzing and improving cross-lingual transfer @xcite @xcite and modeling tasks @xcite . earlier work on tokenization focused on the development of algorithms @xcite while analysis of the effect of tokenization has been undertaken both in the general @xcite and in the multilingual domain @xcite . we propose to investigate the compatibility of tokenizations across languages systematically. to this end, we train both static and contextualized embeddings using different vocabulary sizes and wordpiece tokenization @xcite . subsequently we investigate the similarity of the embedding spaces and their degree of multilinguality. figure 1 shows an example of our research question: here we trained static embedding spaces and investigated the similarity of the embedding spaces by computing the singular value gap (svg) measure. the plots show that different vocabulary sizes yield very different similarities. in addition, language pairs behave quite differently. see the caption for more information. in summary, our contributions are: 1 in a position paper, artetxe et al. (2020b) advocated for more rigor in cross-lingual research. the topic of tokenization was raised, especially the fact that current word and subword tokenization algorithms do not adequately capture morphological nuances and do not do justice to the great variety of languages in the world like those that use logographic scripts. these considerations motivate us to propose a method that makes better use of vocabulary space, a facet of multilingual models that is often overlooked. 1 code available at https://github.com/ antmarakis/vocab_size_compat 2.1 multilingual research investigation of the multilinguality of bert has gathered momentum in recent years. wu and dredze (2019); nooralahzadeh et al. (2020); artetxe et al. (2020a); conneau et al. (2020b) investigate cross-lingual transfer, with an extensive study of multilinguality at-scale presented in conneau et al. (2020a). k et al. ( 2020 ) research how linguistic variation in language properties affects multilingual bert models, with dufter and schütze (2020) further investigating the effect of model architecture and linguistic properties on bert's multilinguality. @xcite focus on variations of modeling tasks and their effect on downstream tasks such as xnli @xcite and machine translation. a novel set of crosslingual pre-training tasks was introduced in @xcite . @xcite show that performance in bilingual experiments is not optimal when english is used as the "hub", but it depends on the language pair. work on the representation space of bert has also been abundant. @xcite investigate the representation space of mbert and its properties and @xcite initially researched the degree of multilinguality of mbert. in our work, we further investigate the zeroshot capabilities of bilingual models and present a method for comparing multilingual embeddings across tokenizations. ## conclusion we investigated tokenization compatibility across languages, both for static and contextualized embeddings. to this end, we proposed compression rates and a method to select meaningful vocabulary sizes in an automated manner for any language. we introduced a compatibility measure and showed that it correlates with downstream performance. we gave evidence for two key findings that hold for both static and contextualized embeddings. i) tokenization compatibility can have a significant impact on multilingual performance: performance is generally higher when vocabulary sizes are compatible. ii) tokenization compatibility varies significantly across language pairs: pairs of alphabetic languages show a stronger performance on the diagonal (i.e., for comparable vocabulary sizes) while an alphabetic language like english requires large vocabulary sizes when paired with a logographic language like chinese. our study has clear limitations: we only experimented with the wordpiece tokenizer and with a small number of language pairs and there are only a few larger-scale experiments on xnli. we will continue research in this direction and hope that this paper will spark interest by other researchers in the compatibility of tokenizations across languages and its effect on multilinguality.