False F riends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models (2025)

Abstract: Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages? Prior work offers mixed evidence, partly due to varied setups and confounders, such as token frequency or subword segmentation granularity. To address this question, we devise a controlled experiment where we train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings. Crucially, we explore a new dimension to understanding how overlap affects transfer: the semantic similarity of tokens shared across languages. We first analyze our models’ hidden representations and find that overlap *of any kind* creates embedding spaces that capture cross-lingual semantic relationships, while this effect is much weaker in models with disjoint vocabularies. On XNLI and XQuAD, we find that models with overlap outperform models with disjoint vocabularies, and that transfer performance generally improves as overlap increases. Overall, our findings highlight the advantages of token overlap in multilingual models and show that substantial shared vocabulary remains a beneficial design choice for multilingual tokenizers.

## introduction multilingual tokenizers are commonly trained on the concatenation of corpora from multiple languages @xcite , resulting in subword vocabularies with naturally overlapping tokens across languages. while some of these shared tokens may correspond to semantically aligned units across languages (e.g., cognates, named entities), others may arise from coincidental overlaps or have different meanings (e.g., false we answer this question by training bilingual autoregressive models on data from six language pairs, each under four controlled vocabulary overlap settings (figure 1 ). in contrast to prior work, we distinguish different types of overlap based on semantic similarity of the tokens in the two languages, as semantic alignment has been shown to impact cross-lingual transfer @xcite @xcite , while holding subword segmentation granularity and token frequency distributions fixed. within pre-trained models, we find that token overlap enables the embedding spaces of the two languages to capture cross-lingual semantic relationships-an effect that is substantially weaker in models with disjoint vocabularies. when testing zero-shot transfer between languages on the xnli and xquad downstream tasks, models with any amount of overlap consistently outperform models with no overlap, and transfer performance generally improves as overlap increases. we find that tokens with shared meanings across languages contribute most to transfer performance, though any overlap is beneficial. our findings offer practical guidance on the design of future multilingual tokenizers. more recently, @xcite found that while overlap can benefit sentence-level tasks and ner, it may degrade performance on syntactic tasks. similarly, @xcite show that multilingual corpora contain unexpectedly high levels of overlap, largely due to code-switching and shared vocabularies, which may help explain cross-lingual transfer in dense retrieval models. @xcite extend overlap by merging subwords with different forms but similar meanings into "semantic tokens," preserving downstream performance with smaller vocabularies. @xcite show that similarity-or alignment-weighted overlap correlates with cross-lingual transfer across different scripts. other related work shows that multilingual tokenizers often over-segment lowresource languages, artificially inflating subword overlap @xcite @xcite . this over-segmentation reduces efficiency and degrades representation quality. taken together, these studies paint an unclear picture: while vocabulary overlap can create crosslingual anchors that facilitate transfer, it may introduce interference across languages that hinders modeling. moreover, the conditions under which overlap is beneficial remain insufficiently explored. unlike prior work, we focus on how the semantic similarity of shared tokens affects performance, while carefully controlling for confounders like subword segmentation granularity and token frequency distributions. tokenizer design. vocabulary overlap has likewise been a central consideration in tokenizer design. chung et al. (2020) and liang et al. (2023) use clustering methods to de-emphasize token overlap between lexically distinct languages, citing k et al. (2020) for the thesis that overlap is not the principal factor in multilingual model effectiveness. in contrast, @xcite highlight the importance of overlap for transfer and propose a method to promote token overlap between high-and low-resource languages. at the extreme, byte-and character-level models (e.g., canine, @xcite @xcite @xcite eliminate subword tokenization altogether. this maximizes vocabulary overlap but comes at a cost to efficiency, presenting unique engineering challenges. ## approach: controlled overlap settings to systematically vary the vocabulary overlap between two languages according to our four experimental settings (see figure 1 ), we denote a base tokenizer t with vocabulary v of size n . we assume that @xmath0, 1, . . . , n -1}, i.e. that each token in v is represented by an integer index from 0 to n -1. two languages l 1 and l 2 have corpora c 1 and c 2 , respectively, which we tokenize using t . let v 1 = {unique tokens in c 1 } ⊆ v and v 2 = {unique tokens in c 2 } ⊆ v . in other words, v 1 and v 2 are the individual vocabularies of l 1 and l 2 , respectively. thus, when tokenizing c 1 and c 2 , the native overlap of t is the set @xmath1, and the effective vocabulary size of given a token sequence @xmath2, x 2 , . . . , x n ], where x @xmath3, from language ℓ ∈ {l 1 , l 2 }, we define a modified tokenizer t ′ that produces x ′ = [x ′ 1 , x ′ 2 , . . . , x ′ n ], where each here, o ′ ⊆ o denotes the subset of tokens we choose to share under a given setting: for l 1 , all tokens remain unchanged, and for l 2 , tokens in o ′ remain unchanged while all others are offset by n . this guarantees that l 1 and l 2 only share o ′ , and t ′ has a new effective vocabulary size the four choices of o ′ define our four settings, listed below. full overlap. o ′ = o. since this only renames certain tokens x i / ∈ o from l 2 , t ′ is behaviorally equivalent to t , and high-similarity overlap. o ′ = o hi , where o hi ⊆ o is the set of tokens whose meanings align closely between l 1 and l 2 . only these tokens remain shared, so 4 implementation details datasets. we use ccmatrix @xcite , a large collection of high-quality webmined parallel texts, for bilingual model pretraining. this allows us to control for the content and the approximate quantity of data in each language. we train on six language pairs: english-spanish, english-german, english-turkish, english-chinese, english-arabic, and english-swahili. english is included in every pair to reflect realistic training scenarios, as english is typically the dominant language in multilingual pretraining datasets. the second language is selected to cover various language families, scripts, and typological distances from english. the pre-training corpus for each pair is constructed by shuffling and interleaving sentences from both languages. tokenizer and overlap partitioning. our base tokenizer t is the multilingual xlm-r tokenizer @xcite , which uses sen-tencepiece @xcite with a unigram lm @xcite . we found that it offers more effective compression across languages than other tokenizers (see appendix a). to divide the native overlap o into high-and low-similarity subsets (o hi and o lo ), we rank tokens in o by their semantic similarity across languages. for each token @xmath4, we extract 100 occurrences from c 1 (the ccmatrix corpus for l 1 ), pass the sentences through xlm-r, and mean-pool the layer-l contextual embeddings of t to obtain a static embedding e 1 (following @xcite . the layer l is pre-determined by a sweep we conducted on sets of cognates and non-cognates, as detailed in appendix b. we repeat this for c 2 (the corpus for l 2 ) to compute e 2 . the cosine similarity between e 1 and e 2 serves as the token's cross-lingual similarity score. we rank tokens in o by these scores, assigning the top half to o hi and the bottom half to o lo . for detailed corpus statistics and overlap metrics for each setting, refer to appendix c. ## embedding similarity analysis as a first step in analyzing our pre-trained models, we test how sharing semantically similar or dissimilar tokens influences the model's learned representations. we take the 500 most and least similar overlapping tokens for each language pair, ranked using xlm-r as described in the previous section. from a middle layer (@xmath5, we extract contextual embeddings for each token to construct a single static embedding of the token for each language using the same method as before. we then ask whether models learn more similar representations for high-similarity tokens and more distinct ones for low-similarity tokens. crucially, whether these tokens are shared depends on the overlap setting: in the high-similarity overlap condition, the top 500 tokens are shared; in the english swahili similarity score token set high-sim. low-sim. random low-similarity overlap condition, the bottom 500 are shared. to control for the high baseline cosine similarities observed in transformer embeddings due to anisotropy @xcite , we additionally measure similarity scores for 500 randomly selected non-overlapping token pairs. results. figure 2 summarizes the results across all language pairs. with the exception of the lowsimilarity overlap setting for english-spanish and english-german, the difference between the highand low-similarity token sets was statistically significant for every language pair and overlap condition (unpaired t-tests, all bonferroni-corrected @xmath6.05). the effect size (cohen's d) varied with the overlap condition (see in contrast, the low-similarity overlap setting revealed a split in results based on language family. for the closely related language pairs english-spanish and english-german, no significant differences were observed (bonferroni-corrected @xmath9, d ≈ 0). however, for more typologically distant language pairs (english-turkish, english-chinese, english-arabic, english-swahili), the effect reversed: low-similarity tokens scored higher than high-similarity tokens, with large negative effect sizes (@xmath10.6, -1.0]). these reversals indicate that in the low-similarity overlap setting, tokens that do not share meaning become aligned in the embedding space simply because they are shared in the vocabulary, producing misleading or inverted similarity effects. this demonstrates that the type of overlap-whether it links semantically similar or dissimilar tokens-critically shapes how cross-lingual models align token representations. effects are especially pronounced for more distant language pairs, where there is less contextual signal available to counteract the bias introduced by shared but semantically unrelated tokens. 6 downstream task performance we further fine-tune and evaluate our models on two downstream tasks, namely, natural language inference (nli) and question answering (qa), in a standard zero-shot transfer setup. for nli, we train on english multinli @xcite and evaluate on xnli @xcite . for qa, we train on english squad @xcite and evaluate on xquad @xcite . fine-tuning hyperparameters and optimization details are provided in appendix e. ## conclusion in this paper, we present a detailed study of vocabulary overlap in multilingual language models. our experimental design isolates the effect of overlap by controlling for token frequency and subword segmentation quality. we also uniquely disentangle how semantically similar or dissimilar vocabulary overlap affect multilingual representations and task transfer. while prior work has raised concerns that highly polysemantic tokens from vocabulary sharing may hinder performance, we find that overlap (1) promotes alignment of the embedding spaces between languages in bilingual models and (2) enables cross-lingual transfer on downstream tasks. overlapping tokens with the same meaning across languages contribute most, though any overlap proves beneficial. we therefore argue that, rather than reducing overlap, tokenizer development should focus on other determinants of quality, such as per-language compression rates.