Beyond Literal Token Overlap: Token Alignability for Multilinguality (2025)

Abstract: Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. However, these very literal metrics assign large distances to language pairs with different scripts, which can nevertheless show good cross-linguality. This limits the explanatory strength of token overlap for knowledge transfer between language pairs that use distinct scripts or follow different orthographic conventions. In this paper, we propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation. In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. We analyse this metric in the context of both encoder and decoder models, look at data size as a potential distractor, and discuss how this insight may be applied to multilingual tokenisation in future work. We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer, as well as to guide the construction of better multilingual tokenisers in the future. We publish our code and reproducibility details.

## introduction highly multilingual language models have received plenty of research attention in recent years. crosslingual alignment of representations, that is, the similar representation of similar meanings regardless of input language @xcite , as well as good downstream cross-lingual transfer ability (cf. @xcite @xcite , a measure of token alignability, predicts downstream transfer performance better than the previous metric of distributional token overlap @xcite . the difference is especially stark for language pairs with different scripts (•), compared to language pairs with the same script (×). the orange line shows the linear fit across all included pairs. 2024, etc.), have been considered desirable properties for such models. representation alignment is typically seen as a key contributing factor to transfer ability, which in turn enables efficient handling of numerous task-language combinations. a number of papers have asked when and why information is shared across language boundaries in multilingual models and enables cross-lingual transfer @xcite @xcite @xcite . token overlap, i.e., the occurrence of identical tokens in the corpora of multiple languages, has been shown to affect the cross-lingual capabilities of models @xcite . another approach is to compare the distributions of token literals in parallel corpora @xcite . still, both metrics have a crucial limitation: they cannot explain why related languages with different scripts are well-aligned by the models (see § 2.1). here, we propose another angle: token alignability. this concept captures the intuition that models may rely on statistical correspondences between subword tokens ('token alignment') that are more nuanced than literal string matching. from token alignments produced by a statistical word aligner, we derive two kinds of token alignability scores for any language pair in a multilingual tokeniser: one directional, one symmetrised ( § 3.2). we compute correlations of these scores both to downstream transfer performance on classification and sequence labelling tasks (cf. § 3.3), and to measures of cross-lingual alignment in the model representations (cf. § 3.4). our primary object of study is a set of small encoder models trained with several different multilingual tokenisers (bpe, unigram, and 'tokmix'). furthermore, we also consider recent larger, pre-trained decoder models. in addition to showing that token alignability is a better predictor of downstream cross-lingual transfer than distributional overlap ( § 4.1), we consider the impact of pre-training data size ( § 4.2), and show the correlation of token alignability with representation alignment inside the model (also § 4.1). finally, we discuss how this insight may be applied to future multilingual tokenisers ( § 5). ## related work subword tokenisation is currently the standard input processing approach of language models, with bpe @xcite and unigramlm (kudo, 2018) being the most common algorithms for deriving these tokens. however, there has been increased interest in recent years in addressing limitations of the subword token paradigm (e.g., @xcite @xcite or even moving beyond it (e.g., @xcite . ## methodology our central analysis relies on rank correlations, showing which tokeniser metrics ( § 3.1, § 3.2) are more predictive of downstream cross-lingual transfer ( § 3.3) and cross-lingual alignment of representations ( § 3.4). we ensure that within each task, the metrics are always compared over the same set of language pairs. ## future work we showed here that good tokeniser alignability correlates well with crosslinguality, an important factor for the performance of multilingual language models. hence, the eflomal score may be applied to improve vocabulary learning for fairer multilingual tokenisers (see also @xcite . however, a naive implementation, where alignability score is checked at every decision point (merges for bpe, or pruning tokens for unigram), is far too intensive. therefore, future work in this area will require finding suitable approximations, like calculating alignability score difference for some fraction (e.g., on the order of 10%) of all candidate tokens at a time. ## conclusion we have proposed a new metric for describing the quality of a multilingual tokenisation, with implications for cross-lingual alignment in multilingual pre-trained models: token alignability. this metric is particularly relevant for language pairs with different scripts and thus no literal token overlap. we showed correlations with transfer performance on downstream classification tasks, as well as with measures of cross-lingual alignment. these findings show the potential of our token alignability metric to guide the development of robust multilin-gual tokenisers and to identify suitable language pairs for cross-lingual transfer.