Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries (2025)

Abstract: Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages.Existing approaches that utilize monolingual or parallel corpora face challenges when applied to languages with limited resources.In this work, we propose a simple yet effective vocabulary transfer method that utilizes bilingual dictionaries, which are available for many languages, thanks to descriptive linguists.Our proposed method leverages a property of BPE tokenizers where removing a subword from the vocabulary causes a fallback to shorter subwords.The embeddings of target subwords are estimated iteratively by progressively removing them from the tokenizer.The experimental results show that our approach outperforms existing methods for low-resource languages, demonstrating the effectiveness of a dictionary-based approach for cross-lingual vocabulary transfer.

## introduction vocabulary transfer, or adaptation, is a method to bridge the gap between languages in language models @xcite to improve their performance for new languages. vocabulary transfer can also address token overfragmentation @xcite @xcite , where a sentence is split into spuriously many tokens, causing slower inference and more significant cost. previous works rely on monolingual or parallel corpora to align source and target languages or focus on vocabulary overlaps between them @xcite @xcite @xcite @xcite . however, these methods face a significant challenge when dealing with low-resource random ours dictionary taci mbi : learn taci kū : school languages, where parallel corpora are often unavailable @xcite , or when there is little lexical overlap between source and target languages that use different scripts. dictionaries offer a potential solution to this issue. when reading in an unfamiliar language, bilingual dictionaries can aid in comprehension by connecting words in the unknown language to their equivalents in a known language, and they can also enhance the performance of language models @xcite . such dictionaries are often available, even when other linguistic resources are scarce, thanks to descriptive linguists, who produce them as part of their work in documenting languages @xcite . in this light, we propose an embedding initialization method for cross-lingual vocabulary transfer based on bilingual dictionaries as illustrated in figure 1 . our proposed method utilizes dictionaries to map as many target subwords as possible to source subwords for embedding initialization. this method is inspired by trans-tokenizer @xcite that maps target subwords to source subwords through word alignment using parallel corpora. our proposed method differs from trans-tokenizer in that we utilize dictionaries, which are more available for low-resource languages and have helpful features for subword mapping, and we estimate the target embeddings iteratively. to perform iterative mapping, we utilize the bpe tokenizer algorithm @xcite , but progressively remove subwords, forcing the tokenizer to decompose words into shorter and shorter subwords. we estimate the embeddings for as many target subwords as possible by removing subwords mapped to source subwords from the vocabulary and reapplying tokenization. the experimental results demonstrate that our dictionarybased approach outperforms existing methods and multilingual language models in terms of f1 score on a downstream task and perplexity, across lowresource languages with different scripts. our finding is that our dictionary-based method can improve the performance in low-resource languages that are genealogically distant from english and show typologically isolating or agglutinative characteristics (e.g., uyghur, khmer, and manchu). ## background and related work cross-lingual vocabulary transfer consists of subword embedding initialization and languageadaptive pre-training. ## proposed method in this study, we propose a simple approach for lowresource languages utilizing bilingual dictionaries, which comprise of words (entries) in a target language and their definitions in a source language. in this approach, we suppose that source models are trained with high-resource languages and have tokenizers with pre-trained embeddings. the proposed method consists of three parts: training a tokenizer for the target language, aligning subwords between languages, and initializing target language subword embeddings. ## experimental setups languages. we use english as a source language and german, japanese, old english, uyghur, sanskrit, khmer, and manchu as target languages. we treat german and japanese as high-resource languages and other languages as low-resource languages. as source models, we use roberta (base) @xcite , xlm-r (base) @xcite , llama 3.1 8b @xcite , and gemma 2 9b @xcite . the detailed information about the models is provided in appendix a evaluation metrics. we evaluate the performance of our initialization method using f1 for the ner performance of a masked language model (roberta) and perplexity for causal language models (llama 3.1 and gemma 2). the perplexity is normalized by word length, not subword length, for fair comparison across tokenizers. this is be- ## results and discussion ## conclusion in this study, we propose a dictionary-based crosslingual vocabulary transfer approach. this method utilizes bilingual dictionaries, which are available even for low-resource languages. it iteratively maps a subword in a target language to subwords in a source language and initializes the target subword embeddings. the experimental results demonstrate that our approach effectively initializes the subword embeddings for low-resource languages, indicating it is a promising approach for building language models for those languages.