Specialized Monolingual BPE Tokenizers for U ralic Languages Representation in Large Language Models (2024)

Abstract: Large language models show significant inequality in language representation, particularly for Uralic languages. Our analysis found that existing tokenizers allocate minimal tokens to Uralic languages, highlighting this imbalance. To address this, we developed a pipeline to create clean monolingual datasets from Wikipedia articles for four Uralic languages. We trained Byte Pair Encoding (BPE) tokenizers with a vocabulary size of 256,000 tokens, though Northern Sami had only 93,187 due to limited data. Our findings revealed most tokens are unique to each language, with 8,102 shared across all four, and 25,876 shared among Estonian, Finnish, and Hungarian. Using the Compression Ratio metric, our tokenizers outperformed popular ones like LLaMA-2 and Gemma 2, reducing Finnish’s compression ratio from 3.41 to 1.18. These results demonstrate the importance of specialized tokenizers for underrepresented languages, improving model performance and lowering costs. By sharing our tokenizers and datasets, we provide crucial resources for further research, emphasizing the need for equitable language representation.

## introduction large language models have rapidly integrated into our daily lives, yet they exhibit significant inequality in language representation due to issues of digital vitality. this imbalance is particularly pronounced in the uralic language family, which includes well-represented languages like finnish, estonian, and hungarian, as well as underrepresented ones like northern sami. our initial assessment revealed that existing tokenizers inadequately represent uralic languages, with a minimal number of tokens allocated to them. however, there is significant inequality in language representation at all stages due to the issue of digital vitality @xcite , where there is an insufficient amount of digitized text available @xcite . as a result, training datasets are dominated by texts in popular languages, leading to an imbalance @xcite ). it's not just the training datasets; the same problem exists with tokenizers @xcite , as was shown earlier. the efficiency of a tokenizer directly impacts performance in that language and, consequently, the cost of tasks, since pricing is tightly linked to the number of tokens. on the other hand, new methods are emerging that allow adjustments to the tokenizer's vocabulary during fine-tuning, and even better, future models could pay closer attention to token representation in tokenizers during training @xcite . however, to achieve this, a resource is needed that enables such adjustments @xcite . the approach we propose for creating monolingual tokenizers serves as exactly this kind of resource. uralic languages provide a useful model, as the group is relatively small and includes three languages that are comparatively well represented on wikipedia, one that is poorly represented, and a significant number of languages that are not represented at all. first, we decided to assess how many uralic language tokens are present in existing tokenizers. after evaluating the token representation, we realized that these languages are almost entirely absent. this led us to the question: can we train a bpe tokenizer both for individual uralic languages and for all uralic languages combined? we encountered the challenge that detecting uralic languages, beyond finnish, estonian, and hungarian, is a complex task. for the tokenizer, we used wikipedia articles from four languages: in addition to the three mentioned, we also included northern sami. we developed a pipeline to extract the cleanest monolingual texts for these four languages, and based on these texts, we trained four monolingual tokenizers with a size of 256k, as well as one tokenizer for the combined languages. although the initial goal was to create monolingual tokenizers, the tokenizers themselves turned out to be an interesting new tool for comparative analysis. they provide an additional perspective for approaching many classic tasks @xcite . we have made both the tokenizers and the clean monolingual wikipedia datasets publicly available. these datasets can also serve as a foundation for other research projects. finally, we evaluated how much more effectively our proposed tokenizers perform compared to existing ones. we also assessed the impact of tokenizer size on tokenization efficiency and proposed methods for determining how many additional tokens need to be added to a tokenizer for it to effectively handle uralic languages. ## conclusion in this work, we addressed the inequality in language representation within large language models, focusing on the uralic language family. using highquality monolingual datasets from wikipedia for estonian, finnish, hungarian, and northern sami, we trained specialized bpe tokenizers. our analysis showed that existing tokenizers poorly represent uralic languages, leading to inefficient tokenization and increased computational costs. our monolingual tokenizers outperformed widely used opensource tokenizers with lower compression ratios, improving both performance and cost-efficiency for underrepresented languages. additionally, these tokenizers offer valuable tools for comparative linguistic analysis, highlighting shared and unique features of uralic languages. by sharing our tokenizers and datasets, we provide key resources for further research in natural language processing for low-resource languages. this work underscores the need for equitable tokenization, especially for languages with low digital vitality.