When Every Token Counts: Optimal Segmentation for Low-Resource Language Models (2025)

Abstract: Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model performance. While subword tokenizers like Byte-Pair Encoding (BPE) are widely used, questions remain about their optimality across model scales and languages. In this work, we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation, yielding improvements in token-saving percentages and performance benefits, particularly for smaller models. We evaluate tokenization performance across various intrinsic and extrinsic tasks, including generation and classification. Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource (LR) language applications, highlighting a promising direction for further research and inclusive NLP.

## introduction the development of large language models (llms) has significantly advanced natural language processing. these models @xcite @xcite have demonstrated unprecedented capabilities in tasks ranging from text generation and translation to complex problem-solving and creative writing. however, despite these advancements, challenges remain in effectively processing low-resource (lr) languages and optimizing models of varying scales. a critical aspect influencing model performance is tokenization -the process of converting text into tokens that the model can understand. tokenization methods are pivotal in large language models, with popular techniques including word-piece @xcite , sentence-piece @xcite , and unigram-lm @xcite . wordpiece, used in models like bert @xcite , tokenizes words into subword units based on their frequency in the training data, improving the model's handling of rare or out-of-vocabulary words. sentencepiece and unigram-lm, commonly used in models like gpt, employ a character or byte-based approach that doesn't rely on predefined word boundaries, making them versatile across languages. lr languages face two significant challenges in natural language processing: a lack of high-quality and diverse datasets and novel methods to represent this data. @xcite . without ample data, models struggle to learn the complex linguistic patterns necessary for tasks such as machine translation, sentiment analysis, and summarization. secondly, compression challenges in tokenization exacerbate the difficulties faced by lr languages. common tokenization techniques, such as bpe, often fragment words into smaller, frequently occurring subwords. the bloating of tokens leads to higher computational and memory costs, as models must process longer sequences @xcite . inefficient tokenization also results in less accurate representations, leading to fragmented or improperly segmented tokens, which negatively impacts model performance in tasks requiring precise language understanding @xcite . we refer to the strategy adopted by bpe as the greedy segmentation algorithm. the widely used gpt-2 tokenizer @xcite handles any input without unknown tokens, yet it compromises tokenization efficiency, especially for non-english text and special characters. this english-centric model often splits languages like turkish, indonesian, or malay into byte sequences, unnecessarily lengthening token sequences and reducing the effective context window for non-english content. while the gpt-4 (openai et al., 2024) tokenizer cl100k_base improves with a larger vocabulary and more diverse training data, it still shows biases in token distribution. for agglutinative languages (e.g., turkish, finnish) or languages with complex word structures, tokenization may create excessive token splits, impacting both efficiency and model performance. examples of the inefficient segmentation of cl100k_base is shown in motivated by the need to enhance tokenization strategies for lr languages and models of varying scales, we present an optimal bpe segmentation algorithm that reduces token counts, especially in morphologically complex and low-resource languages, achieving more efficient and meaningful segmentation. we demonstrate the algorithm's token-saving capacity across diverse languages, reducing token counts by 3-5% compared to greedy segmentation. this improvement is particularly impactful for rare and complex words, with compression rates increasing by up to 20%. our comparative study reveals that models using optimal segmentation see up to a 10% increase in accuracy on downstream tasks, including text classification and generation. ## related work recent research has focused on the effects that compression has on tokenization, which are particularly relevant for optimizing language models in resource-constrained environments. a study by @xcite shows the correlation that compression has on downstream tasks such as classification and generation. in contrast, @xcite 's exploration of greedy algorithms and @xcite 's introduction of path-piece have provided new insights into optimizing tokenization for both performance and efficiency, without looking into compression. note that while @xcite and @xcite demonstrate the effectiveness of their tokenizer, they show results on english tasks but do not show the impact on linguistic diversity. the paper by @xcite demonstrates this to some extent; however, their experiments focus primarily on english. @xcite provide a task-oriented perspective on the challenges that llms encounter with low-resource languages, highlighting the need for tailored approaches in multilingual contexts. the quality of tokenization has been a subject of intense study, with comparative analyses by @xcite , @xcite , and @xcite providing valuable insights into the relative performance of different tokenization methods across various languages and tasks. in multilingual settings, subword tokenizers lead to disproportionate fragmentation rates for different languages and writing script @xcite . similarly, monolingual optimized tokenizers may not be as efficient for multilingual settings @xcite . @xcite introduces a new concept known as parity or premiums in tokenizers which has shed light on the importance of balanced tokenization across different languages in multilingual models. the disparities are particularly pronounced in african and indian languages, as noted by @xcite and @xcite , respectively. while @xcite and (velayuthan and sarveswaran, 2024) demonstrate the critical role of tokenization in addressing challenges related to compression and parity in tokenization, they do not show the performance of llms on extrinsic tasks -especially for lr languages. these studies highlight the need for more inclusive tokenization and pre-training strategies that can serve diverse linguistic communities. our work shows that by improving tokenization methods -specifically compression -we can achieve performance on extrinsic tasks on lr languages. our approach allows us to optimize inference time and cost and have an equally good as the original tokenization. @xcite also highlights the economic implications of these disparities, comparing the pricing of language model usage across different languages and revealing systemic biases in current nlp technologies. ## background we first provide a brief description of the steps involved in tokenization that is pre-tokenization, vocabulary construction, and segmentation. we then describe the token saving ratio (tsr) metric used to compare results throughout our paper. ## optimal segmentation in this section, we define the problem of optimal segmentation mathematically and follow it up with a discussion of our algorithm presented in algorithm 1. ## experimental setup for our work, we extended on openai's foot_0 family of tokenizers which are available in three distinct vocabulary sizes: 50k, 100k, and 200k tokens, as detailed in ## results in this section, we present the results of intrinsic evaluation on the cc-100 dataset. we first highlight qualitative examples to showcase the inefficiency of bpe with greedy segmentation compared to bpe with optimal segmentation we also showcase an interesting observation that word length has on the tsr. finally, to validate our optimal segmentation algorithm, we conduct extensive extrinsic evaluations across multiple downstream tasks. first, we report improvement upon greedy bpe's performance across language boundaries for non-english tasks. at the same time, we report an increase in improvements for the t sr * split of the dataset, thus highlighting the need for token saving in downstream performance. at the end, we report perplexity scores on english datasets to state that the improvement provided by our optimal segmentation doesn't reduce the tokenizer's performance in english. ## conclusion in the scope of this work, we identified the inefficient greedy segmentation method used in the bpe tokenizer and proposed an optimal segmentation algorithm that results in efficient token utilization, particularly for lr languages. we established the optimality of our algorithm by showing its impact in both intrinsic and extrinsic experiments as done in the literature. by studying multiple languages, we observed a strong correlation between improvements in token saving ratios and linguistically better segments, with this effect being especially pronounced for morphologically complex words and propagating to performance improvement in downstream tasks. these findings underscore the need for research in tokenization approaches that can boost model effectiveness, especially for language models serving low-resource languages.