Studying the Effect of H indi Tokenizer Performance on Downstream Tasks (2025)

Abstract: This paper deals with a study on the effect of training data size and tokenizer performance for Hindi language on the eventual downstream model performance and comprehension. Multiple monolingual Hindi tokenizers are trained for large language models such as BERT and intrinsic and extrinsic evaluations are performed on multiple Hindi datasets. The objective of this study is to understand the precise effects of tokenizer performance on downstream task performance to gain insight on how to develop better models for low-resource languages.

## introduction large language models (llms) have shown extraordinary performance in a range of natural language processing (nlp) tasks, including both text classification and text generation. they are made use of across the world. after the success of many monolingual llms such as bert @xcite and gpt, multilingual llms were built over these foundational models, increasing the number of languages they were pre-trained on, using different architectures and expanding the number of parameters. some multilingual language models such as mbert, mbart @xcite , llama @xcite , the more recent gpt versions, bloom @xcite have been trained on more than hundred languages. however, there is a skewed distribution in the quantity of the different languages they have been trained, causing bias in their predictions, in terms of languages as well as cultures. for indic languages in specific, many llms have been built by using the aforementioned models trained on large corpora of multiple indian languages. these include indicbert @xcite , indicbart @xcite , muril @xcite , openhathi (sar). while india has hundreds of languages, most of them are however very low-resource, making training on them very hard. the process of pre-training these llms involve processing large amounts of text data and make them perform tasks like masked language modeling (mlm) and next sentence prediction (nsp) to learn semantic embeddings of the sentences that are input. while a lot of work has been done for the mechanisms and architecture of the models, a relatively under-investigated aspect of tokenizers is the impact of the tokenizer performance on the performance of the model. tokenizer performance plays a crucial role in the performance of llms. the quality of tokenization can have a huge effect on the contextual understanding and linguistic ability of the model. this project aims to investigate the effect of tokenizer performance on llms for indian languages. since there are hundreds of indian languages, for this project, only hindi was chosen as a representative language to conduct experimentation and evaluate results. initial rudimentary investigations showed that there exist issues even with existing indian llms. hindi represents its vowel sounds within a word using 'matras' for the vowel letters, as do many other indian languages. however, these are encoded as accent marks in digital representations. it was seen that when indicbert @xcite was used to tokenize hindi text, it removed these accents as part of its pre-tokenization. the removal of these 'matras' would remove a lot of the semantic sense behind the words, equivalent to removing all the vowel characters from english words. furthermore, it can also be seen in figure 1 that the model splits the words very frequently into small-length tokens, to an average of around 2-4 characters, essentially removing the meaning behind the words, to simply represent repeating character groups. such tokenization would deprive the model of any semantic understanding of the words, and would not allow it to gain meaningful context of the given text in order to perform its desired task. this project thus extensively investigates the effect of the training data and tokenizer performance on subsequent downstream task performance by developing several monolingual tokenizers and models in hindi, making use of different training data sizes as well as tokenization algorithms. the remainder of the paper is structured as follows. section 2 discusses related studies of tokenizer parameters and their effect on low-resource languages, section 3 outlines the methodology of the research in terms of tokenizers used, models trained and intrinsic and extrinsic evaluations performed along with the data used. finally, section 4 presents the results of the experiments. section 5 highlights the final findings and inferences and section 6 discusses limitations of the current research with future scope. ## related work tokenizers are a relatively unexplored aspect in the training of llms. @xcite investigated the intrinsic and extrinsic performance of tokenizers in monolingual and multilingual settings for 5 european languages. 24 tokenizers were trained with corresponding transformer-based decoder models making use of them, which were fine-tuned on a range of downstream tasks. there experiments showed that there was some correlation between certain metrics and downstream task performance, however, a more fine-grained analysis was required. kaya and tantug (2024) also investigate tokenizers for turkish, a morphologically rich and relatively less-studied language. they studied the finegrained effect of tokenization granularity based on the training data, vocabulary size and algorithm. they displayed that these factors play a role in tokenization quality as well as downstream task performance, especially for morphologically complex words so the model can attain contextually meaningful tokens. @xcite investigated the effect of the tokenization algorithm on low-resource african languages for neural machine translation. being agglutinative languages, they showed the improvement in performance when sentencepiece bpe was used instead of bpe tokenization, since it encodes the whitespace character and does not require words to be space separated. ## results the silhouette scores of the sentence embeddings (before being processed through the classification head) also show similar trends, being the lowest for bpe, and comparable for the other 3 algorithms. this indicates the model's inherent understanding of the language based on how well it can represent the sentences. ## conclusion in this paper, fine grained analysis of the impact of tokenizer performance on downstream performance of bert models in hindi was conducted. the results showed that there is a significant correlation between intrinsic tokenizer performance and extrinsic downstream task performance. the unigram, wordpiece and sentencepiece models that showed the best tokenizer performance also showed the best results in sentiment analysis as well as in named-entity recognition tasks. this suggests that the quality of words in the models' vocabulary allows it to segment words in the input text more meaningfully, thereby allowing it to learn better semantics during the pre-training phase and subsequently when being fine-tuned for the downstream tasks.