Can Perplexity Predict Finetuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for N epali (2025)

Abstract: The impact of subword tokenization on language model performance is well-documented for perplexity, with finer granularity consistently reducing this intrinsic metric. However, research on how different tokenization schemes affect a model’s understanding capabilities remains limited, particularly for non-Latin script languages. Addressing this gap, we conducted a comprehensive evaluation of six distinct tokenization strategies by pretraining transformer-based language models for Nepali and evaluating their performance across multiple downstream tasks. While recent prominent models like GPT, RoBERTa, Claude, LLaMA, Mistral, Falcon, and MPT have adopted byte-level BPE tokenization, our findings demonstrate that for Nepali, SentencePiece tokenization consistently yields superior results on understanding-based tasks. Unlike previous studies that primarily focused on BERT-based architectures, our research specifically examines sequential transformer models, providing valuable insights for language model development in low-resource languages and highlighting the importance of tokenization strategy beyond perplexity reduction.

## introduction nepali, an indo-aryan language written in devanagari script, serves as the official language of nepal. according to the nepal population and housing census 2021, approximately 13 million people (44.9%) speak nepali as their mother tongue, while an additional 13.5 million (46.2%) use it as their second language. the language extends beyond nepal's borders into neighboring regions of india, bhutan, brunei, and myanmar. nepali follows a subject-object-verb sentence structure, distinguishing it from many indo-european languages. despite its significant speaker population, computational research in nepali natural language processing remains underdeveloped due to limited high-quality datasets and computational resources. nepali's rich morphological complexity and extensive vocabulary pose unique challenges for creating accurate and concise content. investigating the applicability of state-of-the-art nlp technologies to nepali not only benefits researchers and speakers but also has potential implications for other devanagari-script languages such as hindi, sanskrit, maithili, and bhojpuri. tokenization-the process of segmenting text into smaller units such as words or subwordsforms the foundation of natural language processing pipelines. this critical preprocessing step enables computational systems to analyze and process human language by converting raw text into discrete units that algorithms can efficiently manipulate. the choice of tokenization strategy significantly impacts a model's ability to handle vocabulary coverage, out-of-vocabulary words, and morphological complexity. recent advances in subword tokenization have revolutionized nlp by balancing vocabulary size constraints with linguistic flexibility, particularly for morphologically rich languages like nepali. contemporary language models generate human-like text by leveraging transformer architectures trained on massive text corpora. these models primarily follow two paradigms: masked language modeling (mlm), exemplified by bert @xcite , where models learn bidirectional context by predicting masked tokens; and autoregressive language modeling, implemented in models like gpt @xcite and palm @xcite , where models predict the next token based on preceding context. while masked language models excel at learning powerful bidirectional representations suitable for downstream tasks, autoregressive models offer superior capabilities for text generation. unlike previous studies that predominantly focused on bert-based architec-tures for nepali, our work specifically examines sequential (autoregressive) transformer models similar to @xcite , trained with various tokenization strategies and evaluated on multiple downstream tasks. the major contributions of our paper are as follows: ## related works language modeling fundamentally aims to predict the next word given contextual words. bengio et al. (2000) introduced the neural probabilistic language model (nplm), which learns distributed word representations alongside probability functions for word sequences. before recurrent neural networks (rnns) gained prominence, approaches based on parse trees and n-gram statistics dominated the field. mikolov et al. (2010) demonstrated the superiority of rnn-based language models over standard n-gram techniques in speech recognition applications, despite their substantial computational complexity. building on this foundation, sutskever et al. (2011) advanced character-level modeling for text generation by training rnns with the hessian-free optimizer. the field was revolutionized by @xcite with the introduction of the transformer architecture, which implemented attention mechanisms to develop state-of-the-art machine translation models capable of generating text in one language given context in another. the transformer's parallelization capabilities effectively addressed the computational and training limitations of previous sequential models, leading to the development of influential architectures like bert @xcite and gpt @xcite ) that now underpin numerous contemporary nlp tasks. recent years have witnessed growing research interest in pretraining and finetuning nlp models for low-resource languages like nepali. @xcite pretrained a text generation model following @xcite 's configuration on a combined dataset comprising oscar, cc100, and scraped nepali wikipedia articles, employing sen-tencepiece tokenization with a 24,576 vocabulary size. @xcite trained three distinct transformer-based masked language models (distilbert-base, deberta-base, and xlm-roberta) for nepali text sequences, evaluating and comparing them against other transformerbased models on downstream classification tasks. in parallel work, @xcite finetuned multilingual bert specifically for named entity recognition tasks in nepali. @xcite developed another bert-based language model for nepali using wordpiece vocabulary with 30,522 subword tokens, demonstrating superior performance compared to other bertbased language models @xcite when finetuned on four distinct tasks: content classification, named entity recognition, part-of-speech tagging, and categorical pair similarity. despite these various pretraining and finetuning efforts in nepali, a comparative analysis of language model performance on downstream tasks using different tokenization approaches remains unexplored. several studies have investigated tokenization impacts in other languages. @xcite analyzed the efficiency (training time, carbon emissions) and effectiveness (performance) of various tokenization techniques by finetuning a turkish bert-based language model on multiple downstream nlp tasks, finding that for similar and smaller vocabulary sizes, character-level bpe and wordpiece outperformed other approaches like word-based tokenization. for korean, @xcite discovered that morpheme tokenization followed by character-level bpe achieved optimal performance, as this approach prevents bpe from considering byte sequences spanning multiple morphemes. alrefaie et al. (2024) observed similar results for arabic, where combining bpe with morpheme-based approaches proved most effective. additionally, @xcite evaluated different tokenization methods on three arabic nlp classification tasks, though without employing transformer-based architectures. our approach differs from these previous stud-ies in three significant ways. first, we finetune sequential (autoregressive) language models rather than bert-based architectures. second, we specifically analyze the performance of byte-level bpe tokenization algorithms-an aspect not thoroughly examined in prior work. finally, we provide empirical evidence challenging the predictive validity of perplexity-the commonly used intrinsic metric during language model pretraining-regarding downstream finetuning performance. ## conclusion in this paper, we compared perplexity scores across different tokenization methods using autoregressive language models for nepali. we found that more granular tokenization typically produces fewer high-frequency tokens, resulting in lower perplexity. increasing vocabulary size in wordbased tokenization correspondingly increased perplexity. however, our finetuning experiments on various nlu tasks revealed that tokenization methods with the best perplexity scores (byte-level bpe with/without morphemes) did not yield superior performance on understanding tasks. instead, sen-tencepiece consistently outperformed other methods across tasks despite having worse perplexity scores.