B angla B y T 5: Byte-Level Modelling for B angla (2025)

Abstract: Large language models (LLMs) have achievedremarkable success across various natural lan-guage processing tasks. However, most LLMmodels use traditional tokenizers like BPE andSentencePiece, which fail to capture the finernuances of a morphologically rich languagelike Bangla (Bengali). In this work, we introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla.Built upon a small variant of Google’s ByT5architecture, BanglaByT5 is pre-trained on a14GB curated corpus combining high-qualityliterary and newspaper articles. Through zero-shot and supervised evaluations across gen-erative and classification tasks, BanglaByT5demonstrates competitive performance, surpassing several multilingual and larger models.Our findings highlight BanglaByT5’s potentialas a lightweight yet powerful tool for BanglaNLP, particularly in resource-constrained orscalable environments. BanglaByT5 is pub-licly available for download from https://huggingface.co/Vacaspati/BanglaByT5.

## introduction large language models (llms) have redefined natural language processing (nlp) by achieving strong results across multiple tasks like machine translation, question answering, and paraphrasing. however, these models rely on subword tokenization (e.g., bpe, sentencepiece), which fragments words inconsistently and poses significant challenges when applied to morphologically rich indian languages like bangla @xcite . in contrast, byte-level modelling operates directly on raw bytes, enabling models to handle linguistic variations uniformly across scripts and domains. in this paper, we introduce banglabyt5, the first monolingual byte-level encoder-decoder model for bangla, built on the small byt5 architecture and pre-trained on a 14gb balanced corpus combining v ācaspati (literature) @xcite and indiccorp (news) corpora @xcite . banglabyt5 is evaluated on classification and generative tasks under zero-shot and fine-tuned settings. it outperforms similar-size models like indicbart and banglat5, as well as bloom-1.1b @xcite , and performs within 5% of gpt2-xl @xcite despite being 5 times smaller. our contributions are as follows: • we propose banglabyt5, the first monolingual byte-level encoder-decoder model for bangla. ## related work elmo @xcite and bert @xcite have achieved strong results on nlu benchmarks. for generation tasks, decoder-only models like gpt-2/3 @xcite and encoder-decoder models like t5 @xcite , and mt5 @xcite ) have become prominent. byt5 @xcite , a bytelevel extension of mt5, has shown advantages for morphologically rich languages. while characteraware models like characterbert @xcite and others @xcite @xcite @xcite incorporate subword-free representations, they still rely on token boundaries. other efforts @xcite address tokenization challenges through vocabulary adaptation or randomized subword segmentation. recent multilingual models like llama-3 @xcite , mistral @xcite , and indicbart @xcite include bangla. byt5 model for mor-phologically rich language like sanskrit @xcite has also been adopted. monolingual models like banglat5 @xcite and paramanu @xcite are available for bangla. ## banglabyt5 in this section, we discuss in detail the various aspects of our proposed model, banglabyt5. ## evaluation in this section, we explore the efficacy of banglabyt5. we adopted a two-fold approach. first, we asked banglabyt5 to generate responses to curated bangla prompts to evaluate its generative abilities in the zero-shot setting. zero-shot evaluation is particularly important because it reveals the model's inherent generative ability without reliance on domain-specific adaptation. then, we evaluated the performance of banglabyt5 on both classification and generation-based downstream tasks in supervised fine-tuning mode. ## deployability in this section, we evaluate the scalability of the banglabyt5 model under cpu-only and gpuaccelerated environments to assess its deployment potential. gpu-scalability reflects how well the model leverages parallelism for high-throughput or real-time deployment, while cpu-scalability captures performance in low-resource environments. this dual perspective is essential for understanding the potential of deployment in cloud and offline settings. latency denotes the average time (in seconds) required to generate an output of a single prompt, including tokenization, model forward pass and decoding. throughput, on the other hand, focusses on number of prompts processed per second. we also monitor the peak memory usage for a prompt in both cpu and gpu mode, as this is a critical consideration for deployment. byte-level modelling produces more tokens than subword tokenization, thereby increasing training and inference time. large training time is a bottleneck for deployment. to evaluate the performance ## conclusions we presented banglabyt5, a byte-level monolingual language model explicitly tailored for morphologically rich languages like bangla. through rigorous evaluation, we demonstrate that banglabyt5 surpasses existing bangla models and matches or outperforms larger multilingual models in both generation and classification tasks. moreover, its scalability in low-resource environments positions it as a practical and impactful tool for bangla nlp. banglabyt5 is available for download from https: //huggingface.co/vacaspati/banglabyt5 .