T itu LLM s: A Family of B angla LLM s with Comprehensive Benchmarking (2025)

Abstract: In this paper, we present TituLLMs, the first large pretrained Bangla LLMs, available in 1b and 3b parameter sizes. Due to computational constraints during both training and inference, we focused on smaller models. To train TituLLMs, we collected a pretraining dataset of approximately ∼ 37 billion tokens. We extended the Llama-3.2 tokenizer to incorporate language- and culture-specific knowledge, which also enables faster training and inference. There was a lack of benchmarking datasets to benchmark LLMs for Bangla. To address this gap, we developed five benchmarking datasets. We benchmarked various LLMs, including TituLLMs, and demonstrated that TituLLMs outperforms its initial multilingual versions. However, this is not always the case, highlighting the complexities of language adaptation. Our work lays the groundwork for adapting existing multilingual open models to other low-resource languages. To facilitate broader adoption and further research, we have made the TituLLMs models and benchmarking datasets publicly available.

## introduction the rapid advancements in large language models (llms) have reshaped the field of ai, showcasing remarkable versatility across numerous tasks @xcite @xcite . these models demonstrate not only an ability to perform various nlp tasks but also an intriguing potential for self-assessment and continuous improvement @xcite @xcite . despite these advancements, llm development-both open and closed-has predominantly focused on multilingual models, with a stronger emphasis on high-resource languages @xcite . while some mod-1 https://github.com/hishab-nlp/titulm els have extended coverage to medium-and lowresource languages @xcite @xcite , their representation remains limited. although some initiatives have aimed to train language-centric llms @xcite , these efforts remain scarce due to the high costs associated with computational resources and data collection. consequently, recent research has shifted towards adapting existing llms for new languages @xcite . similarly, in the context of benchmarking llms, most efforts have primarily focused on highresource languages @xcite , while low-resource languages, such as bangla, have received limited attention @xcite @xcite . @xcite developed llms for bangla using llama, leveraging only the bangla subset of culturax @xcite , which consists of 12.4 million diverse bangla news articles. they further fine-tuned their model on 172k instruction samples from subsets of the alpaca @xcite and openorca @xcite datasets, which were translated from english. their models were benchmarked on 120 queries across nine different generation tasks. addressing this gap is crucial, as linguistic and cultural diversity significantly impact language understanding and generation. therefore, in this study, we focus on adapting existing llms (e.g., llama) by expanding the model's vocabulary and performing continual pretraining. this process required extensive data collection from diverse sources. given the relatively low availability of digital content in bangla, we also developed synthesized datasets to supplement our training data. benchmarking llm capabilities for bangla remains challenging due to the lack of specialized datasets, particularly in areas such as world knowledge and commonsense reasoning. although some efforts have been made to generate such datasets through translation @xcite , they remain limited in scope. to address this gap, we have developed several native and translated datasets. compared to @xcite , our pretraining corpus is significantly larger (∼ 37b tokens) and is benchmarked on five different datasets covering world knowledge and commonsense reasoning, with a total dataset size of 132k entries. our study reveals several interesting findings: vocabulary extension: we explore the impact of vocabulary extensions on the base llama tokenizer by increasing the number of new tokens from 32k to 96k in increments of 16k. we found that average tokens per word (tpw) decreases as the number of tokens increases up to a certain point, after which it declines only minimally. therefore, when adding new tokens, we must also consider the fertility rate to balance the trade-off between training and inference. commonsense capability: titullms demonstrates strong commonsense knowledge but has limited capability in world knowledge (e.g., bangla mmlu). further training with instruction finetuning may enhance its performance in this area. ## pretraining data pretraining data for bangla is very limited compared to very high quality data available for english and other high resource languages @xcite . hence, we needed collect pretraining dataset for training titullms. we have compiled a substantial bangla raw dataset from a diverse range of sources, encompassing both formal and informal linguistic styles. the dataset is primarily derived from three key sources: web documents, books, and synthetically generated text. the synthetically generated data includes translated data, transliterated data, audio transcripted data, etc. an outline of our data collection and preprocessing pipeline is shown in figure 7 . the final highquality dataset amounts to approximately 268 gb, with 22 gb designated as a validation set, sampled proportionally to maintain the heterogeneity of the data. in total, the corpus contains ∼ 37 billion tokens, optimized for training and evaluation across various bangla language processing applications. in ## results and discussion we evaluate each model in 0-shot and 5-shot settings to assess their few-shot adaptability. ## conclusion in this study, we present the first pretrained bangla llms, titullms, trained on ∼ 37b tokens by adapting llama-3.2 models. we extended the tokenizer to incorporate language-and culture-specific knowledge, which also enable faster training and inference. pretraining data collection remains challenging for languages with low digital representation. to address this, we provide a comprehensive approach, including raw web data collection, translation, and synthetic data generation. given the lack of llm-based benchmarking datasets, we developed five datasets comprising 137k samples, covering both knowledge and reasoning. the benchmarking dataset includes manually curated samples as well as a novel translation-based (est) approach. using these datasets, we benchmarked various llms, including titullms, demonstrating its superior performance in reasoning tasks without instruction tuning. future work includes collecting larger pretraining datasets and fine-tuning with instruction-based datasets.