Pula: Training Large Language Models for Setswana (2025)

Abstract: In this work we present Pula, a suite of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, Pula 8B and Pula 14B outperform GPT-4o and Gemini 1.5 Pro on English-Setswana translation tasks and achieve state-of-the-art performance on Setswana reasoning tasks for their size. We release the weights for Pula 1B, 3B, 8B, and 14B as well as training logs and training and evaluation code. Alongside Pula, we release the largest-ever Setswana text corpus, Marothodi, and the first comprehensive Setswana instruction-tuning dataset, Medupi, consisting of reformatted datasets, translated corpora, and synthetic LLM-generated text. To accompany this data, we release the code used for dataset construction, formatting, filtering, and scraping. Last, we release two Setswana LLM-translated benchmarks, MMLU-tsn and GSM8K-tsn, to measure Setswana knowledge and reasoning capabilities.

## introduction setswana, also known as tswana, is a bantu language spoken by an estimated five to ten million people worldwide @xcite . closely related to northern sotho and southern sotho, setswana holds official status in botswana (government of botswana, 2024), south africa @xcite , and zimbabwe (the parliament of zimbabwe, 2013), and is also used in countries like namibia, often interchangeably with english @xcite . despite its significance in the lives of millions, setswana has been largely overlooked in natural language processing (nlp) research, despite being classified by some works as a high-resource language @xcite . this work aims to bridge the gap between setswana and other high-resource languages by making open generative large language models capable of high-quality setswana available to the community for the first time, significantly increasing data availability, and laying the groundwork for future setswana-centric research. large language models (llms) have demonstrated powerful capabilities across various domains after training on web and synthetic data @xcite @xcite , excelling in areas such as mathematics @xcite , programming @xcite , creative writing @xcite , and translation tasks @xcite . however, developers continue to primarily target english and certain high-resource languages in training and evaluation. while existing approaches yield impressive capabilities, they may produce models which lack knowledge of certain cultures, limit production use-cases outside majority demographics, or prevent a significant portion of the global population from utilizing language models effectively. african languages like setswana, with little textual data available compared to languages such as english or french, subsequently suffer in performance and are underutilized in research. recent progress has been made to address the lack of language diversity in language models. releases such as mbart @xcite , @xcite , and bloom (bigscience @xcite were among the earliest and most influential advancements in multilingual language models. building upon these technologies, newer models including @xcite , claude @xcite , gemini @xcite , llama @xcite , and gemma @xcite have also found success in multilingual domains, often demonstrating reasoning and translation capabilities in languages not officially sup-ported. releases such as aya 101 @xcite and aya 23 @xcite have continued to improve language coverage for translation and generative tasks, and open corpora such as roots @xcite , oscar @xcite , and mc4 @xcite have made multilingual pre-training data readily available. however, setswana comprises only a small fraction of these datasets. for instance, just 0.0002% of the roots corpus is written in setswana. moreover, much of the available setswana text in open multilingual corpora is of lower quality or predominantly religious in nature, resulting in significantly worse conversational, translation, and reasoning capabilities in current open models. to help address this issue, we introduce the pula series of language models. this series consists of lora @xcite and qlora @xcite fine-tuned versions of llama 3.2 1b, llama 3.2 3b, llama 3.1 8b, and qwen 2.5 14b @xcite . by training a range of models across different parameter counts, we aim to provide the research community and millions of setswana speakers with models that are both highly performant and capable of running on various hardware configurations ranging from data centers to consumer laptops and mobile phones. to fuel the training behind these models and to provide the research community with resources to further improve future models and research, we hand-curate the largest-ever corpus of setswana text. in doing so, we merge several existing corpora which have not yet been consolidated, restore document-level text in certain data subsets, locate new sources of setswana text which to our knowledge have not yet been utilized in nlp research, develop several dedicated scrapers and parsers to obtain this new data, reformat existing datasets to function as instruction-tuning corpora, translate existing english instruction-tuning datasets to setswana using state-of-the-art large language models and translation models, and utilize gpt-4o for the generation of entirely new synthetic setswana text. we name our pre-training dataset marothodi and our instruction-tuning dataset medupi. following the open approach of projects like olmo @xcite and dolma @xcite , we perform a fully open release. this includes model weights, all training data, metadata on data sources, training logs, and code for training, evaluation, dataset curation, dataset translation, synthetic data generation, and web scraping. all model weights and data can be accessed on hugging face foot_0 . all code and training logs can be accessed on github foot_1 . ## related work although much of current nlp research is english-centric, there have been significant recent advancements in setswana-centric language models and data access. tswanabert (motsoehli, 2020) represents one of the earliest examples, trained on over ten thousand setswana sentences from the leipzig corpora collection @xcite , sabc news headlines @xcite , and various blogs and websites. more recently, the nchlt setswana roberta model @xcite was released, having been trained on over fourteen million tokens of setswana text from the nchlt (eiselen and puttkammer, 2014), autshumato @xcite , @xcite , and common crawl corpora, and an internal ctext corpus. puoberta (marivate et al., 2023a) marked a significant step forward in masked language modeling, achieving state-of-the-art performance while being the first language model trained from scratch. puoberta was released alongside puo-data, the largest collection of curated setswana text at the time, totaling 4.5 million puoberta tokens excluding jw300 @xcite . much of the literature on african languages has targeted massively multilingual developments with a focus on machine translation and transfer learning. corpora such as opus @xcite and madlad-400 @xcite provide access to large volumes of parallel web text data, while work such as mafand-mt @xcite has enabled improved translation performance across many african languages through additional human-generated data. meta's no language left behind (nllb) @xcite has facilitated high-quality machine translation between over 200 languages, including setswana, although it is noted as one of the 21 languages with the lowest accuracy on flores-200 @xcite . madlad-400 @xcite has allowed for in-creased multilinguality, but we find suffers from worse performance on setswana-english translations as seen in thanks to the cumulative improvements made by the research community in setswana-centric nlp and increased levels of multilinguality, we believe there are significant opportunities to enhance existing setswana nlp systems. through our investigations we find several smaller corpora of setswana text are often underutilized in the literature, potentially due to differing distribution methods and vastly differing formats, making data difficult to locate and utilize. for example, the south african center for digital language resources foot_2 (sadilar, 2024) has made publicly available several datasets of setswana text and audio. however, these texts are not distributed in a standardized format and are not typically available on commonly used external platforms such as hugging face and kaggle. as such, much of this data is left out of existing corpora. we also find many websites hosting content written in setswana are excluded from existing training datasets, meaning a significant portion of the available high-quality, and especially educational, setswana data is not being utilized. through the development and release of marothodi and medupi, we aim to reduce this burden and make data significantly easier to access. ## data one key consideration in pula's design and subsequent training data selection is identifying target languages. there are several parallel corpora available for setswana, and some works such as opus contain setswana text paired alongside multiple other languages. however, significantly more english-setswana parallel text is available. for example, opus contains over four times more parallel setswana-english sentences than setswana-french sentences foot_3 . in addition, many publicly available, commonly used, high-quality, educa- total tokens (millions) tional, and instruction-tuning datasets are written predominantly in english. consequently, many state-of-the-art llms already excel in english tasks, including the llama and qwen series of models which we select for continued pre-training. additionally, native speakers in regions such as botswana and south africa often utilize english in legal, official, and government documents, with english sometimes being spoken interchangeably with setswana (parliament of botswana, 2022). we observe this trend continued in web text documents, where many data sources are written either exclusively in setswana or in both setswana and english, either incorporating code-switching or providing direct translations @xcite . due to the strong link between these two languages and in an effort to reduce scope and computational requirements, we focus our efforts primarily on curating a high-quality dataset consisting of setswana and english texts. we refer to our corpus of raw setswana text as marothodi, and our instruction-tuning dataset as medupi. our datasets build upon several prior works in the african nlp research community. we measure token counts using the llama 3.1 tokenizer throughout this paper for consistency. we release both marothodi and medupi in their entirety on hugging face foot_4 , as well as per-sequence identification metadata including a corpus identifier and an exact source url where available. ## evaluation we evaluate our models on a variety of tasks in both setswana and english, including translation, natural language understanding, multiple choice question-answering, and mathematical reasoning. all local evaluations are performed using bf16 precision using vllm @xcite . translation. to measure translation performance, we evaluate on the mafand-mt (adelani et al., 2022a), lego-mt @xcite benchmarks. these benchmarks cover a variety of translation domains, including news headlines, web articles, and miscellaneous web documents. to gauge performance, we utilize the bleu @xcite and chrf (popović, 2015) metrics. translation performance results are presented in to evaluate pula's knowledge and reasoning capabilities in setswana we develop mmlu-tsn and gsm8k-tsn -setswana translations of the entirety of the test splits of the original massive multitask language understanding (mmlu) @xcite and grade school math 8k (gsm8k) @xcite benchmarks. we translate using gpt-4o and gemini 1.5 pro, respectively. we acknowledge this method's reliance on translation systems to translate this task, especially given its technical nature, is likely to suffer from "translationese" and other errors @xcite . many sequences may be incorrectly translated, biased, or otherwise impossible to solve without lucky guessing. however, we find these translated benchmarks to still be a useful proxy for a model's performance in reasoning, knowledge, and instruction-following capabilities. these results are provided in we find the pula series excels at translating between english and setswana, with both pula 14b and pula 8b on average outperforming significantly larger frontier models such as gpt-4o and gemini 1.5 pro. these models exceed the performance of all tested open llms as well as the nllb 200 and madlad 400 machine translation models. we note the pula series tends to yield higher quality translations when translating from setswana to english -a trend we see in other tested llms but not in machine translation models. reasoning. we evaluate reading comprehension using meta's belebele benchmark, a corpus of multiple-choice questions regarding passages sourced from flores-200. we evaluate world knowledge and question-answering via the mmlu and mmlu-tsn benchmarks, containing multiple-choice questions on topics such as mathematics, computer science, law, and more. last, we evaluate mathematical reasoning via gsm8k and gsm8k-tsn, which consist of open-ended gradeschool math word problems. we find pula 14b to, on average, outperform gpt-4o mini as well as all tested open models across setswana and english reasoning tasks. pula 8b outperforms or is competitive with llama 3.1 70b on all three benchmarks, and pula 14b significantly outperforms llama 3.1 70b on all setswana benchmarks. in addition, we find pula's performance steadily grows with scale in both setswana and english tasks; a trend displayed in the performance of existing llama and aya models. however, we do note the roughly equivalent performance of aya 23 8b and 35b on setswana belebele. given the lack of intentional setswana data present in aya 23's training corpus, this indicates reading comprehension may be a less transferable skill for llms when working with largely unseen languages compared to question-answering and mathematical reasoning. we also note large discrepancies between setswana and english versions of benchmarks, such as mmlu and mmlutsn. we attribute these differences to accumulative errors during translation such as ambiguous or incorrect wording, impossible questions, or damaging modifications to the correct answer. ## conclusion in this work we introduce pula, the first series of large language models tailored for setswana. our models demonstrate significantly improved translation and reasoning performance, rivaling models much larger than themselves on setswana reading comprehension, question-answering, and mathematical reasoning tasks while retaining existing performance on english tasks. pula exceeds in translating between setswana and english, with pula 8b and 14b on average outperforming gpt-4o and gemini 1.5 pro, with pula 14b also outperforming gpt-4o-mini in setswana reasoning tasks. we introduce marothodi, the largest-ever single corpus of raw setswana text, and medupi, the first-ever comprehensive setswana instructiontuning dataset. we develop and release mmlutsn and gsm8k-tsn, setswana translations of the mmlu and gsm8k benchmarks translated using gpt-4o and gemini 1.5 pro. our results indicate there may be significant performance gains not yet reached in other languages which may be available using existing underutilized data and synthetic data generation. to support future nlp research and production use cases, we release model weights, data, data curation code, benchmarks, training and evaluation code, and training logs.