Subasa - Adapting Language Models for Low-resourced Offensive Language Detection in S inhala (2025)

Abstract: Accurate detection of offensive language is essential for a number of applications related to social media safety. There is a sharp contrast in performance in this task between low and high-resource languages. In this paper, we adapt fine-tuning strategies that have not been previously explored for Sinhala in the downstream task of offensive language detection. Using this approach, we introduce four models: “Subasa-XLM-R”, which incorporates an intermediate Pre-Finetuning step using Masked Rationale Prediction. Two variants of “Subasa-Llama” and “Subasa-Mistral”, are fine-tuned versions of Llama (3.2) and Mistral (v0.3), respectively, with a task-specific strategy. We evaluate our models on the SOLD benchmark dataset for Sinhala offensive language detection. All our models outperform existing baselines. Subasa-XLM-R achieves the highest Macro F1 score (0.84) surpassing state-of-the-art large language models like GPT-4o when evaluated on the same SOLD benchmark dataset under zero-shot settings. The models and code are publicly available.

## introduction a major challenge in the field of nlp are the disparities between high-and low-resource languages. these impact foundational language models as well as downstream tasks such as offensive language detection @xcite , an important task at the intersection of social media analysis and nlp. as people increasingly spend a significant portion of their day on online platforms like social media, their exposure to offensive or abusive language has surged @xcite . this trend is equally visible in sri lanka, where a substantial amount of social media content is generated in sinhala. studies show that an alarming amount of this content is hateful, and the severity of this issue is evident from several instances in recent years where the sri lankan government had to block social media platforms entirely to curb its spread, as it had fueled real-world unrest @xcite . sinhala (සිංහල) is an indo-aryan language spoken by over 17 million people in sri lanka and remains a low-resource language @xcite . for offensive language detection specifically, systems for sinhala lag behind those developed for resource-rich languages like english, spanish, and mandarin @xcite . to the best of our knowledge, fewer than five annotated offensive language datasets exist for sinhala, demonstrating its status as a low-resource language @xcite . while state-of-the-art large language models (llm) like gpt-4o demonstrate strong performance in many languages, our evaluations suggest they struggle to reliably identify offensive language in sinhala (results detailed in section 4). at the time of submission, the perspective api @xcite which is utilized extensively in both academia and industry for the purpose of identifying offensive content does not provide support for sinhala. our work addresses these shortcomings by introducing subasa (''සුබස''), which translates to wholesome language. in this paper, we present four variants of subasa. these models improve the current state of offensive language detection for sinhala by adapting fine-tuning strategies previously unexplored for sinhala. we address the following research questions: rq1: can intermediate pre-finetuning tasks-specifically masked rationale prediction (mrp)effectively improve pre-trained language models (plms) for offensive language detection in sinhala? rq2: can task-specific fine-tuning strategies improve the effectiveness of llms for offensive language detection in sinhala? pretrained language models (plm) have emerged as a powerful approach for a number of nlp tasks including offensive language detection. bert variants have shown success when fine-tuned for this task across both highresource languages like english @xcite and lower-resource contexts like arabic (althobaiti, 2022) and sinhala @xcite . while intermediate task training has shown promise in enhancing plm performance across various nlp tasks-from semantic parsing @xcite to natural language understanding @xcite -its application to offensive language detection emerged only recently with the introduction of masked rationale prediction (mrp) by @xcite . though mrp demonstrated significant improvements for english, its potential remains unexplored for low-resource languages. we are the first to adapt mrp to sinhala, addressing the language's data scarcity. llms are transformer-based models with billions of parameters trained on massive training corpora @xcite . while llms perform well in high-resource languages like english, their effectiveness in low-resource languages is often limited, as highlighted in various studies @xcite . adapting llms for lowresource languages is challenging because most are pre-trained primarily on english data. approaches to address this include; (i) continuing training with non-english data, (ii) transferring knowledge via supervised fine-tuning, and (iii) extending the llms vocabulary to include non-english tokens @xcite . for instance, @xcite demonstrated that finetuned llms can achieve strong performance even with limited data, as shown for turkish. jayakody and dias (2024) evaluated the gpt-4o, llama, and mistral models for various tasks in the sinhala language, revealing unsatisfactory results. notably, offensive language detection was not attempted. prior work on offensive language detection has explored fine-tuning open-source llms like llama and mistral, primarily for highresource languages like english @xcite and low-resource languages like vietnamese @xcite . however, prior work has not explored open-source llms (e.g., llama, mistral) for sinhala offensive language detection, despite their success in other lowresource languages like vietnamese @xcite . ## results and discussion concerning rq1, our subasa-xlm-r model achieves a macro-f1 of 0.84 ( ablation study insights show that mlm with 50% masking matches mrp's performance (0.83 vs 0.84 macro-f1). this suggests that in lowresource settings, any token-level intermediate task (mlm/mrp) can enhance downstream performance by reinforcing local context understand-ing. while both mrp and mlm improve performance, their similar results warrant further study into task-specific intermediate objectives for lowresource languages. when comparing results from ## conclusion this study addresses the challenge of offensive language detection in sinhala, a low-resource language, by introducing four novel models: subasa-xlm-r, subasa-llama (two variants), and subasa-mistral. to the best of our knowledge, our work is the first to adapt intermediate prefinetuning and task-specific fine-tuning strategies for sinhala, demonstrating significant advancements over existing baselines and state-of-the-art llms like gpt-4o. below, we summarize our findings in relation to our initial research questions posed in section 1: rq1: can intermediate pre-finetuning tasks (e.g., masked rationale prediction) improve plms for offensive language detection in sinhala? our results confirm that intermediate pre-finetuning with mrp enhances model performance, with subasa-xlm-r achieving a macro-f1 of 0.84, surpassing vanilla fine-tuned xlm-r (0.82). ablation studies reveal that token-level intermediate tasks-whether mrp or standard mlm-improve downstream task performance for sinhala (a low resource setting). notably, mlm with 50% masking nearly matches mrps gains (0.83 vs. 0.84 macro-f1), suggesting that reinforcing local context understanding through intermediate tasks aids the performance of the downstream task for sinhala. rq2: can task-specific fine-tuning improve llms for offensive language detection in sinhala? our results indicate that qlora enables opensource llms to specialize effectively for sinhala and surpass gpt-4o's zero-shot performance. (e.g., subasa-llama-3.1-8b achieves a macro-f1 of 0.826, outperforming gpt-4o (0.733) and its base model (0.586).) we publicly release all models and code to support sinhala nlp research. our results establish that strategic fine-tuning is beneficial for lowresource offensive language detection, with implications for other underrepresented languages.