A fro XLMR -Social: Adapting Pre-trained Language Models for A frican Languages Social Media Text (2025)

Abstract: Language models built from various sources are the foundation of today’s NLP progress. However, for many low-resource languages, the diversity of domains is often limited, more biased to a religious domain, which impacts their performance when evaluated on distant and rapidly evolving domains such as social media. Domain adaptive pre-training (DAPT) and task-adaptive pre-training (TAPT) are popular techniques to reduce this bias through continual pre-training for BERT-based models, but they have not been explored for African multilingual encoders. In this paper, we explore DAPT and TAPT continual pre-training approaches for African languages social media domain. We introduce AfriSocial, a large-scale social media and news domain corpus for continual pre-training on several African languages. Leveraging AfriSocial, we show that DAPT consistently improves performance (from 1% to 30% F1 score) on three subjective tasks: sentiment analysis, multi-label emotion, and hate speech classification, covering 19 languages. Similarly, leveraging TAPT on the data from one task enhances performance on other related tasks. For example, training with unlabeled sentiment data (source) for a fine-grained emotion classification task (target) improves the baseline results by an F1 score ranging from 0.55% to 15.11%. Combining these two methods (i.e. DAPT + TAPT) further improves the overall performance. The data and model resources are available at HuggingFace.

## introduction pre-trained language models (plms) are initially trained on vast and diverse corpora, including encyclopedias and web content @xcite . subsequently, these pretrained models are used in supervised training for a specific natural language processing (nlp) task 1 https://huggingface.co/tadesse/ afroxlmr-social figure 1 : continual pre-training illustration. a general-purpose pretrained model, such as afrox-lmr, is first adapted to a social domain, resulting in afroxlmr-social. this model then undergoes crosstask tapt using sentiment analysis, emotion, and hate speech data without the labels for further fine-tuning. by further finetuning. fine-tuned plms achieve strong performance across many tasks and datasets from various sources @xcite . however, this raises a question: do plms function universally across domains, or does continual training of plms with domain-specific data offer better performance? while some studies have shown the benefit of continual pre-training on a domain-specific unlabeled data @xcite , one domain may not be generalizable to other domains and languages. moreover, it is unknown how the benefit of continual pre-training may vary with factors like the amount of unlabeled corpus, the source domain itself, the evaluation task, the resource richness of the target languages, and the trained target model @xcite . this raises the question of whether pre-training on a corpus more directly tied to the task can further improve performance. this work addresses these questions by continual domain adaptive pre-training (dapt) and task adaptive pre-training (tapt) on the downstream subjective nlp tasks in a low-resource language setup. we consider the social media domain (x) and news for a continual pre-training from a highperforming multilingual baseline model, afrox-lmr @xcite . afroxlmr is a continually pre-trained model for african languages based on xlm-roberta @xcite . we explore subjective nlp task results from baseline (the base model result), dapt, tapt, and dapt + tapt, on a smaller but directly domain-and taskrelevant unlabeled corpus. the results show that dapt and tapt highly benefit from similar source nlp tasks. figure 1 illustrates the general highlevel continual pre-training strategies. in summary, our contributions are: ## afrisocial: domain adaptation corpus social domain in our work in computational linguistics, domain boundaries can be defined through various dimensions, including content, style, and purpose @xcite . in this work, we define the social domain based on two key factors: 1) convergent textual characteristics, x and news sources provide a public discourse, comments, reactions, and conversational linguistic patterns that facilitate social interactions, and cultural expressions and 2) functional similarity in downstream tasks, the selected evaluation task datasets are sourced from the two sources, shown in we create afrisocial, a social domain-specific corpus comprising x and news for 14 african languages, shown in ## evaluation tasks and datasets we select subjective nlp tasks for our evaluation based on the following reasons. 1) subjective tasks face more disagreement during annotations, leading to less performance in the evaluation, especially for low-resource languages @xcite 5 language models ## conclusion this work explored the effects of domain-adaptive pre-training (dapt) and task-adaptive pre-training (tapt) across three subjective tasks involving african languages. we created the afrisocial corpus, a social domain-specific corpus sourced from x and news. using afrisocial, we further developed the afroxlmr-social language model, which specialized in the social domain. we improved the performance of evaluated tasks, sentiment analysis (afrisenti), emotion analysis (afriemo), and hate speech classification (afri-hate) using dapt and tapt techniques. we showed that pre-training the model towards a small domain-specific corpus and related task-relevant data can provide significant improvements. while the combination of the two methods, dapt + tapt, also achieved better results than the baseline models, tapt followed by dapt would be susceptible to catastrophic forgetting of the taskrelevant corpus. we achieved better state-of-theart results using a small domain-related corpus from the encoder-only model than state-of-the-art large-language models (llms). afrisocial and afroxlmr-social will support the development of african languages in the nlp and improve similarsourced tasks. it opened further domain explorations as the afrisocial x and news domains are also available separately.