A mharic News Topic Classification: Dataset and Transformer-Based Model Benchmarks (2025)

Abstract: News classification is a downstream task in Natural Language Processing (NLP) that involves the automatic categorization of news articles into predefined thematic categories. Although notable advancements have been made for high-resource languages, low-resource languages such as Amharic continue to encounter significant challenges, largely due to the scarcity of annotated corpora and the limited availability of language-specific, state-of-the-art model adaptations. To address these limitations, this study significantly expands an existing Amharic news dataset, increasing its size from 50,000 to 144,000 articles, thus enriching the linguistic and topical diversity available for the model training and evaluation. Using this expanded dataset, we systematically evaluated the performance of five transformer-based models: mBERT, XLM-R, DistilBERT, AfriBERTa, and AfroXLM in the context of Amharic news classification. Among these, AfriBERTa and XLM-R achieved the highest F1-scores of 90.25% and 90.11%, respectively, establishing a new performance baseline for the task. These findings underscore the efficacy of advanced multilingual and Africa-centric transformer architectures when applied to under-resourced languages, and further emphasize the critical importance of large-scale, high-quality datasets in enabling robust model generalization. This study offers a robust empirical foundation for advancing NLP research in low-resource languages, which remain underrepresented in current NLP resources and methodologies.

## introduction amharic, ethiopia's official working language, is spoken by millions and is the second most widely used semitic language after arabic.as internet penetration expands, there is a significant increase in the consumption of online content, particularly in digital news. the shift from traditional to digital media has amplified the volume of unstructured text data, presenting opportunities for advanced nlp applications, such as text classification(tc). topic classification (tc) plays a vital role in organizing unstructured textual data, enabling automated news categorization, and supporting recommendation systems. although transformer-based models and large language models (llms) have significantly advanced tc in high-resource languages, amharic remains notably underrepresented in this domain. this gap is largely due to the scarcity of large-scale, labeled datasets and the limited application of modern nlp techniques specifically tailored to the linguistic and contextual characteristics of amharic. earlier work, such as (azime and mohammed, 2021), introduced a foundational amharic news dataset, but regular updates are required. other studies relied on smaller datasets and lacked reproducibility due to the inaccessibility of resources @xcite , highlighting the need for more robust approaches. this paper presents an in-depth evaluation of transformer-based models for the classification of amharic news. the contributions of this study are twofold: the findings of this study offer valuable insight into the application of state-of-the-art transformer models for low-resource languages, such as amharic. the results support the development of more accurate and efficient news classification systems, with potential applications in content aggregation, personalized recommendations, and automated news filtering. ## related work the availability of organized and machine-readable data in high-resource languages has privileged them in nlp research, while low-resource languages in africa and asia remain underrepresented. masakhanews (adelani et al., 2023) explored multilingual transformers, such as mbert and xlm-r, to classify news topics in 16 african languages, including amharic. using transfer learning, the study achieved promising results but faced challenges in generalization due to the limited availability of annotated data and linguistic complexity. similarly, masakhaner introduced ner datasets for 10 languages, which later expanded to 20 @xcite , but excluding major ethiopian languages such as afaan oromo, tigrigna, and somali, revealing a research gap. afrisenti @xcite advanced sentiment analysis in 14 african languages, including amharic, afaan oromo, and tigrigna. although models such as afriberta perform well, they still struggle with language-specific issues such as imbalance and structural variation. afrib-erta @xcite , trained in less than 1gb of text in 11 african languages, outperformed mbert and xlm-r in some tasks, but could not fully address concerns about data quality or diversity. other efforts include monolingual models such as puoberta for setswana and transformer-based tc work on ewe, swahili, and kinyarwanda, demonstrating that language-specific models often outperform general-purpose models. among semitic languages, arabic leads the development of nlp research, with dedicated models such as arabert, marbert, and arabicbert @xcite . hebrew has also benefited from monolingual models such as alephbert and hebbert @xcite , which surpass multilingual baselines. ethiopia has more than 85 languages; however, nlp research remains limited to amharic, the most studied language. a review by @xcite emphasized the fragmented state of nlp for ethiopian languages, with a lack of datasets, benchmarks, and transformer-based models in particular. even major projects such as masakhanews and afrisenti rarely go beyond amharic, neglecting other widely spoken languages such as afaan oromo and tigrigna. the reviewed studies have several key limitations. although amharic is ethiopia's most widely spoken and official language, there has been limited progress in developing robust nlp resources. many studies rely heavily on multilingual models that often struggle to capture the unique linguistic and contextual features of amharic. in addition, there is a lack of language-specific standardized benchmarks for text classification in amharic and other ethiopian languages. this study focuses on classifying amharic news using five transformer-based models: mbert, xlm-r, distilbert, afriberta, and afroxlm. its main contributions include fine-tuning and evaluating these models on a significantly expanded and original amharic news dataset, establishing benchmark results through comparative performance analysis , and providing enriched data and insights that support future nlp research in amharic and can also encourage similar studies in other underrepresented languages. ## experimentation the expanded amharic news dataset developed for this study comprises 144,201 articles published between 2011 and late 2024. it integrates 92,792 newly collected articles with an existing, manually filtered set of 51,409 articles from the original dataset (reduced from 51,471 after excluding entries with incomplete or ambiguous content). articles were collected from 12 major amharic news outlets, using beautifulsoup. data scraping followed ethical and responsible practices, involving only publicly accessible content, excluding paywalled material, and adhering to the terms of service of each source. the dataset was constructed through a semiautomatic pipeline, preserving editorial category labels provided by the sources (e.g., local news, entertainment, sports, business, international, politics). a manual quality assurance step was applied to remove records with missing or invalid data. each entry includes a headline, article body, category label, and source url link. when available, the publication date and view count of the news article are also included; missing metadata is marked as na. records lacking the headline or body of the article were excluded to ensure data quality. compared to previous amharic news datasets @xcite , this expanded corpus significantly increases scale and metadata richness, offering improved support for news topic classification and other low-resource nlp tasks. the preprocessing steps included text cleaning, metadata curation, and stratified partitioning into training (70%), validation (10%), and test (20%) sets. the dataset not only advances amharic text classification but also enables exploration of imbalance-handling methods and finer-grained categorization in future work. tokenization used a pre-trained tokenizer with padding/truncation to 512 tokens, and the category labels were encoded using scikit-learn's labelencoder. this study fine-tuned and evaluated five transformer-based models: mbert, distilbert, xlm-r, afroxlm, and afribertafor amharic news classification. bert and its variants, such as mbert and distilbert, leverage masked language modeling and prediction of the next sentence to generate deep contextual representations @xcite @xcite . xlm-r, which is trained with a large multilingual corpus, offers strong cross-lingual performance @xcite . afroxlm and afriberta, trained with african languages, improve generalization for underrepresented and morphologically complex languages, such as amharic @xcite . fine-tuning was performed using hugging face's trainer api with batch sizes of 16 and 32, gradient accumulation steps of 4, a learning rate of 5 × 10 -5 , weight decay of 0.1, and mixedprecision training (fp16). models were trained for five epochs, evaluated using f1 score, and implemented in a linux kaggle environment (tesla p100 gpu, python 3.10.14). pre-trained models were tokenized using autotokenizer and padded via dat-acollatorwithpadding. the input text combined cleaned headlines and content, tokenized with truncation, and the maximum length per model. the cross-entropy loss and adamw optimizer were used. the evaluation metrics (accuracy, precision, recall, and f1-score) were logged using the w&b. the dataset was normalized, label-encoded, and split using stratified sampling (70% train, 10% validation, 20% test) across six news categories. we used a confusion matrix to assess the model's generalization. ## results and discussion to assess the effect of batch size on model performance, each transformer-based architecture was fine-tuned using batch sizes of 16 and 32. although the performance differences between the configurations were relatively modest, the final reported results corresponded to the setting that yielded the highest macro f1-score on the test set. macro f1 was selected over weighted f1 as the primary evaluation metric to provide a balanced assessment across all classes, particularly considering the inherent class imbalance in the dataset. this choice ensured that the evaluation did not disproportionately favor majority classes, thereby supporting a the evaluation results show that the expanded amharic news dataset significantly improved model performance across the board, especially for models with larger parameter capacities. as shown in per-class f1 analysis for example, mbert's f1-score for international improved from 0.19 to 0.45, and business from 0.34 to 0.43, indicating a meaningful reduction in class imbalance and better coverage of lowresource categories. although categories such as entertainment remain challenging due to limited examples and semantic overlap, the overall classification balance was markedly improved. afroxlmr also maintained strong and stable performance across both datasets, while lightweight models such as mbert and distil-bert, despite lower overall accuracy, still benefited from the data expansion. the expanded dataset substantially enhanced classification performance by providing more representative training samples, particularly benefiting large-scale transformer models. these results reaffirm the importance of domain-specific and linguistically aligned data for advancing nlp in low-resource languages such as amharic. ## conclusion and recommendation this study assessed the performance of transformerbased models for amharic news topic classification using an expanded dataset comprising over 144,000 articles. the results demonstrated that afriberta and xlm-r consistently delivered superior performance in both accuracy and f1 scores, underscoring the effectiveness of language-specific or regionally pretrained models for low-resource languages. afroxlmr also achieved strong results, reinforcing the value of pretraining strategies that incorporate african linguistic features. in contrast, general-purpose models such as mbert and distil-bert struggled to capture the linguistic complexity of amharic, particularly in terms of morphology and syntax. building on the findings of this study, future work should consider adopting more sophisticated classification strategies, such as multi-label and hierarchical models, to better capture topic overlap commonly found in news content. incorporating cross-lingual transfer learning and few-shot learning techniques could also enhance model adaptability across other under-resourced african languages. given the reliance of the datasets on editorially assigned labels, future research should investigate possible labeling inconsistencies or bias, which can impact classification performance. introducing human-in-the-loop validation can further improve data quality and support the development of a gold-standard benchmark subset. additionally, the rich metadata structure of the dataset opens opportunities for broader nlp applications, including news summarization, headline generation, and temporal topic modeling. to enable deployment in real-world and low-resource settings, future efforts should focus on compressing large-scale models, such as developing distilled versions of afriberta without significantly compromising performance. finally, ongoing attention to the linguistic characteristics of amharic, including its complex morphology, orthographic variations, and context sensitivity, will be essential to build more robust and generalizable language technologies.