G in G er at S em E val-2025 Task 11: Leveraging Fine-Tuned Transformer Models and L o RA for Sentiment Analysis in Low-Resource Languages (2025)

Abstract: Emotion recognition is a crucial task in natural language processing, particularly in the domain of multi-label emotion classification, where a single text can express multiple emotions with varying intensities. In this work, we participated in Task 11, Track A and Track B of the SemEval-2025 competition, focusing on emotion detection in low-resource languages. Our approach leverages transformer-based models combined with parameter-efficient fine-tuning (PEFT) techniques to effectively address the challenges posed by data scarcity. We specifically applied our method to multiple languages and achieved 9th place in the Arabic Algerian track among 40 competing teams. Our results demonstrate the effectiveness of PEFT in improving emotion recognition performance for low-resource languages. The code for our implementation is publicly available at: https://github.com/AylinNaebzadeh/Text-Based-Emotion-Detection-SemEval-2025.

## introduction sentiment analysis plays a crucial role in understanding human emotions in text, impacting various applications such as customer feedback analysis, social media monitoring, healthcare, and finance. assigning weights to emotions enhances the precision of sentiment classification, enabling more nuanced decision-making @xcite . with the advancement of deep learning and transformerbased models, sentiment analysis has become more efficient @xcite @xcite . however, achieving robust accuracy in emotion recognition remains a challenge, especially for low-resource languages, where data scarcity and linguistic diversity hinder model performance. we focus on categorical emotion classification, where emotions are assigned to discrete categories. early approaches to textual emotion classification primarily relied on handcrafted features, such as lexicons and rule-based methods @xcite . while modern deep learning models have significantly improved performance @xcite , they are highly dependent on large-scale datasets. when trained on limited data, these models often struggle with overfitting and poor generalization @xcite , making emotion recognition in low-resource settings particularly challenging @xcite . furthermore, we focus on weighted multi-label text classification, a more complex task where multiple emotions are assigned with varying intensities. while weighting mechanisms enhance emotion modeling, they also come with challenges such as data sparsity, label imbalance, and the difficulty of handling overlapping emotions effectively (kementchedjhieva and chalkidis, 2023). we focus on low-resource languages by leveraging transformer-based models, evaluating various architectures, including multilingual models. to mitigate overfitting and enhance generalization, we employ parameter-efficient fine-tuning (peft) techniques such as lora (low-rank adaptation) @xcite , enabling efficient adaptation while maintaining model robustness. to summarize, we conducted the following experiments on the semeval 2024 task 11 dataset: ## related work early text classification, including multi-label tasks, relied on traditional machine learning methods such as bag-of-words (bow) and tf-idf for feature extraction, using classifiers like naive bayes, support vector machines (svm), and logistic regression @xcite . these approaches represented text as sparse vectors and utilized statistical patterns for classification. with the rise of deep learning, models like convolutional neural networks (cnns) and recurrent neural networks (rnns) became popular for text classification @xcite . cnns captured local patterns, while long short-term memory (lstm) networks in rnns excelled in modeling sequential dependencies. although these methods improved performance by learning dense representations, they struggled with large datasets and long-range dependencies. the introduction of attention mechanisms and transformer architectures represented a major advancement @xcite . models like bert and gpt utilized selfattention to capture contextual relationships across documents, surpassing traditional methods in multilabel classification. however, their high computational costs remain a challenge. to mitigate these issues, parameter-efficient fine-tuning (peft) techniques have emerged, allowing large models to be fine-tuned with reduced computational and memory overhead @xcite . techniques such as lora (low-rank adaptation) @xcite , adapters @xcite , and prefix tuning (li and liang, 2021) facilitate efficient adaptation of pre-trained models to specific tasks, making them more feasible for resource-constrained environments. ## methodology our main focus in the first track was on afrikaans (afr), arabic algerian (arq), hindi (hin), and swedish (swe) languages. for the second track, we worked on russian (rus) and romanian (ron). to tackle this task, we employ several transformer-based architectures, which are detailed in the results section. in our experiments, we utilized a consistent set of hyperparameters, including a learning rate of 1e -5, 100 training epochs, a batch size of 8 for both training and evaluation, and a weight decay of 0.01. figure 2 represents the methodology of our work. we provide more information about the methodology for each task in separate subsections. ## results the output of confusion matrices and auc curves on the development datasets are in the appendix section. performance metrics in tables 3,4 reveal varying effectiveness of models across languages for emotion detection. the xlm-roberta-base model @xcite ## conclusion emotion detection and sentiment analysis remain challenging tasks in nlp, particularly for lowresource languages. in this paper, we presented our work and the performance of our models on six low-resource languages in a multi-label classification task using text-based data. our approach, which leveraged both multilingual and monolingual transformer-based classifiers, demonstrated that these models can achieve notable success. for future work, we aim to explore various hyperparameter settings and investigate the potential of generative models through prompting techniques.