INFOTEC - NLP at S em E val-2025 Task 11: A Case Study on Transformer-Based Models and Bag of Words (2025)

Abstract: Leveraging transformer-based models as feature extractors, we introduce a hybrid architecture that integrates a bidirectional LSTM network with a multi-head attention mechanism to address the challenges of multilingual emotion detection in text. While pre-trained transformers provide robust contextual embeddings, they often struggle with capturing long-range dependencies and handling class imbalances, particularly in low-resource languages. To mitigate these issues, our approach combines sequential modeling and attention mechanisms, allowing the model to refine representations by emphasizing key emotional cues in text.

## introduction since its inception, artificial intelligence has sought to solve human and social problems through techniques such as natural language processing (nlp), which combines computational and linguistic methods to enable computers to understand human languages in formats such as text and audio/voice @xcite . several initiatives have introduced emotion detection tasks to encourage the research community to develop competitive tasks for processing, understanding, and generating text. these efforts present new research challenges and establish state-of-theart results in this field. there are works that have significantly advanced the field of emotion detection by tackling different aspects, e.g., @xcite laid a strong foundation by introducing multi-label emotion classification and emotion intensity prediction in tweets, providing a benchmark for fine-grained affect detection. building on this, the work of @xcite extends the challenge to conversational contexts, emphasizing emotion shifts and their reasoning, a crucial step toward developing systems capable of understanding emotional transitions. meanwhile, @xcite contribute to the field by introducing emotion detection in spanish-language tweets, addressing the gap in multilingual emotion classification, and enhancing nlp applications for non-english texts. these joint efforts highlight the growing importance of emotion-aware systems, particularly in social media monitoring, mental health applications, and human-computer interaction @xcite . by broadening the scope of affective computing to include emotion intensity, conversational reasoning, and language diversity, these advancements improve the depth and applicability of the field. their contributions drive the development of more context-aware, explainable, and adaptable models. despite significant advancements in recent years to address the challenges of sentiment analysis, emotion classification remains a complex task for nlp systems, whose relevance has continued to grow over time. this paper presents a model for track a of semeval-2025 task 11: bridging the gap in text-based emotion detection @xcite @xcite , which focuses on predicting a speaker's perceived emotions based on a given text snippet. we propose to solve the challenge as a multi-label classification problem using an approach that leverages a transformer-based encoder to extract deep semantic representations, with a bi-lstm for cap-turing temporal dependencies and multi-head attention mechanisms to refine emotion-specific features further. this architecture enables the effective detection of multiple emotions in multilingual text, contributing to advancements in affective computing. the rest of the paper is organized as follows: section 2 describes the problem and the related work. section 3 presents the system proposed and discusses the experimental setup to tackle task a; meanwhile, section 4 analyzes the results. the conclusions are presented in section 5. ## background emotion detection involves identifying a person's sentiments or feelings. in computational linguistics, it identifies a discrete emotion in a text @xcite . this task is challenging due to cultural differences, linguistic ambiguity, and the use of slang. the techniques for emotion detection include lexicon-based, machine learning, and deep learning approaches @xcite . lexiconbased approaches use dictionaries of words with sentiment values to determine the predominant emotion in a text. on the other hand, machine learning models rely on labeled datasets to train supervised models that predict emotions. the most recent methodologies are based on deep learning, which applies neural networks with minimal feature engineering. the work of @xcite tackles the problem of multi-label emotion classification using an approach that combines multiple attention mechanisms with recurrent neural networks and pre-trained transformer models (through transfer learning). this approach achieves results that surpass the state of the art in terms of accuracy. @xcite propose an emotion detection model that distills knowledge from a high-performing english monolingual model to a multilingual model. this approach enhances emotion detection, demonstrating the effectiveness of knowledge transfer for robust and explainable multilingual models. despite these advancements, emotion recognition research has primarily focused on highresource languages, leaving low-resource languages underrepresented. nevertheless, recently, two studies have introduced large-scale multi-label emotion datasets aimed at improving multilingual emotion classification. @xcite introduce brighter, a collection of emotionannotated datasets spanning 28 languages from africa, asia, eastern europe, and latin america. the dataset integrates diverse sources and employs various annotation strategies by fluent speakers. it features multi-label annotations and intensity levels, enabling a more nuanced understanding of emotions. experimental results highlight the variability in large language model (llm) performance across languages and explore cross-lingual transfer learning. findings show that multilingual models achieve better results when trained on linguistically related languages, while llms performance drops significantly in low-resource settings. belay et al. ( 2025 ) introduce ethioemo, a multilabel emotion dataset for ethiopian languages, compiled from diverse sources such as social media and news. the study extensively evaluates multiple language model architectures and explores translationbased evaluation methods. the findings highlight the challenges of multi-label emotion classification in low-resource settings and expose the limitations of llms in capturing emotional nuances across linguistic structures. these datasets and experiments advance the state-of-the-art in multilingual and multi-label emotion classification as part of semeval 2025 task 11. ## system overview the proposed system is designed for multilingual emotion classification using a hybrid deep learning approach. it integrates a pre-trained transformerbased model @xcite , a bidirectional lstm @xcite (bi-lstm) network, and a multi-head attention mechanism @xcite . the decision to integrate bi-lstm and attention mechanisms with transformer-based architectures stems from the complementary advantages each component brings to natural language processing tasks. bi-lstm is well suited for capturing sequential dependencies and maintaining word order, both of which are crucial in tasks such as text classification and sentiment analysis, where the temporal structure of input matters. pure transformer models can struggle with fine-grained temporal relationships (particularly in low-resource settings), as they lack inherent sequence modeling capabilities @xcite . incorporating bi-lstm allows the architecture to preserve these structural cues, while the attention mechanism adds the ability to dynamically highlight important words or phrases, boosting both accuracy and interpretability. this synergy has been demonstrated in several contexts. for instance, combining bi-lstm with bert has shown improved entity recognition in medical text due to better handling of position-sensitive features @xcite , and sentiment analysis in social media content benefits from this hybrid by capturing emotional nuances more effectively @xcite . attention-enhanced bi-lstm architectures also contribute significantly to explainability, an increasingly vital consideration in modern nlp systems @xcite . additionally, in scenarios such as microblog sentiment classification, this hybrid approach helps address overgeneralization issues commonly seen in transformer-only models @xcite . altogether, the combination of bi-lstm, attention, and transformer components provides a well-rounded solution that balances semantic depth, sequential awareness, and interpretability, making it a strong alternative to standalone transformer models. below is an overview of the system components and their functionalities: finally, predictions are binarized using a threshold, where values greater than the threshold are classified as positive labels. to evaluate the effectiveness of our proposed system, we trained multiple variants using different transformer-based models as feature extractors. our experimental setup aims to assess how the choice of transformer architecture influences multilingual emotion classification performance, specifically focusing on their impact on learned representations and downstream classification capabilities. in the following paragraphs, we describe our experiments' specific model configurations and dataset. we trained two model variants. the first leverages xlm-roberta (xlm-r) @xcite as the feature extractor for all languages in the dataset, while the second employs afriberta @xcite , specifically for african languages. for each, we compared the performance of the base and large configurations. we evaluate our models on a multilingual emotion classification dataset annotated for six emotion categories. each instance can have multiple emotion labels, making this a multi-label classification task. ## results the systems' performance analysis starts with the information presented in in some languages (yor, vmw, tir, som, orm, kin, ibo, and hau), the bow outperforms or is statistically equivalent to the large model, which may indicate insufficient data to fine-tune the mlm. when comparing the predictions from xlm-roberta-based models with those from afriberta-based models, we observe that xlm-roberta generally outperforms afriberta in most cases. this is particularly evident for amh, hau, ibo, orm, pcm, and som, where xlm-r-large scores higher than afriberta-large. however, there are exceptions where afriberta slightly surpasses xlm-roberta. this is the case for hau, where afriberta-large marginally outperforms xlm-r-large. for yor and kin, the base and large versions of afriberta outperform their xlm-r counterparts. it is worth noting, however, that swa performs worse across all afriberta versions. additionally, afriberta-based models fail to outperform the baseline scores, except for amh and hau. this suggests that our afriberta-based approach may not be well-suited for african languages, as it struggles to outperform the baseline consistently. as the results obtained by the xlm-r-large-based model were the highest performing, these model's predictions were submitted to the competition. ## conclusions this study evaluates the performance of transformer-based models for multilingual emotion classification, comparing xlm-roberta and afriberta across a diverse set of languages. several key takeaways emerge from our analysis. first, model scaling improves classification performance, with large variants (xlm-r-large and afriberta-large) consistently outperforming their base counterparts. xlm-roberta achieves higher f1 scores than afriberta in most cases, particularly for amh, hau, ibo, orm, pcm, and som, indicating that xlm-roberta's multilingual pretraining provides more robust representations. however, afriberta outperforms xlm-roberta for specific languages such as yor and kin, suggesting that afriberta's training data may better capture linguistic characteristics of certain african languages. interestingly, scaling does not always lead to improved performance, as some languages (ibo, yor, and vmw) experience a decrease in scores when moving from base to large models. this suggests that simply increasing model capacity does not necessarily enhance classification performance for all languages, possibly due to data sparsity or overfitting. among all evaluated models, xlm-r-large emerges as the best-performing approach for multilingual multi-label emotion detection, making it a strong candidate for robust and scalable nlp applications. despite its design for african languages, afrib-erta fails to outperform the baseline in multiple cases, with the exception of amh and hau. this raises concerns about its adequacy as feature extractor for our system, specially for underrepresented languages. furthermore, high-resource languages such as eng, esp, rus, hin, mar, and ron achieve significantly higher scores, benefiting from wellestablished pretraining data, whereas low-resource languages such as vmw, yor, swa, kin, som and sun exhibit weaker performance, likely due to limited resources. we also emphasize the importance of initiatives such as semeval in advancing emotion detection, affective computing, and broader nlp challenges. tasks like semeval-2025 task 11 provide highquality annotated datasets that enable the systematic development and evaluation of models across diverse languages and domains. by fostering both competition and collaboration, these shared tasks drive the advancement of more robust models and methodologies. additionally, they help uncover limitations in existing approaches, particularly for low-resource languages, and encourage research efforts toward more inclusive and generalizable nlp systems. our findings highlight the persistent challenges in modeling african languages for emotion classification. future work should explore domainadaptive pretraining, data augmentation techniques, and specialized architectures to improve multilingual and low-resource language performance