AGHNA at S em E val-2025 Task 11: Predicting Emotion and Its Intensity within a Text with E mo BERT a (2025)

Abstract: This paper presents our system that have been developed for SemEval-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection. The system is able to do two sub-tasks: Track A, related to detecting emotion(s) in a given text; Track B, related to calculate intensity of emotion(s) in a given text. The system will have EmoBERTa as the model baseline, despite some minor differences used in the system approach between these tracks. With the system designed above, Track A achieved a Macro-F1 Score of 0.7372, while Track B achieved Average Pearson r Score of 0.7618.

## introduction semeval-2025 is the 19th edition of semeval. semeval-2025 presents 11 different tasks, one of which is the task titled as "semeval-2025 task 11: bridging the gap in text-based emotion detection". this task focuses on text-based emotion recognition. in the repository provided, there are 3 sub-tasks that can be done, which will be referred to as "tracks", namely: there are several applications on text-based emotion detection (tbed) in the modern world. first, tbed can help to detect or diagnose a user's mental health through their posts on social media @xcite . second, integrating tbed into an ai system allows for better understanding and interaction between the ai or computers and humans @xcite . and the last example being its integration to business and finances allows data analysts to understand customer reviews more efficiently @xcite . in this paper, we propose a system named aghna (automated generalized human-emotion detection with a neural approach). aghna utilizes emoberta as the base model for both tasks mentioned before @xcite . emoberta is a model developed or fine-tuned from roberta to achieve better results specifically in emotion recognition in conversation (erc) tasks. emoberta is able to generate better performance compared to other erc models, such as dialogxl and cesta @xcite . to further enhance emoberta's performance for both tasks, aghna incorporates several additional approaches to the system. these approaches are: experimenting different feature extraction methods, applying optimization with adamw, using binary cross entropy (for track a) and mean squared error (for track b), and other approaches that will be elaborated even further throughout the paper. ## related works related work in the field of tbed has produced various new methods and approaches, but several aspects are still not perfect. the biggest challenge is the high computing resources required due to the complexity of the models being built and the large amount of data that must be trained before testing. one of the related studies is a research on creating an annotated corpus for bangla multi-label emotion detection @xcite by collecting data in the form of comments totaling 136,583 data taken from 11 different news stories on facebook. the implemented approach uses feature extraction methods such as tokenization and tf-idf. after that, various methods were applied, using machine learning (ml) algorithm (logistic regression, random forest, multinomial naive bayes, support vector machine, and k-nearest neighbors), deep learning (dl) algorithm (lstm, bil-stm, and hybrid cnn-bilstm and cnn-lstm), and transformer-based algorithms (banglabert, mbert, bangla-bert-base, and bangla-electra). the results are as follows: there are several advantages offered from the results of this research. these advantages include monovab's contribution to providing a tbed in bangla which was previously minimal, the use of various approaches (ml, dl, and transformers) to obtain the most optimal results, as well as an annotation process using a context-based approach. however, there are also limitations in this research. first, the high complexity due to the implementation of various approaches requires high computing resources. second, there are data that can't be adapted to the corpus because the data cannot be processed using a context-based approach, suggesting for a more suitable lexical-based approach. another related research discuss about emotion prediction in text and multi-turn conversations by combining advanced nlp, transformers-based networks, and linguistic methodologies @xcite which was carried out based on tasks from "wassa 2022 shared task: predicting empathy, emotion and personality in reaction to news stories" and "wassa 2023 shared task: empathy, emotion and personality detection in conversation and reactions to news articles", both of which are related to emotion prediction. the dataset used comes from wassa 2022 and wassa 2023, with the statistics provided in the the approach taken in this research involves using a feedforward neural network (ffnn) with relu activation and pytorch, experimenting with various embedding models as input to the neural network, hyperparameter tuning, overcoming data imbalances, utilizing lexicon features, and an ensemble method using two svr models to model the relationship between text features and emotions. the final results of the study showed an average score increase of 33.59% over the baseline for wassa 2022 and 64.02% over the baseline for wassa 2023. there are several advantages obtained from the results of this research. these advantages include the use of transformers that are integrated with various linguistic features and ensemble methods that can mitigate bias by combining multiple predictions. however, there are also drawbacks in this research. the most noticeable drawback in this research is the high complexity of the model due to the combination of various approaches that requires high computational resources. ## dataset this research will use the dataset provided by the organizers of this semeval task. there will be an equal amount of data and texts given for both track a and track b in the english language, with the statistics provided in split # of rows train 1,860 dev 525 test 270 since the task involves a multi-label dataset, there are multiple instances where a sentence may have more than one detected emotion, either in the train dataset or as the result of system's predictions. after a quick analysis, each emotion's frequency in the train dataset are provided in there is a noticeable discrepancy in terms of emotion frequency within the given dataset. for example, the emotion fear appears in 1,611 different texts, whereas the emotion anger appears in only 333 different texts, approximately five times less than fear. this, in return, causes data imbalance and may lead into biases in the system's predictions. ## benchmark the benchmark used for evaluating the performance of the proposed system in this research is based on the official baseline scores provided by the task organizers. these baseline scores are derived from the organizers' own research effors related to the task and serves as the reference for the evaluation of our system's final performance. the baseline system utilizes the rembert model, a model that can be used for multiple tasks including text classification, the main topic of this research. in detail, the baseline has a macro-f1 score of 0.7083 for track a, and an average pearson correlation coefficient (r) score of 0.6415 for track b @xcite . ## system overview although several adjustments are required to handle track a and track b separately, it is important to note that, due to the similar nature of processing for both tracks (making predictions on given texts), figure 1: aghna's architecture design the overall system design remains similar to that shown in figure 1 . the system designed for this task will be based on emoberta. unlike most existing works on erc that combine different kinds of neural network architectures, and therefore deemed too complex, emoberta simply utilizes the existing roberta model while encoding the speaker's information along with multiple utterances. emoberta demonstrated very good results when tested with meld (multimodal emotion-lines dataset) and iemocap (interactive emotional dyadic motion capture) datasets outperforming several erc models. on meld, emoberta achieved a weighted f1-score of 65.61%, slightly better than the next model, cosmic @xcite with 65.21%. and, on iemo-cap, emoberta achieved a weighted f1-score of 68.57%, slightly better than the next model, cesta @xcite with 67.1% @xcite . after understanding the beneficial performance of emoberta for this task, we aim to improve its performance by implementing additional methods into our final system. the additional methods will focused on hyperparameter tuning in several areas, such as the number of epoch, batch size, learning rate, etc. ## result for the training dataset, we combined 2,768 data from the provided training dataset with an additional 116 data from the dev dataset. the dev dataset were included because they had been assigned gold labels by the organizers, meaning they had been manually reviewed and correctly labeled. giving a total number of 2,884 data prepared to be trained before the system starts to make predictions into the test dataset. during the training process, we conducted several experiments to fine-tune the system's performance. after analyzing the results, we identified and collected the optimal combination of hyperparameters that may yielded the best results for the system. the hyperparameters' values are provided in hyperparameter value seed 42 model tae898/emobertalarge max. sequence length 128 batch size 16 epoch 3 learning rate 4e-5 warmup ratio 0.1 gradient clipping @xmath0, 3) min_df 2 max_df 0.95 alpha 0.5 gamma 2 weight based on class imbalance feature extraction sentiment_polarity sentiment_subjectivity text_length word_count uppercase_ratio exclamation_count question_count to minimize the system's execution time, we opted to ran the system for both tracks using nvidia a100 gpu on google colab, as it is the fastest accelerator compared to other accelerators in google colab. after running the system for both tracks independently, we also run the system for two other different models with the intention of model comparison: bert (bert-large-uncased) and roberta (roberta-large). the system's final results/performance submitted to codabench for the semeval competition, alongside comparisons to other models, are presented in tables 5 and 6 . for the purpose of ranking and evaluation from the organizers, macro-f1's score will be used as the main metric for track a, while average pearson correlation coefficient (r)'s score will be used for track b. in track a, our system aghna achieved a macro-f1 score of 73.72%, placing 38th out of 97 participants, placing it within top 40% of all submissions. this result represents a small improvement of 4.08% over the baseline score of 70.83% provided by the task organizers. meanwhile, for track b, aghna achieved an average pearson correlation coefficient (r) score of 76.18%, placing 11th out of 43 participants, placing it within top 26% of all submissions. this result represents a major improvement of 18.75% over the baseline score of 64.15% provided by the task organizers. from data imbalance remains a significant challenge in track a, provided by the notable difference in f1-score between the emotions anger and fear. the gap between these two emotions is 11.93%, indicating several emotions are more accurately predicted than others. such difference suggests the model struggles to generalize across every emotions in the dataset, most likely due to uneven distribution of emotions within the given dataset. although with such problems faced in track a, track b doesn't seem to suffer as much, with the lowest and highest pearson correlation coefficient (r) score, performed by surprise and joy respectively, differs by only 8.69%. this statistics further highlight track b's overall success while also giving a massive understanding the need of improvement for track a. due to the time constraints given by the task organizers, our research was unable to implement several track b's methods to track a. although, by how successful the result for track b is compared to track a, we have integrated several methods from track b (such as the implementation of textblob) into track a after the semeval test phase ended, and we plan to add more features for both tracks in the future. ## conclusion as seen in the results and ranking statistics from the previous chapter, aghna demonstrates strong capabilities in predicting emotions, both in binary