ITF - NLP at S em E val-2025 Task 11 An Exploration of E nglish and G erman Multi-label Emotion Detection using Fine-tuned Transformer Models (2025)

Abstract: We present our submission to Task 11, Bridging the Gap in Text-Based Emotion Detection, of the 19th International Workshop on Semantic Evaluation (SemEval) 2025. We participated in track A, multi-label emotion detection, in both German and English. Our approach is based on fine-tuning transformer models for each language, and our models achieve a Macro F1 of 0.75 and 0.62 for English and German respectively. Furthermore, we analyze the data available for training to gain insight into the model predictions.

## introduction the american psychological association defines emotion as "conscious mental reactions (such as anger or fear) subjectively experienced as strong feelings usually directed toward a specific object and typically accompanied by physiological and behavioral changes in the body" 1 . most research in emotion detection in natural language processing (nlp) is based on two different types of emotion theories. the first group of theories views emotions as universal categories (e.g. @xcite and @xcite ), and the second focuses on defining emotions in two or three dimensions, such as valance, arousal and dominance e.g. @xcite . furthermore, current research is exploring the possibility of multiple emotions being present in one sentence or utterance. task 11, bridging the gap in text-based emotion detection, at the international workshop on semantic evaluation (semeval), focuses on crosslingual and multi-label emotion detection @xcite . we participated in task a 'multi-label emotion detection' for the languages german and english. the aim of the task was to detect the emotions anger, fear, joy, sadness, and surprise for english, 1 https://www.apa.org/topics/emotions . and the german data also includes the class disgust. notably, a text can contain multiple emotions, thus takes into account the co-existence or even potential overlapping of emotions in one sentence. an example in english can be found below: based on the previous approaches to emotion shared tasks, we decide to focus on fine-tuning a transformers model for each language, english and german, and to use the results as a starting point to analyze the emotion labels in more detail. we chose to draw upon well-established discriminative transformer models instead of generative llms as our main focus is on advancing the understanding of decisions made by those commonly used baseline models. in general, we are interested in exploring the emotion classes, linking model performance to the data, and comparing the differences between the two languages. ## data the following chapter describes the data available for training in english and german that was provided by the task organizers. for both languages, the train/development/test instances stem from reddit and was annotated using language-specific human-annotators. a more detailed overview of the dataset, that contains human-annotated emotion data for 28 different languages, can be found in @xcite . the german data consists of 2803 annotated texts and was labeled for six different emotions: anger, disgust, fear, joy, sadness, and surprise. each text contains one or more labels, with an average of 1.17 emotion label per text, where 24.76% (@xmath0. 41.42% (@xmath1, 25.76% (@xmath2, 7.49% (@xmath3, 0.54% (@xmath4.04% (@xmath5. the english data contains 2884 annotated texts and was labeled for five different emotions: anger, fear, joy, sadness, and surprise. 8.74% (@xmath6. 41.16% (@xmath7, 37.21% (@xmath8, 10.82% (@xmath9, 2.01% (@xmath10.07% (@xmath11. figure 1 illustrates the class imbalance in both the english and german train datasets. for english, the largest class is fear (37%), and the smallest is anger (8%). contrastingly, in the german data, fear (8%) is one of the smallest classes and anger (25%) is the largest. ## system description during the development phase, a number of pretrained models obtained from hugging face's model repository foot_0 were tested with regard to their ability to solve the given task. model training/evaluation was implemented with the help of the simple transformers library @xcite and pytorch @xcite . we used a similar approach for both languages and tested various model combinations and train/development/test splits. in submission 1 in both languages, the models were fine-tuned on the training data provided by the organizers. the development data was used to evaluate the models. for the second and third submissions, the training and development data was reshuffled and split into a new a training, development, and held out test set (distribution 70/20/10%). additionally, weights were calculated for all classes and used during training to lessen the effects of the uneven class distribution for each language individually. the english models submitted during the test phase were based on deberta @xcite and roberta @xcite . the deberta model, used for submission 1, was trained using the training data provided by the organizers for 5 epochs with the following hyperparameters: training batch size: 32, learning rate: 2e-5, max length: 125. for submissions 2 and 3, a roberta model was fine-tuned using an 80/20 train and development split. the parameters are similar to the previous model, except training was conducted with a learning rate of 3e-5 and max length was set to 100. the difference between the two models in submission 2 and 3 is based on a different train split, providing the models with different data for fine-tuning. the best performing model for german, which was subsequently used for all submissions, was xlm-roberta-large-finetuned-conll03-german @xcite . as the name suggests, the model is based on xlm-roberta-large @xcite and was fine-tuned on a german dataset. hyperparameter testing resulted in the following optimal parameter combination: epochs: 5, learning rate: 3e-5, training batch size: 16. ## results for english, the best performing model achieved a macro f1 of 0.7501 (submission 3), and is also the submission on the final rank list. submission 1, the deberta model, did not perform as well as the other two roberta models. this is surprising considering it outperformed the other two models on our own test data. there was a much larger drop in performance between our own test set and the final set for this model compared to the two roberta models. interestingly, the model for submission 1 was trained without using weighting to balance out the uneven class distribution, as the performance on our own test set was similar with or without weighting. the weights were used for model training in submissions 2 and 3. this suggests that using weighing as a strategy to balance the classes contributed to the robustness of the models. for german, the macro f1 scores ranged from 0.6131 to 0.6231. a drop in macro f1-scores between the self-compiled test set and the test set provided by the organizers can be observed for all three submissions. the smallest difference is present in the scores achieved by model 3. while model 3 has the lowest macro f1-score on the selfcompiled test set, it seems to be the most robust when it comes to the prediction of previously unseen data. further analysis would be needed to evaluate if the greater difference between macro f1-scores for the first two submissions could be due to over-fitting. as well as looking at the overall performance, we also inspected the model's performance on the individual classes. the german models struggled to correctly predict the classes fear and surprise specifically. we showed in the data description that these classes are underrepresented in the german data. the weighting that was implemented during fine-tuning to lessen the effect of the unbalanced data distribution was not sufficient. the best performing classes, anger and joy, are also highly represented in the data. for english, a similar pattern of emotion class size and the model's ability to accurately predict the class can be observed. fear outperforms the other classes with an f1 of 0.8398, and anger is by far the smallest emotion class and also achieves the lowest f-score. ## analysis in this section we analyze the corpus data and link the results to possible performance issues in the models. we further explore specific emotion classes, namely anger and disgust in german and fear and sadness in english. ## conclusion we participated in the semeval 2025 shared task on text-based emotion detection. we participated in task a in english and german and our results are a macro f1 of 0.75 and 0.62 respectively. these are similar to those achieved by the organizers for their monolingual models @xcite . our approach was based on fine-tuning well-established transformers models and we optimized the model parameters for each language and experimented with different approaches to optimizing the training data. we found that the most effective strategy to improve both performance and robustness in the models for both languages was to balance the emotion classes in the data. in general, our analysis of the train and test data suggests, for both german and english, that there is a connection between class size and model performance for that specific class. we also demonstrated with our analysis the difficulty in classifying more closely related emotions, specifically anger and disgust in german, and fear and sadness in english. this seems to be related to their more frequent co-occurrence as annotated labels in the training data, which then also has an effect on how closely related the topics in each class are. future work includes expanding the emotion correlation analyses and applying our findings when balancing the dataset in pre-processing. when comparing the two languages, based on the most frequent words it seems as though a majority of the sentences in german data are related to politics, whereas in the english data the prevailing topic seems to be health. a larger dataset with more diverse topics might be helpful in ensuring robustness in future models. it would be interesting to explore the topics present in the data for the other languages in the shared task, and also see what role the topic clusters may play in cross-lingual emotion detection. in general, the definition of emotion already suggests that subjectivity plays a large role in correctly perceiving emotion, and multi-label annotations make the task of emotion detection even more challenging. the example below serves to illustrate the need for annotating multiple emotions in one sentence, as there is evidently an expression of both negative and positive emotion. however, due to the subjectivity of perceiving emotion, the need for all five emotion labels is debatable. we therefore acknowledge the difficulty of collecting and annotating emotion data.