Team K i A m S o at S em E val-2025 Task 11: A Comparison of Classification Models for Multi-label Emotion Detection (2025)

Abstract: The aim of this paper is to take on the challenge of multi-label emotion detection for a variety of languages as part of Track A in SemEval 2025 Task 11: Bridging the Gap in Text-Based Emotion Detection. We fine-tune different pre-trained mono- and multilingual language models and compare their performance on multi-label emotion detection on a variety of high-resource and low-resource languages. Overall, we find that monolingual models tend to perform better, but for low-resource languages that do not have state-of-the-art pre-trained language models, multilingual models can achieve comparable results.

## introduction interlocutors rarely speak in an entirely neutral manner: more often than not, speakers will use emotions in their speech. emotions are an important driving force of conversations and understanding how language and emotions interact is crucial for linguistics. in nlp, the field of emotion recognition is concerned with identifying the emotions of the speaker of an utterance. @xcite defines six basic emotional states: joy, sadness, fear, anger, surprise, and disgust. these have since been used widely in emotion recognition to assign the perceived emotion of a speaker during an utterance. while many systems have been developed that can assign one singular emotion to a text, the challenge of multi-label emotion detection is a newer and less investigated field. nonetheless, it is an important area, since speakers rarely feel emotions in isolation and multiple emotions often occur in conjunction with each other, for example anger and disgust. track a of semeval 2025 task 11, "bridging the gap in text-based emotion detection" @xcite , aims at solving the issue of multi-label emotion detection for 28 different languages @xcite , including many low-resource languages. to solve this task, our team compares different pre-trained language models on their ability to perform multi-label emotion classification, comparing monolingual models like gottbert @xcite and twitter-roberta @xcite to multilingual models such as xlm-t @xcite . our aim is to see how multilingual models perform when they are fine-tuned solely on one language versus multiple languages simultaneously. this could provide important insights into maximizing the usability of multilingual models for low-resource languages. we further employ taskadaptive pre-training and optimized classification thresholds at each epoch to improve performance. ## background early work in nlp largely focused on sentiment analysis, the classification of a text into negative or positive valence classes @xcite . in contrast, emotion recognition deals with assigning texts to distinct emotion classes. sentiment analysis is often a case of binary classification, assigning either positive or negative valence to a text. emotion recognition is often implemented as a multi-class classification problem, selecting the most salient emotion out of multiple emotion classes. however, a multi-class approach neglects the co-occurrence of emotions that cannot be separated from each other @xcite . this type of relation requires a multi-label strategy. furthermore, most existing multi-label emotion classifiers focus on high-resource, predominantly indo-european languages such as english, with fewer systems available for low-resource languages. earlier approaches to multi-label emotion recognition employed classifier chains to account for the correlation between the different emotions. for instance, participants of semeval 2018 task 1 com-bined their best performing classifier for every emotion into a chain which passes the predicted labels to the next classifier in the chain, sorting the classifiers by performance (highest to lowest). their best performing classifier chain achieved a macroaveraged f1 score of 0.493 @xcite . more recent approaches involve latent emotion memory networks, which aim at learning the latent emotion distribution and emotion intensity in a text and leverage it into a classification system. these consist of a variational auto-encoder that learns the emotion from the input and a memory unit that captures the most salient features for that emotion. on the semeval 2018 dataset, they achieved a macro f1 score of 0.567 @xcite . other systems include the sequence-to-emotion (seq2emo) approach, which is essentially a sequence-to-sequence model that encodes the utterance using an lstm and then uses an lstmbased decoder to perform binary classification on the emotions sequentially. this approach achieved a macro f1 score of 0.5192 on the semeval 2018 dataset @xcite . since then, the rise of pre-trained language models and transformer-based architectures has opened up a variety of new ways to approach multi-label emotion detection. however, it is still unclear which pre-trained models are well suited for emotion detection, and how to best fine-tune models for this task. a further open question is how to build multilingual models that can perform emotion detection in a variety of languages. this is the aim of our approach: we compare several monolingual and multilingual pre-trained language models and fine-tune them for emotion classification, comparing the pre-trained models to a logistic regression baseline. the following sections will explore the different systems that we have tried and their respective results. ## system overview for our system, we mainly rely on the xlm-twitter (xlm-t) base model for sequence classification @xcite , which continues pretraining from a publicly available xlm-r checkpoint @xcite using nearly 200m tweets from over 30 languages. we then apply different fine-tuning strategies and observe the effects on model performance. additionally, we contrast the performance of a multilingual model like xlm-t with specialized monolingual models for german and english. due to time and resource constraints, we only analyze a subset of 10 languages that includes both high-resource and low-resource languages: afrikaans, amharic foot_0 , algerian arabic, moroccan arabic, mandarin chinese, german, english, spanish, hausa, and hindi. we use the training, development, and test datasets provided by the semeval2025 task 11 organizers @xcite . ## experimental setup in this section, we provide details on our considerations about the data and training of the models. ## results overall, we were able to outperform the semeval baseline in 7 out of 10 submitted languages, only for amharic, german, and hindi we were not able to achieve a score above the baseline. with the exception of chinese, we could not replicate previous findings showing that applying task-adaptive pre-training significantly increases model performance. in fact, its performance for afrikaans and hausa is quite weak in comparison. however, this system still achieves competitive results for the majority of the languages. we suppose that due to our limited resources, we were not able to fully tap into the potential of tapt, as the pretraining process was aborted after 3 epochs, which is not nearly enough time for the model to converge. this exactly might have been the issue with afrikaans and hausa, which are also the only two languages in our set not present in the top 30 languages xlm-t was originally trained on @xcite . our resource limitations for applying taskadaptive pre-training are an example for the tradeoff between performance and resource use that researchers in this field are continuously faced with. on that note, it is interesting to remark that in our experiments, logistic regression slightly outperforms xlm-t for hausa. we do not have a solid hypothesis for this, especially since calculating the scumble score @xcite for our 10 languages suggests that hausa, along with chinese, has the highest label concurrency (minority labels occurring mostly or only together with majority labels), which should make it especially difficult to get accurate classification results for their minority labels. when comparing the performance of joint multilingual and monolingual fine-tuning, there seems to be no clear winner at first. taking into account whether the target language is low-resource or highresource however, there seems to be a tendency for low-resource languages to prefer multilingual training. afrikaans, and especially amharic, seem to benefit from the additional information present in the other language data with an increase of the f1 score from 0.53 to 0.6 for the latter. conversely, high-resource languages mostly perform better with monolingual training. this is especially highlighted when comparing the language-specific models gottbert and twitter-roberta with the multilingual xlm-t fine-tuned on german or english data. both fully monolingual models outperform the multilingual one. for completeness, it would be interesting to compare the performance of xlm-t with tapt, finetuned for each language data individually, with our jointly fine-tuned tapt-applied xlm-t. it remains an open question whether with our setup we would reach a similar conclusion as @xcite , where the advantages of monolingual training become less pronounced in the presence of task-adaptive pre-training. ## conclusion overall, our systems aimed at comparing the performance of mono-and multilingual pre-trained language models for multi-label emotion recognition. we find that when the necessary resources are available, a specialized monolingual approach outperforms a generalized multilingual one. emotion recognition for high resource languages like german and english works best without the interference of other languages. nonetheless, the strength of multilingual models lies in their versatility and their ability to leverage information from higher-resource languages to make inferences about lower-resource languages. as such, multilingual models allow us to tackle tasks with low-resource languages where a specialized approach is simply not feasible. our example of chinese shows that classifiers can strongly benefit from being fine-tuned on a set of languages they are not even a part of. identifying those source languages that are especially useful for improving classification performance in a target language is a task that researchers tackle in the field of zero-shot classification @xcite , which was also the focus of track c in this semeval task.