Team UBD at S em E val-2025 Task 11: Balancing Class and Task Importance for Emotion Detection (2025)

Abstract: This article presents the systems used by Team UBD in Task 11 of SemEval-2025. We participated in all three sub-tasks, namely Emotion Detection, Emotion Intensity Estimation and Cross-Lingual Emotion Detection. In our solutions we make use of publicly available Language Models (LMs) already fine-tuned for the Emotion Detection task, as well as open-sourced models for Neural Machine Translation (NMT). We robustly adapt the existing LMs to the new data distribution, balance the importance of all emotions and classes and also use a custom sampling scheme.We present fine-grained results in all sub-tasks and analyze multiple possible sources for errors for the Cross-Lingual Emotion Detection sub-task.

## introduction in this project, we address the three sub-tasks (tracks) of emotion detection (ed), emotion intensity estimation (eie), and cross-lingual emotion detection (cl-ed) from semeval-2025 task11 @xcite . while the organizers provided a dataset with samples from 28 different languages @xcite , we only focused on english, german and spanish for the first two sub-tasks and romanian, portuguese, ukrainian, russian, hindi and indonesian in the last one. our solutions mainly rely on language-specific encoders that have already been fine-tuned for the emotion detection task and robustly adapt them to the new data distribution. to bridge the gap between languages in the last task we use an opensource nmt system to translate the test sets of other languages, and also experiment with crosslingual lms @xcite . we find that the emotion-specific performance of our systems correlates well with the frequency of positive examples in the first two sub-tasks. this indicates that data scarcity can still be a problem, even when it is addressed through common means. for the cl-ed sub-task we find multiple factors that lead to degraded performance on new languages, some of which are system-specific, while others are common to both of them. notably, the relatedness of languages does not correlate well with cross-lingual performance in our experiments. the implementation of our solutions will be published on github foot_0 . ## background related work the task of emotion detection has found diverse applications @xcite , such as analyzing social interactions, monitoring mental well-being @xcite , highlighting mental health concerns or understanding people's emotions during stressful events @xcite . while earlier works have tried to use a mix of low-level features and more abstract ones, obtained from deep neural networks @xcite , the transformers architecture @xcite has become the default architecture for this task @xcite in recent years, with people using even specialized pre-training objectives @xcite to improve results in this downstream task. for the cross-lingual variant of the task, multiple solutions have been proposed. some notable ones are: using multilingual encoders and training detectors on language agnostic representations of the texts @xcite @xcite , distilling monolingual detectors into cross-lingual models @xcite , translating texts into a language where annotated training data is available @xcite and even using large language models as zero-shot detectors @xcite . ## system overview in our solutions we use language-specific encoders, based on the transformer @xcite architecture, that have already been fine-tuned for the emotion detection task, in order to extract deep representations of the textual samples. we then add a linear classification layer to the encoders and train them with dynamic weights to balance the importance of each class and emotion. ## experimental setup in all sub-tasks we use only the data provided by the task organizers, without applying data augmentations or any pre-processing before tokenizing the texts. we used the data splits provided by the organizers (train/dev/test), with a single exception in the experiments based on cross-lingual models, where 15% of the train data is used for validation and the dev split is used for testing. all classifiers were trained using the ## results track a the results of the linear probes on the test set of the competition are presented in the frequency of positive samples alone is not a good indicator for the final performance. while joy and sadness have similar frequencies in german data, there is a 10% gap in f 1 score between them in the linear probing scenario ( for certain emotions, the initial linear probes performed better than those trained on the finetuned encoders. thus, we decided to select for each emotion the test set predictions from the linear probe that had the highest f 1 score on the dev set. the results of this final submission are presented in detail in track b the test set results for the previously described experiments in this sub-task are presented in track c the results of our system for this track are presented in in the described framework we have identified multiple sources of errors. the first one is the quality of the translations -we validated that in certain cases the nmt system started repeating a single token multiple times. nonetheless, as almost all languages have at least one emotion with an f 1 higher than 60% (ukrainian is the sole exception), we expect this type of errors to be limited. another possibility is for low-level cues for the perceived emotions (e.g. punctuation) to be lost in the process or for the choice of words to be unusual due to translationese @xcite . we ex-pect these problems to be correlated to the data distribution used for training the nmt model. a general source of errors is the distributional shift between languages, regarding the topics they covered. we manually determined that texts in romanian seemed to be mainly scraped from news websites and covered topics like politics and the recent covid-19 pandemic, whereas the texts in english contained mostly short texts that were likely written on social media websites. these particularities might bias the detectors towards detecting certain topics, not the emotions themselves, resulting in degraded performance in other contexts. through our experiments in the dev period with cross-lingual models we have noticed that they also suffer a great performance degradation on new languages (consult appx. a for the results). we provide in fig. 2 a t-sne visualization of text embeddings extracted with the lealla-large and qwen2.5-7b models. we observed that the data tends to form language-specific clusters, which we assume to be the main reason for the poor generalization of the trained detectors to new languages. this embeddings space structure can be caused both by the topic changes between languages, and the language specificity of the embeddings. in order to quantify the impact of the two factors mentioned above one would require high quality translations pairs, as well as topic annotated samples. we leave this detailed study as future work and only investigate in appendix a the text pairs translated with the nllb model in track c. while no clear conclusion can be reached, we reason based on the observed evidence that severe translations errors are surely present. jinghui zhang, yuan zhao, siqin zhang, ruijing zhao, and siyu bao. 2024. enhancing cross-lingual emotion detection with data augmentation and token-label mapping. in proceedings of the 14th workshop on computational approaches to subjectivity, sentiment, & social media analysis, pages 528-533, bangkok, thailand. association for computational linguistics. a cross-lingual models t-sne visualization regarding the t-sne plots from figure 2 , we want to highlight that the tanh activation of the pooler layer in the lealla-large model is likely the main reasons why all the embeddings are clumped together more tightly than the embeddings of the qwen model.