OZ emi at S em E val-2025 Task 11: Multilingual Emotion Detection and Intensity (2025)

Abstract: This paper presents the OZemi team’s submission to SemEval-2025 Task 11: Multilingual Emotion Detection and Intensity. Our approach prioritized computational efficiency, leveraging lightweight models that achieved competitive results even for low-resource languages. We addressed data imbalance through data augmentation techniques such as back translation and class balancing. Our system utilized multilingual BERT and machine translation to enhance performance across 35 languages. Despite ranking mid-tier overall, our results demonstrate that relatively simple models can yield adequate performance across diverse linguistic settings. We provide an error analysis of emotion classification challenges, particularly for nuanced expressions such as sarcasm and irony, and discuss the impact of emoji representation on model predictions. Finally, we outline future directions, including improvements in sentiment intensity modeling and the integration of semantic prosody to refine emotion detection.

## introduction this paper explores semeval 2025 task 11 @xcite , which focuses on multi-label emotion detection and emotion intensity across various languages based on the datasets provided by the task organizers @xcite . the task is divided into three tracks. track a involves predicting the presence of emotion(s) such as joy, sadness, anger, surprise, and disgust in text snippets. each emotion is labeled in a binary format: (1) if it is present, and (0) if absent. task b focuses on predicting the intensity of a perceived emotion on a scale of 0 (no emotion) to 3 (high intensity of emotion). finally, track c is about using a trained dataset in one language to predict emotion labels in a different language. the datasets cover 35 languages in total, with genres ranging from social media to conversational text. emotions are at the core of human interactions but are notoriously difficult to detect in text @xcite . any technical and theoretical advancements have the potential to aid in customer service automation, online content moderation, and many other tasks -both academic and commercial. the focus on cross-lingual emotion detection assists the recognition of emotions on a global scale, increasing its relevance among different cultural contexts. our team ranked around the middle for all tasks and languages. our approach is not the most technically advanced, but because of this it is also not computationally very intensive. we managed to show that it is possible to achieve adequate results even for low-resource languages with very little computational resources. our code is available on github foot_0 . ## background and previous work the input for this task is text snippets in multiple languages ranging from commonly spoken languages such as english, german, and spanish, to less commonly spoken languages such as emakhuwa. the output differs across various tracks. the objectives of each track are demonstrated using the sentence "i just won the lottery!" as an example. track a (multi-label emotion detection): the output consists of binary labels for each perceived emotion, where (1) indicates its presence and (0) an absence. the output would look something like: joy: 1, sadness: 0, fear: 0, surprise: 1, disgust: 0, anger: 0 for some languages (such as english), the set of perceived emotions does not include disgust. track b (emotion intensity): the output consists of an intensity prediction for each perceived emotion. the output would look something like: joy: 3, sadness: 0, fear: 0, surprise: 2, disgust: 0 the degrees of intensity range from 0 to 3, with 0 indicating no emotion, and 3 indicating a high intensity of emotion. the above output example for the sample sentence indicates a high intensity of joy and moderate intensity for surprise. track c(cross-lingual emotion detection): involves predicting emotion labels for a language using a labeled training set in a different language. track b also has additional challenges, not only does the emotion need to be accurately categorized, but labeled with intensity as well. as @xcite state, rating scales used as annotations for sentiment analysis suffer from various flaws. they can be inconsistent, with gaps in value between annotators despite agreement on general sentiment, biased towards certain parts of the scale, or suffer from either too little or too much granularity. additionally, further difficulty can emerge from methods of data pre-processing or lemmatization that may make data easier to categorize when using traditional methods. exclamation marks, capitalization and more lend context to emotion intensity but may cause difficulty in processing. in addition, sarcasm and irony, prevalent in many common sources of data such as twitter and other social media can also increase the difficulty of categorization and intensity mapping. however, emotion intensity is an important measure that has been shown to correspond well with human interpretations of a text's overall emotional content @xcite . ## system overview and experimental setup in the research on the english model for track a, we handled emojis by using the demojize function of the emoji python library to convert emojis into descriptive textual labels @xcite . track a and c use the f-score, and tack b pearson correlation for evaluating the models. for both track a and track b, the training data was split into two groups: training and testing sets. the preparation of the dataset involved data cleaning process to ensure the text inputs were uniform and to avoid unnecessary characters. this included replacing or removing special characters and standardizing representations for symbols such as quotes. ## results we benchmarked our results against more simple models not fine-tuned on the specific instructions for this task. these benchmark models serve to provide points of reference to evaluate our more complex system. our system proves its effectiveness and added value by achieving higher scores for this specific task. one benchmark model uses the nrc lexicon (mohammad and turney, 2013) as a simple rulebased approach that relies on a predefined set of words labeled with plutchik's 8 core emotions @xcite , matching words to emotions without context or taking negations and valence-shifters into account. as this approach is the simplest, we expect our model that takes context into account to outperform it. the other benchmark model was the cardiffnlp roberta base sentiment multi-label model finetuned for semeval 2018 task 1 @xcite . this model is pre-trained on a large corpus of tweets and captures contextual word representations. however, it has not been fine-tuned for this specific task in this evaluation. its performance already shows a significant improvement over the nrc lexicon due to its ability to understand the semantics of language at a deeper level, showing us that our system can benefit from understanding the nuances specific to this task. ## limitations one limitation we identified is the inaccuracies in the training data tags provided. for example, although the sentence "but not very happy" does not have the sentiment of joy, the training data had it labeled as as "@xmath0." by fine-tuning a model to produce high accuracy scores with respect to an inaccurately tagged dataset, this model, or models produced for this task, may only be accurate for this task, but not when solving other real-world problems. additionally, the provided english dataset for track a does not contain any emojis, which limited the opportunity to directly study the impact of emojis on emotion detection within the english language dataset. our simplistic approach to implement google translate as a machine translation technique might have had a slight influence on our inference. as stated by @xcite , the original sentence and the sentence translated by google translate may differ in terms of semantic relations. therefore, it is likely that some input sentences were semantically changed when its translation, and hence had a negative influence on the performance. alternative ways including use of betterperforming machine translation models were considered, but we chose the model on the basis of costs, assuming that the possible influence discussed above would not be significant compared to other factors. ## future work while brainstorming, we explored the application of semantic prosody @xcite to the task. while not included in the final model, interesting trends were found that can be included in the refining of future models. introduced by @xcite , a word's semantic prosody refers to its tendency to co-occur with specific sentiments or emotions. for example, the words "bring about" and "cause" have similar semantics, in terms of how they both speak about the reason behind an occurrence. yet, "bring about" is more likely to co-occur with positive occurrences, while "cause" is more likely to co-occur with negative occurrences (mcgee 2012; see figures 1 and 2 for collocates of "bring about" and "cause" respectively, analyzed using antconc (anthony, 2024)). therefore, "bring about"'s semantic prosody can be viewed as positive, and "cause"'s negative. in line with this concept, we hypothesized that certain n-grams may occur more frequently with certain emotion tags than others. hence, we ran an analysis for the most frequent 1-gram and 2-grams figure 2: kwic of "cause" for each emotion tag, stratified by each parts-ofspeech tag. we found that there were unique ngrams for each emotion tag, occurring at a frequency of more than 1. it might thus be valuable to run this analysis with larger corpora, to find unique n-grams that co-occur with an emotion at a statistically significant frequency. these n-grams could then be used to further refine sentiment analysis models, especially multi-label ones that have relatively lower accuracy scores. ## conclusions throughout prior discussions in this paper, it can be suggested that our straightforward approach under limited compute resources performs well even for low-resource languages. we have succeeded to maximize the benefits of lightweight models with experiments such as the use of back translation and hyperparameter tuning. additionally, we have combined that with machine translation techniques and also leveraged both multi-lingual and languagespecific models over some non-english languages as well. in those ways, we firstly established our basis on emotion detection in english sentences, and then, applied the methodology to various languages. although we had limited compute resources, this straightforward approach was shown to work well and be relatively competitive as well. hence, our future work could also include some improvements in the same perspective, opening the door for its wider application to emotion detection in various languages. john sinclair. 1996. the search for units of meaning. textus, 9(1):75-106. hidetsune takahashi, xingru lu, sean ishijima, deokgyu seo, yongju kim, sehoon park, min song, kathylene marante, keitaro-luke iso, hirotaka tokura, et al. 2024. ozemi at semeval-2024 task 1: a simplistic approach to textual relatedness evaluation using transformers and machine translation. in proceedings of the 18th international workshop on semantic evaluation (semeval-2024), pages 7-12.