S em E val-2025 Task 11: Bridging the Gap in Text-Based Emotion Detection (2025)

Abstract: We present our shared task on text-based emotion detection, covering more than 30 languages from seven distinct language families. These languages are predominantly low-resource and spoken across various continents. The data instances are multi-labeled into six emotional classes, with additional datasets in 11 languages annotated for emotion intensity. Participants were asked to predict labels in three tracks: (a) emotion labels in monolingual settings, (b) emotion intensity scores, and (c) emotion labels in cross-lingual settings.

## introduction people use language in diverse and sophisticated ways to express emotions across languages and cultures @xcite @xcite . emotions are also perceived subjectively, even within the same culture or social group. recognising these emotions is central to language technologies and nlp applications in healthcare, digital humanities, dialogue systems, and beyond @xcite . in this work, we use emotion recognition to refer to perceived emotions, i.e., the emotion most people believe the speaker * equal contribution might have felt based on a sentence or short text snippet. despite the linguistic diversity of regions such as africa and asia, which together account for more than 4,000 languages, few emotion recognition resources exist for these languages. prior semeval shared tasks on emotion recognition have primarily focused on high-resource languages such as english, spanish, and arabic @xcite @xcite . in this task, we provide participants with new datasets covering more than 30 languages from seven distinct language families, spoken across africa, asia, latin america, north america, and europe @xcite . our manually annotated emotion recognition datasets, curated in collaboration with local communities, consist of over 100,000 multi-labeled instances drawn from diverse sources, including speeches, social media, news, literature, and reviews. each instance is labeled by fluent speakers and annotated with six emotion classes: joy, sadness, anger, fear, surprise, disgust, and neutral. additionally, eleven datasets include four emotional intensity levels ranging from 0 to 3 (i.e., absence of emotion to high intensity). the task consists of three tracks: (a) multilabel emotion detection, (b) emotion intensity detection, and (c) cross-lingual emotion detection. the languages for each track are listed in figure 1 . each team could submit results for one, two, or all three tracks in one or more languages. our official evaluation metrics were the average f-score for tracks a and c and the pearson correlation coefficient for track b, which measures how well systempredicted intensity scores align with human judgments. our task attracted over 700 participants, with 220 final submissions and 93 teams submitting system description papers. track a (multi-label emotion detection) received the most submissions (114), followed by track c (cross-lingual emotion detection) with 51, and track b (emotion intensity detection) with 32. most teams participated in multiple languages, averaging 11 languages per team. our task was the most popular competition on cod-abench in 2024.all task details and resources are available on the task's github page. additionally, most prior emotion recognition shared tasks have focused on high-resource languages such as english, spanish, german, and arabic @xcite @xcite . in contrast, this shared task covers more than 30 languages, including several low-resource languages. ## task description participants were given text snippets and asked to determine the emotions that people may attribute to the speaker based on a sentence or short text snippet uttered by the speaker. the task consists of three tracks, and participants could participate in one or more of these tracks. ## discussion popular methods unsurprisingly, most topperforming teams favored fine-tuning and prompting large language models (llms) such as gemma-2, mistral, phi-4, qwen-2.5, deepseek, llama-3, gpt, and gemini models. for fine-tuning, both full fine-tuning and parameter-efficient fine-tuning were the most commonly used strategies to enhance performance. for prompting, few-shot, zero-shot, and chainof-thought prompting were the most frequently used techniques. many participants also experimented with traditional transformer-based models, particularly xlm-roberta, mbert, deberta, and in-dicbert @xcite (see figure 3 and ?? in the appendix). ## conclusion we presented our shared task on text-based emotion recognition, which covered three tracks and a total of 32 languages. the submitted systems were ranked based on macro f1-scores for tracks a and c, comparing predicted labels to gold labels, and based on the ranking of predicted intensity scores for track b. we summarised the reported results, discussing the best-performing and most innovative methods. overall, performance varied significantly across languages. our results highlight that emotion recognition remains an open challenge, particularly for under-served languages and in low-resource settings.