UT - NLP at S em E val-2025 Task 11: Evaluating Zero-Shot Capability of GPT -4o mini on Emotion Recognition via Role-Play and Contrastive Judging (2025)

Abstract: Emotion recognition in text is crucial in natural language processing but challenging in multilingual settings due to varying cultural and linguistic cues. In this study, we assess the zero-shot capability of GPT-4o Mini, a cost-efficient small-scale LLM, for multilingual emotion detection. Since small LLMs tend to perform better with task decomposition, we introduce a two-step approach: (1) Role-Play Rewriting, where the model minimally rewrites the input sentence to reflect different emotional tones, and (2) Contrastive Judging, where the original sentence is compared against these rewrites to determine the most suitable emotion label. Our approach requires no labeled data for fine-tuning or few-shot in-context learning, enabling a plug-and-play solution that can seamlessly integrate with any LLM. Results show promising performance, particularly in low-resource languages, though with a performance gap between high- and low-resource settings. These findings highlight how task decomposition techniques like role-play and contrastive judging can enhance small LLMs’ zero-shot capabilities for real-world, data-scarce scenarios.

## introduction semeval-2025 task 11 @xcite addresses emotion recognition in text across multiple languages, ranging from high-resource to lowresource languages, which is a crucial area in nlp with far-reaching applications in social media analytics, customer service, and healthcare. by providing a multilingual, multi-labeled dataset of 28 languages, the task highlights the challenges of building robust emotion detection systems under limited training data conditions. in this paper, we describe our team's participation in semeval-2025 task 11, specifically in: our primary goal was to evaluate the zero-shot capability of a small-sized llm, gpt-4o mini, which is a more cost-efficient model in the gpt-4o family @xcite . as llms have demonstrated impressive zero-shot capabilities in recent years @xcite , we did not finetune the model on any task-specific data but instead introduced a zero-shot approach: role-play and contrastive judging, illustrated in figure 1 . 1. role-play rewriting: prompt the model to rewrite a given sentence as if it inherently conveyed a target emotion-altering only minimal surface details while preserving meaning. we hypothesize that rewriting the text to inject various emotional tones helps the model disentangle subtle cues, while contrastive judging ensures a reasoned final decision. by relying solely on prompting and structured reasoning, we aimed to investigate whether a smaller-scale llm could discern subtle emotional cues without specialized training. our zero-shot method, although not striving for state-of-the-art performance, yielded results around the baseline for both track b and track c. we observed that: we have released our code, prompts, and intermediate outputs (rewritten sentences and model reasoning) on github for reproducibility and further research github repository. ## system overview our approach relies on a minimalistic twostep pipeline built using gpt-4o mini (gpt-4o-mini-2024-07-18) as the processing core: since our method is self-contained, switching llm providers or models requires only updating the client and model name. the simplicity of the pipeline (two api calls, standardized prompt structure) and the absence of fine-tuning or other methods that require labeled data make our approach a plug-and-play solution. ## experimental setup we tested our pipeline on track b and track c of semeval-2025 task 11 using the official splits provided in brighter @xcite . we crafted our prompts using the dev set (treating it only for evaluation and prompt iteration) and then ran the final prompts on the test data. this included languages of varying resource levels, from english and chinese to ukrainian and isizulu. we did not apply any explicit preprocessing or domain adaptation (e.g., removing special characters), relying on gpt-4o mini's internal handling. for each test instance, we called the model twice: once per emotion candidate (for role-play rewriting) and once for the final contrastive judgment, both using the structured output built-in feature of the gpt-4o mini model. the prompts and schemas are provided in the appendix. we employed the openai python client library to run our prompts and used macro-averaged f1 (for emotion recognition) and mean absolute error (for intensity), aligned with the task's official evaluation metrics. ## results overall, our system provided near-baseline results on both tracks, reflecting the minimalist nature of our zero-shot approach. for track b (figure 2 ), english performed best (0.5693 average pearson r), followed by ukrainian (0.3517), hausa (0.3372), and chinese (0.3068). we see that english data achieved higher scores for fear and joy, which aligns with the fact that gpt-4o mini is primarily trained on extensive english corpora. chinese exhibited the lowest consistency for surprise (@xmath0.133), suggesting that the model struggled with subtle intensities in non-latin scripts. figure 3: track b language comparison for track c (figure 4 ), macro-f1 ranged from 0.5598 in english down to 0.1894 for isizulu. notably, languages with sparse resources-like isizulu (0.1894) and ukrainian (0.237)-lagged behind. even within medium-resource languages (e.g., indonesian at 0.5055 macro-f1), performance varied across emotions: fear and surprise were particularly challenging, possibly due to cross-lingual semantic gaps and fewer training signals in gpt-4o's domain knowledge. this discrepancy highlights how zero-shot performance can fluctuate significantly by language (figures 3, 5 ) and emotion category. figure 5: track c language comparison a closer look at error patterns revealed that many mistakes occurred when the role-play rewriting step excessively modified or insufficiently modified the texts. when the rewrite did not accurately reflect the target emotion, the contrastive judging step struggled to pick a correct match. conversely, when the rewrite was successful, the model often selected the correct emotion label. we suspect that isolating the contrastive judging mechanism (i.e., removing the rewriting stage) might help gauge how much rewriting errors degrade final predictions-an avenue for future systematic ablation. in terms of competition ranking, our system was not among the top-scoring submissions. however, ## conclusion we presented a novel zero-shot pipeline for multilingual emotion recognition using gpt-4o mini, employing a role-play rewriting step followed by contrastive judging. despite near-baseline official results, our method remains highly adaptable, requiring no labeled data and minimal computational cost. future work can explore replacing or refining the rewriting step, comparing this approach across different llms, and systematically evaluating each module (rewriting vs. judging) in isolation. we believe this line of research will open opportunities for more accessible, modular, and interpretable emotion detection solutions-especially valuable in low-resource and multilingual settings.