E mo B ench- UA : A Benchmark Dataset for Emotion Detection in U krainian (2025)

Abstract: While Ukrainian NLP has seen progress in many texts processing tasks, emotion classification remains an underexplored area with no publicly available benchmark to date. In this work, we introduce **EmoBench-UA**, the first annotated dataset for emotion detection in Ukrainian texts. Our annotation schema is adapted from the previous English-centric works on emotion detection (Mohammad et al., 2018; Mohammad, 2022) guidelines. The dataset was created through crowdsourcing using the Toloka.ai platform ensuring high-quality of the annotation process. Then, we evaluate a range of approaches on the collected dataset, starting from linguistic-based baselines, synthetic data translated from English, to large language models (LLMs). Our findings highlight the challenges of emotion classification in non-mainstream languages like Ukrainian and emphasize the need for further development of Ukrainian-specific models and training resources.

## introduction recent trends in natural language processing indicate a shift from predominantly monolingual english-centric approaches toward more inclusive multilingual solutions that support less-resourced and non-mainstream languages. although crosslingual transfer techniques-such as adapter modules @xcite or translation from resource-rich languages @xcite have shown promise, the development of highquality, language-specific datasets remains essential for achieving robust and culturally accurate performance in these settings. for the ukrainian language, significant progress has been made in the development of resources for various stylistic classification tasks, such as sentiment analysis @xcite and toxicity detection @xcite . however, to the best of our knowledge, no publicly available emobench-ua а я сьогодні біжу до щастя:) and today i am running to happiness:) 😄 joy ти з мене іздіваєшся?! are you kidding me?! 😡 anger шось мені ця арома кава не подобається, фу i don't like this flavored coffee, ew 🤢 disgust такого ви ще не бачили! you have never seen anything like it! 😲 surprise я скучила за цим місцем… i missed this place... ## emobench-ua collection here, we present the design of the crowdsourcing collection pipeline, detailing the task setup, annotation guidelines, interface design, and the applied quality control procedures used to obtain emobench-ua. the overall schema of the pipeline is presented in figure 2 . ## emobench-ua after filtering out low-confidence and ambiguous samples from the annotation results, we obtained a final emobench-ua of 4949 labelled instances (145 samples were dropped due to label conflicts). krippendorff's alpha agreement score was 0.85. then, we partitioned the dataset into fixed train/de-7 https://toloka.ai/docs/guide/project rating stat velopment/test subsets following a 50/5/45% split ratio. an overview of the label distribution across these subsets is presented in figure 4a . we also provide distribution of texts with one or more emotions per split in appendix e. the dataset examples can be found in appendix f. we were able to collect at least one hundred, and in some cases several hundred, instances for each emotion category. the majority of the sentences have clear distinguished one emotion with rare cases of texts with several emotions. nevertheless, the dataset remains imbalanced, with joy and sadness being the most prevalent emotions among the labeled samples, alongside a substantial portion of texts assigned the none label. such imbalance is a common characteristic of emotion detection datasets, reflecting the natural distribution of emotions in real-world text and contributing to the overall complexity of the task. additionally, in figure 4b , we provide a closer analysis of the collected emotional data by extracting the top-10 keywords for each emotion label (lemmatization done using the spacy foot_5 library). the resulting keywords reveal clear and intuitive associations with the corresponding emotional categories, further confirming the quality and relevance of the annotated texts. ## models we test various types on models on our collected dataset: (i) linguistic-based approaches; ## results the results of models evaluation on the test part of our novel emobench-ua dataset on the binary multi-label classification task are presented in the ## conclusion we introduced emobench-ua-the first manually annotated dataset for emotion detection in ukrainian texts. the proposed pipeline combines data preprocessing with a two-stage annotation procedure, incorporating multiple quality control measurements to ensure the high quality annotation. we benchmarked a wide range of approaches for the multi-label emotion classification task, demonstrating that although the latest llms, such as deepseek, achieved the strongest results, more efficient encoder-based models perform competitively. we hope this work also encourages further research on ukrainian-specific emotion detectors, including ensemble strategies and augmentation with other resource-rich languages resources. although the collected ukrainian dataset is smaller than comparable english resources, its natural, culturally grounded data has already proven more effective than purely cross-lingual transfer approaches. medium-sized encoder-based transformers fine-tuned on our dataset achieve performance on par with larger llms. we therefore believe that our openly released with all required details data collection pipeline offer a replicable framework for building high-quality and enough in size resources for other underrepresented languages. finally, we believe that our experimental setup-baselines selection and prompts design-offers a straightforward, extensible evaluation framework for other languages providing a possibility to select a corresponding state-of-the-art approach for text-based emotion analysis.