Improving Sentiment Analysis for U krainian Social Media Code-Switching Data (2025)

Abstract: This paper addresses the challenges of sentiment analysis in Ukrainian social media, where users frequently engage in code-switching with Russian and other languages. We introduce COSMUS (COde-Switched MUltilingual Sentiment for Ukrainian Social media), a 12,224-texts corpus collected from Telegram channels, product‐review sites and open datasets, annotated into positive, negative, neutral and mixed sentiment classes as well as language labels (Ukrainian, Russian, code-switched). We benchmark three modeling paradigms: (i) few‐shot prompting of GPT‐4o and DeepSeek V2-chat, (ii) multilingual mBERT, and (iii) the Ukrainian‐centric UkrRoberta. We also analyze calibration and LIME scores of the latter two solutions to verify its performance on various language labels. To mitigate data sparsity we test two augmentation strategies: back‐translation consistently hurts performance, whereas a Large Language Model (LLM) word‐substitution scheme yields up to +2.2% accuracy. Our work delivers the first publicly available dataset and comprehensive benchmark for sentiment classification in Ukrainian code‐switching media. Results demonstrate that language‐specific pre‐training combined with targeted augmentation yields the most accurate and trustworthy predictions in this challenging low‐resource setting.

## introduction sentiment analysis has long been one of crucial tasks in natural language processing (nlp), with wide-ranging applications in business, media, and the social sciences. the field saw significant progress with the adoption of deep learning techniques and the introduction of transformer-based architectures, which enabled state-of-the-art sentiment classifiers to routinely achieve over 90% ac-curacy and f1-scores on english-language benchmarks @xcite . however, these advancements are unevenly distributed, as high-resource languages benefit from abundant labeled data. this gap is especially pronounced in informal, multilingual settings such as ukrainian social media, where users frequently mix dialects, use transliterations, and code-switch with russian and other languages. this involves not only mixing lexicon and morphems, but also grammatical forms and structures between several languages within one linguistic utterance or text @xcite . with nearly 20% of users engaging in content beyond ukrainian @xcite , there is a clear need for sentiment analysis systems that are multilingual and code-switching aware. to address these gaps, we propose and test a comprehensive framework for sentiment analysis in ukrainian social media, tailored to the unique linguistic landscape. our contributions are threefold: in addition, we apply an explainable ai (xai) lime analysis @xcite and model cal-ibration analysis to verify the reliability of our classifier approaches. our findings contribute to both the ukrainian nlp landscape and the broader field of sentiment analysis in low-resource and codemixed language environments. ## related work most previous studies on sentiment analysis in code-switching linguistic settings focused on spanish and english (sp-eng cs) @xcite and the variable linguistic landscape of indian languages @xcite , including their intra-sentential codeswitching with english, such as in the case of dravidian @xcite . while codeswitching in ukrainian has been increasingly studied across various genres-including parliamentary discourse @xcite , mixed-speech transcripts @xcite , and social media platforms @xcite -few studies directly address the problem of sentiment analysis. existing sentiment analysis research in ukrainian primarily focuses on monolingual contexts or uses russian as a dominant language. @xcite explore sentiment trends in ukrainian and russian news articles, applying lexicon-based techniques, while @xcite present a rule-based method for analyzing user reviews written in ukrainian. more recent datasets, described e.g. in @xcite and ustyianovych and barbosa @xcite , incorporate mixed-language content; however, their primary focus remains on russian-dominant corpora or use sentiment orientations related to political stance rather than emotion polarity. entity-level sentiment classification has been applied in ukrainian-language media @xcite , demonstrating the viability of transformer-based models fine-tuned on domainspecific data. yet these approaches often assume standardized language inputs, omitting the hybrid linguistic characteristics seen on platforms like telegram, where code-mixing, dialectal variation, and transliteration are common. while general-purpose multilingual models like mbert and xlm-r have been applied to sentiment analysis in low-resource european languages @xcite , their robustness in ukrainian-russian code-switched settings remains unexplored. recent experiments be-gan to explore fine-tuning large multilingual transformers or llms (e.g., gpt-4, llama3) for this task @xcite , with mixed results and limited evidence of generalization to informal social media discourse. in summary, although foundational work exists for sentiment detection in ukrainian, there remains a notable absence of approaches tailored to the challenges of code-switching. ## methodology to address the identified gaps, we propose a sentiment classification approach for ukrainian social media data, encompassing data preprocessing, annotation, and a structured experimental methodology. in this study, we do not differentiate between code-switching (intersentential) and code-mixing (intra-sentential) and refer to the phenomenon as a whole as code-switching. ## discussion overall, ukrroberta's stronger performance (0.73 vs 0.67 micro f1) confirms that language-specific pre-training, combined with targeted word-substitution augmentation, is a more effective strategy for sentiment analysis in the linguistically complex landscape of ukrainian social media. while our peak accuracy of 73.6% is lower than the 90%+ performance often reported for monolingual english sentiment analysis systems @xcite , the performance relationship we observe between general-purpose llms and smaller, task-specific fine-tuned models aligns with findings from prior work @xcite . this indicates that our approach performs comparably to existing solutions despite the inherent complexities of ukrainian-russian code-switching in social media content. our calibration analysis findings confirm that good discrimination does not automatically entail good calibration: while mbert achieves competitive f1 on russian texts, its reliability sharply degrades on ukrainian and mixed inputs. conversely, ukrroberta delivers more trustworthy probability estimates in the linguistically diverse conditions typical of ukrainian social media. while bias is generally undesirable, some degree of bias may be contextually appropriate in ukrainian historical and political context. although ukrroberta slightly underperforms in russian and exhibits a tendency to associate russian lexical items with negative sentiment, this trade-off seems more acceptable than reversed mbert's scenario, more favorable towards the russian language. since ai naturally amplifies and re-enforces existing biases, and considering that ukrainian language is historically downplayed and discriminated against, the choice between mbert and ukrroberta should involve these additional socio-linguistic considerations. additionally, ukrroberta may reflect realworld patterns of usage and sociopolitical framing of sentiment in ukrainian wartime discourse. finally, we storngly advocate that interpretability and calibration are essential in evaluating sentiment models beyond f1 scores-especially when language identity and political stance are intertwined. while our best-performing model (ukrroberta with word substitution) shows promising robustness, further work is needed to handle sarcasm, negation, and mixed affect more reliably. ## conclusion we present cosmus, the first publicly available, 12,224 texts corpus of ukrainian, russian and code-switched social media texts with four-way sentiment labels and substantial annotator agreement. fine-tuning the ukrroberta with gpt-4o-driven data augmentations yields the top accuracy of 73.6%, surpassing mbert and few-shot llm baselines. reliability diagrams and lime analysis show ukrroberta is also better calibrated across most language subsets and exhibits less language bias on ukrainian and codemixed samples.