The UNLP 2025 Shared Task on Detecting Social Media Manipulation (2025)

Abstract: This paper presents the results of the UNLP 2025 Shared Task on Detecting Social Media Manipulation. The task included two tracks: Technique Classification and Span Identification. The benchmark dataset contains 9,557 posts from Ukrainian Telegram channels manually annotated by media experts. A total of 51 teams registered, 22 teams submitted systems, and 595 runs were evaluated on a hidden test set via Kaggle. Performance was measured with macro F1 for classification and token‐level F1 for identification. The shared task provides the first publicly available benchmark for manipulation detection in Ukrainian social media and highlights promising directions for low‐resource propaganda research. The Kaggle leaderboard is left open for further submissions.

## introduction the disinformation and manipulative content on social media platforms poses significant challenges to information integrity. in ukraine, the spread of propaganda through channels like telegram has underscored the need for advanced nlp techniques to detect and mitigate such content. recent studies have emphasized the importance of automatic approaches for identifying disinformation, including work focused on russian-and ukrainian-language content @xcite @xcite . to address these challenges, the fourth workshop on ukrainian natural language processing (unlp) 2025, together with texty.org.ua 1 , organized a shared task focused on the detection of social media manipulation in ukrainian information space. the task comprised two subtasks: the dataset for this shared task was created by texty.org.ua and consists of 9,557 ukrainian telegram posts annotated by media experts for manipulation techniques. this initiative aims to encourage the development of nlp models capable of understanding and detecting nuanced manipulative strategies in ukraine. participants received the datasets, task descriptions, and evaluation metrics via the official github repository foot_0 . both subtasks were hosted as kaggle competitions: technique classification foot_1 and span identification 4 . this paper presents an overview of the shared task, including the dataset, evaluation methodology, and a synthesis of participants' approaches and results. by analyzing the outcomes, we aim to highlight the progress in ukrainian nlp and identify areas for future research and development. the remainder of this paper is organized as follows. section 2 reviews previous work on propaganda detection and span-level manipulation identification. section 3 outlines the unlp 2025 shared-task setup. section 4 presents the dataset and manipulation-technique taxonomy. section 5 describes the evaluation metrics and ranking procedure. section 6 reports the leaderboard results and summarises the submitted systems. section 7 concludes the paper, while section 8 provides an ethics statement and section 9 discusses current limitations and future work. ## related work early work in domain of disinformation detection focused on identifying biased or manipulative rhetoric in english-language news sources @xcite . subsequent shared tasks such as semeval 2020 task 11 @xcite and the nlp4if workshop @xcite further advanced the field by providing benchmark datasets and introducing more finegrained classification of propaganda techniques. span-based propaganda detection, introduced in da san martino et al. (2020), treats the problem as a sequence labeling or span extraction task and remains a challenging low-resource setting. in multilingual contexts, limited annotated data has led to the adoption of transfer learning approaches using multilingual transformers like xlm-r @xcite and fine-tuned mbert @xcite for classification and span identification. ## data the dataset consists of 9,557 telegram posts annotated for the presence of manipulation techniques. the content was collected from ukrainian news and political blog channels on telegram, comprising texts in ukrainian and russian languages. this bilingual composition provides diverse examples of manipulative language used across different segments of the ukrainian information space. the dataset includes both manipulative and nonmanipulative posts, with the distribution by language shown in the dataset is available through the official repository of the shared task foot_3 and is licensed under the cc by-nc-sa 4.0 license. ## results and system descriptions the shared task drew broad engagement: 51 teams registered, and 22 ultimately submitted solutions. nine of these teams participated in both subtasks, while eleven entered only the technique classification track and two focused solely on span identification. in total, 595 submissions were evaluated -386 for technique classification and 209 for span identification. ## conclusion we believe that the unlp 2025 shared task is instrumental in facilitating research on propaganda detection and span-level manipulation identification in ukrainian-language social media content. teams explored a variety of techniques -from threshold optimization and span post-processing to lora fine-tuning and multi-stage inference pipelines -demonstrating the creative potential of the nlp research community when working in low-resource settings. all datasets used in the shared task are publicly available on github, and all participating teams agreed to open-source their final systems. this ensures the reproducibility of results and contributes to the development of more accessible and transparent models for the ukrainian language. top-performing systems employed models such as gemma 2-27b, xlm-roberta, and mdeberta. we hope this shared task will serve as a foundation for future work in ukrainian nlp, and that the tools, data, and approaches developed through this competition will continue to support progress in trustworthy ai systems for media analysis.