Comparing Methods for Multi-Label Classification of Manipulation Techniques in U krainian Telegram Content (2025)

Abstract: Detecting manipulation techniques in online text is vital for combating misinformation, a task complicated by generative AI. This paper compares machine learning approaches for multi-label classification of 10 techniques in Ukrainian Telegram content (UNLP 2025 Shared Task 1). Our evaluation included TF-IDF, fine-tuned XLM-RoBERTa-Large, PEFT-LLM (Gemma, Mistral) and a RAG approach (E5 + Mistral Nemo). The fine-tuned XLM-RoBERTa-Large model, which incorporates weighted loss to address class imbalance, yielded the highest Macro F1 score (0.4346). This result surpassed the performance of TF-IDF (Macro F1 0.32-0.36), the PEFT-LLM (0.28-0.33) and RAG (0.309). Synthetic data slightly helped TF-IDF but reduced transformer model performance. The results demonstrate the strong performance of standard transformers like XLM-R when appropriately configured for this classification task.

## introduction the volume of online content requires effective methods to identify manipulative language. this work focuses on detecting specific manipulation techniques -defined here as rhetorical or stylistic methods aimed at influencing audiences without clear factual support -within ukrainian social media content, specifically from telegram. this investigation is part of our more extensive research on the challenges posed by generative ai in the defense of sybil's attacks on social media. @xcite . understanding these manipulation techniques is therefore crucial for countering coordinated information operations, especially in contexts such as the ongoing hybrid warfare against ukraine, where social networks are actively used for disinformation campaigns. @xcite . this paper investigates the effectiveness of different modeling approaches for the specific task of identifying manipulation techniques, using data from the unlp 2025 shared task (subtask 1). this shared task aims to assess the ai capabilities in detecting the manipulation of social media within the ukrainian context. we compare: our findings indicate that fine-tuning standard transformer models like roberta yields strong performance on this multi-label classification task, especially with limited data. concurrently, we explored the potential of advanced methods such as retrieval-augmented generation (rag) and parameter-efficient fine-tuning (peft) using lora for smaller llms, providing insights into their applicability compared to the established finetuning paradigm under data constraints. ## conclusion this paper compared several methods for multilabel classification of manipulation techniques in ukrainian telegram content. a standard fine-tuned xlm-roberta-large model with weighted loss achieved the highest performance (0.4346 macro f1), outperforming the tf-idf baselines, pefttuned llms (gemma, mistral) and an rag approach. the attempted augmentation of synthetic data using mistral large slightly benefited tf-idf but harmed transformer/llm performance, which shows challenges in generating effective synthetic data for complex models. our results show the continued effectiveness of appropriately tuned standard transformer architectures for specific classification tasks, especially when addressing dataset properties like class imbalance. although our rag implementation performed poorly here, the strategy shows potential, particularly for its ability to incorporate up-to-date information, which is important for dynamic analysis tasks. we suggest that rag could be useful in a production pipeline, perhaps using llms finetuned with a dedicated classification head. some competition participants reportedly achieved results that exceeded our xlm-r score, possibly employing such custom llm classifiers, indicating room for improvement over standard transformers. furthermore, the increasing sophistication of aigenerated content used for targeted manipulation @xcite , requires the development of adaptive, potentially hybrid detection systems. a key focus for future work will be improving the adaptability to new manipulation campaigns and evolving language, favoring further investigation of rag and reasoning models.