Framing the Language: Fine-Tuning Gemma 3 for Manipulation Detection (2025)

Abstract: In this paper, we present our solutions for the two UNLP 2025 shared tasks: manipulation span detection and manipulation technique classification in Ukraine-related media content sourced from Telegram channels. We experimented with fine-tuning large language models (LLMs) with up to 12 billion parameters, including both encoder- and decoder-based architectures. Our experiments identified Gemma 3 12b with a custom classification head as the best-performing model for both tasks. To address the limited size of the original training dataset, we generated 50k synthetic samples and marked up an additional 400k media entries containing manipulative content.

## introduction over the past decade, rapid progress in nlp has coincided with growing concerns about the influence of fake news on electoral outcomes, particularly during the 2016 u.s. presidential election @xcite . it is perhaps no coincidence that the pioneering efforts to apply nlp methods to the automated detection of manipulative news took place in the late 2010s @xcite @xcite . however, these early attempts mostly relied on n-gram feature heuristics and only offered binary classification of the entire document as manipulative. the first fine-grained approach was proposed in 2019 @xcite . the idea of fine-grained analysis of propaganda in the news became the foundation of task 11 of the semeval-2020 competition @xcite , where manipulation span detection and technique classification has been presented as separate subtasks. the unlp 2025 shared task competition comprises of two subtasks: manipulation span identification (si) and manipulation technique classification (tc). the two subtasks share the same dataset, which included texts from the ukrainerelated social media content (specifically, telegram) in ukrainian and russian. the objective of the si is to identify manipulative words in the provided text without the need of classifying the manipulation technique. the tc task is a multilabel classification task, which requires identify whether a text contains one or several manipulation techniques from the following list: loaded language, glittering generalities, euphoria, appeal to fear, fud (fear, uncertainty, doubt), bandwagon/appeal to people, thought-terminating cliché, whataboutism, cherry picking, and straw man. this taxonomy differs from that of semeval-2020, including categories like euphoria and glittering generalities, which are characteristic for the ukrainian media landscape. the similarity between the semeval-2020 and unlp 2025 tasks offers a unique opportunity to highlight the evolution of nlp methods for solving such problems since 2020, which we explore in section 2. the paper is structured as follows. a brief overview of the training dataset along with the description of additional datasets used for this task is provided in section 3. our proposed solutions for the two subtasks are described in section 4. section 5 contains brief overview of exploratory experiments that we have conducted during development. the final section 6 contains information on the obtained results along with discussions. ## related work as mentioned in the introduction, semeval-2020 task 11 @xcite ) marked a milestone in the early days of fine-grained manipulation detection in news. the task demonstrated the dominance of bert-like encoder-based models, with only sparse use of earlier architectures such as tf-idf, elmo, rnns, and cnns. at the time, only one team out of 27 experimented with a decoderstyle model (gpt-2), but ultimately reverted to using roberta. the dominance of bert-like models began to be challenged in 2023 with the rise of both proprietary and open-weight decoder-based llms. notably, gpt-4 was reported to match the performance of state-of-the-art bert models on semeval-2020 task 11 @xcite . by the end of 2023, the use of decoder-style llms for classification tasks in ukrainian had become increasingly common @xcite . however, studies have shown that open-weight models such as llama 2 @xcite and mistral @xcite can still be outperformed by strong bert-like baselines in binary fake news classification @xcite . the shared tasks of unlp 2025 demonstrate that by 2025, generative llms have become as dominant as bert-based models were in 2020. ## solution description for both subtasks, we fine-tuned the gemma 3 12b (team., 2025) model as a base, replacing the original language modeling head with a classification head. the classification head consisted of one-head attention pooling over the final hidden states, followed by a dense output layer for classification. we used a significantly higher learning rate for the classification head layers (7e-5) that for the base model layers (2e-6). the best performing model was selected from 10 epochs of training based on validation set. the training curricula for the two subtasks are schematically illustrated in figure 1 . ## experiments during the development of our solution for the tc subtask, we experimented with parameter-efficient fine-tuning of a variety of models. we believe that sharing these experiments may be of interest to the community, as they provide insight into the tradeoffs and capabilities of different approaches. in the following, we describe the most notable experiments. the results of each experiment are presented in the llama 3 8b and llama guard 3 8b we have started our experiments by fine-tuning llama 3 8b @xcite as the baseline model in the class of 8-12b parameters. in particular, we were interested whether it can beat the bert baseline. we assumed that the guard @xcite model type is more sensitive to the manipulation techniques. ## results and discussions both subtasks of unlp-2025 were evaluated using the macro f1 score. for the ts subtask, we experimented with various models and ensembles (see sections 4 and 5), with the results summarized in the obtained results highlight a shift since semeval-2020: generative llms now consistently outperform bert-like models and have become the solution of choice for text classification tasks, even despite the limitations imposed by their causal nature. throughout our experiments, we fine-tuned several decoder-based models, including bert @xcite , ukr-roberta (youscan, 2023) and liberta (haltiuk and smywiński-pohl, 2024). however, none of these encoder models matched the performance of compact generative llms such as mistral 7b, llama 3 8b, or gemma 3 12b. the obtained results also provide insights into the factors that contribute to model performance on this type of task. it is no surprise that gemma 3 12b outperforms the other tested models, as it has the largest vocabulary, the highest parameter count, and is the most recent. llama 3 guard demonstrates the weakest performance among the evaluated models, possibly due to its lack of support for the ukrainian language. mistral-ua, on the other hand, nearly matches the larger and more advanced llama 3, likely due to its extended vocabulary and additional pretraining on ukrainian corpora. a notable characteristic of the shared task dataset is reflected in the underperformance of models trained on synthetic data. a possible reason is that machine-generated samples lack contextual awareness of the ukrainian media landscape, particularly with respect to relatively new slang (e.g., "тцк" (territorial center of recruitment), "чмобик" (poorly trained, unwilling, or inept mobilized russian soldier), "пэтриот" (mim-104 patriot, surface-to-air missile system)), uncommon or domain-specific terms (e.g., "ту-22м3" (tupolev tu-22m military plane), "контрнаступ" (counteroffensive, reffered to ukrainian liberation campaign), "ухилянт" (someone who evades mobilization)), and words used in non-standard or culturally specific senses (e.g., "ґрад" (bm-21 grad, a soviet-designed multiple rocket launcher system or heavy rain), "мясо" (term used to describe poorly trained, expendable soldiers)). we hypothesize that the llms used in our experiments were trained primarily on pre-invasion data, and therefore lack adequate exposure to this updated vocabulary and context. to test this hypothesis, we trained wide & deep @xcite )-inspired classifier. the model showed higher performance on the validation set than on the test submission. after removing the "wide" component (lemmatized vocabulary per language with stanza ( qi et al., 2020) ) the scores became aligned, indicating that the model likely memorized it. overall, we find that the unlp-2025 shared task provides valuable insights into both the progress of the nlp field and the importance of language-and culture-specific contextual training.