Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence (2025)

Abstract: Propaganda detection on social media remains challenging due to task complexity and limited high-quality labeled data. This paper introduces a novel framework that combines human expertise with Large Language Model (LLM) assistance to improve both annotation consistency and scalability. We propose a hierarchical taxonomy that organizes 14 fine-grained propaganda techniques (CITATION) into three broader categories, conduct a human annotation study on the HQP dataset (CITATION) that reveals low inter-annotator agreement for fine-grained labels, and implement an LLM-assisted pre-annotation pipeline that extracts propagandistic spans, generates concise explanations, and assigns local labels as well as a global label. A secondary human verification study shows significant improvements in both agreement and time-efficiency. Building on this, we fine-tune smaller language models (SLMs) to perform structured annotation. Instead of fine-tuning on human annotations, we train on high-quality LLM-generated data, allowing a large model to produce these annotations and a smaller model to learn to generate them via knowledge distillation. Our work contributes towards the development of scalable and robust propaganda detection systems, supporting the idea of transparent and accountable media ecosystems in line with SDG 16. The code is publicly available at our GitHub repository.

## introduction fake news and disinformation have become a significant challenge, particularly in geopolitical conflicts like the russia-ukraine war @xcite . disinformation campaigns strategically manipulate public opinion and shape narratives @xcite , with pro-russian biases linked to reduced ability to identify propaganda @xcite . propaganda, defined as "the deliberate and systematic attempt to shape perceptions, manipulate cognitions, and direct behavior to achieve a response that furthers the desired intent of the propagandist" @xcite , lies at the core of these campaigns. detecting such manipulative content is critical for preserving public trust and safeguarding democratic processes @xcite . while propaganda in long-form text is well studied (martino et al., 2020), short-form propaganda remains more challenging due to limited annotated data, sparse context, and the use of informal language, abbreviations, and hashtags @xcite . although automated methods for disinformation and propaganda detection have advanced @xcite , the task remains difficult. subtle linguistic cues, context-dependent interpretations, and low inter-annotator agreement highlight the complexity of human annotations @xcite , particularly in fine-grained classification @xcite , as propaganda often exploits cognitive biases and undermines critical thinking, making individuals more susceptible to conspiratorial narratives @xcite . propaganda detection aligns with the united nations sustainable development goal (sdg) 16 foot_0 , which promotes peaceful, inclusive societies and effective institutions. misinformation and propaganda undermine these aspirations by fueling social divisions, eroding trust in institutions, and obstructing transparent communication @xcite , especially when amplified by automated bots @xcite . in this work, we propose a methodology that advances propaganda detection through the following five key contributions: first, we develop a finegrained propaganda taxonomy that categorizes 14 distinct techniques by @xcite into three broader groups based on their intent: those that trigger emotional responses, those that simplify or distort complex issues, and those that undermine trust through authority and group dynamics. second, we conduct an initial human annotation study on a statistically significant subset of propagandistic tweets from the hqp dataset @xcite . this study highlights the challenges of manual fine-grained labeling, revealing that the process is highly subjective, time-consuming, and prone to low inter-annotator agreement. third, to overcome these limitations, we propose a novel llm-assisted annotation methodology. in our pipeline, llms first extract relevant propaganda spans from the text, explain why these spans are considered propagandistic, and then assign fine-grained labels at the span level before determining a global label for the entire post. fourth, we perform a secondary human verification study on a stratified sample of llm-annotated posts. in this stage, human annotators are presented with the extracted spans and their local labels, and tasked with annotating the global propaganda label. we observe that annota-tion agreement increases, and time investment is reduced by introducing llms as pre-annotation tool. finally, we fine-tune small language models on the llm-generated annotations to perform structured span-based labeling and explanation, enabling scalable training through knowledge distillation without relying on human-labeled data. ## related work early research on automatic propaganda detection approached the problem at the document level, aiming to classify entire news articles @xcite . for instance, some systems labeled texts into four broad categories (trusted, satire, hoax, or propaganda) @xcite , while others framed it as a binary task (propaganda, nonpropaganda) @xcite , which limited granularity and explainability @xcite . an advance came with the work of @xcite , who introduced span-level analysis with the ptc corpus, which comprises news articles annotated at the sentence level and fragment level with 18 distinct propaganda techniques. this scheme was adopted by the semevak-2020 shared task @xcite which consolidated the 18 techniques into a set of 14 widely used labels @xcite @xcite , that we also follow in our work. early models used bert-based architectures to perform span identification and technique classification @xcite . building on this, recent work explores how llms can further enhance propaganda detection, in terms of reducing annotation time and cost while improving label agreement and quality across classification tasks @xcite @xcite . however, the use of llms may also exhibit stronger systematic bias than human annotators, especially in politically sensitive contexts (vera and driggers, 2024), and may suffer from generation-related issues such as hallucinations @xcite . within propaganda detection, jose and greenstadt (2025) evaluated gpt-3.5, gpt-4, and claude on identifying six propaganda techniques in news articles. @xcite employed gpt-4 as an llm-as-annotator approach to annotate arabic text spans with 23 propaganda techniques using multilabel and sequence tagging tasks, and trained bert-based models on the generated annotations. similarly, @xcite examined the performance of multiple gpt-3 and gpt-4 variants for multi-label classification of 14 propaganda techniques at article-level using the semeval-2020 task 11 dataset @xcite , employing a range of prompt engineering and fine-tuning strategies. their results show that gpt-4 can approach state-of-the-art performance. our work builds on these efforts by grouping the 14 fine-grained techniques @xcite into a novel coarse-grained taxonomy of three broader categories to support human annotator clarity and enable hierarchical modeling. by using a fully open-source llm (llama3-70b), we extract propaganda spans from tweets and assign fine-grained local labels based on the 14 techniques from @xcite . in addition, it assigns a global propaganda label that captures the tweet's overall framing. while the llm also generates explanations for why each span was classified as propagandistic, these are not shown to human annotators but are used as an intermediate reasoning step to guide models towards their prediction. moreover, we distill four small student models on the generated outputs of the larger model as teacher to enable propaganda span in resource-constrained environments through an open-source modeling pipeline. ## methodology and results in this section, we outline our novel methodology that combines human expertise with computational techniques, as displayed in figure 2 , and their results. we first define a labeling framework for both coarse-grained and fine-grained categories in section 3.1. next, we describe our human annotation study (study 1, see section 3.2) on the hqp dataset @xcite . we then detail our llm few-shot inference in section 3.3 and annotation approach, to automatically extract propaganda spans, generate explanations, and assign fine-grained labels, followed by a second human verification study (study 2, see section 3.4). finally, we fine-tune smls via knowledge distillation in section 3.5. ## discussion and conclusion in this paper, we introduced an llm-assisted annotation framework that combines automated extraction of propaganda spans with human verification. our experiments demonstrate that integrating llm-assisted pre-annotation with human verification significantly improves the consistency and efficiency of propaganda detection. in study 1, manual fine-grained labeling suffered from low interannotator agreement and long annotation times. study 2, which incorporated llm-generated preannotations based on extracted propaganda spans, yielded higher agreement metrics and reduced annotation time, although part of the efficiency gain may stem from annotators' familiarity with the task. notably, our results suggest that a single global label is sometimes insufficient to capture the complexity of propagandistic content, as our analysis shows most tweets include more than one extracted propaganda span. this granular perspective may offer better insights than traditional sequence-level classification, and it is more scalable across different text lengths. these findings, in line with emerging trends such as those highlighted in semeval-2023 task 3 @xcite , indicate that future work should consider reformulating the problem to emphasize alternative propaganda detection strategies. exploring multi-label and hierarchical annotation strategies may better accommodate the overlapping nature of propaganda techniques. finally, integrating richer contextual information and real-time fact-checking modules could further refine detection performance @xcite . we also advocate for iterative human-in-the-loop systems that continuously update few-shot examples and label definitions to minimize bias and enhance model robustness.