cocoa at S em E val-2025 Task 10: Prompting vs. Fine-Tuning: A Multilevel Approach to Propaganda Classification (2025)

Abstract: The increasing sophistication of natural language processing models has facilitated advancements in hierarchical text classification, particularly in the domain of propaganda detection. This paper presents our submission to SemEval 2025 Task 10, Subtask 1, which focuses on multilevel text classification for identifying and categorizing propaganda narratives in online news. We investigate two primary approaches: (1) prompt-based classification using large language models (LLMs) like GPT, which offers flexibility but struggles with hierarchical categorization, and (2) fine-tuning transformer-based models, where we employ a hierarchical structure—one model classifies the main propaganda category, followed by three separate models specializing in subcategory classification. Our results indicate that while LLMs demonstrate some generalization ability, fine-tuned models significantly outperform them in accuracy and reliability, reinforcing the importance of task-specific supervised learning for propaganda detection. Additionally, we discuss challenges related to data sparsity in subclassification and explore potential enhancements such as multi-task learning and hierarchical loss functions. Our findings contribute to the broader field of automated propaganda detection and emphasize the value of structured classification models in combating misinformation. All code and data used in our experiments will be made publicly available on our GitHub

## introduction the spread of propaganda and strategically crafted information in online media presents a growing challenge to public discourse and democratic processes. as propaganda techniques evolve, so too must the systems built to detect them. traditional approaches to propaganda detection have largely focused on identifying specific rhetorical strategies-such as appeals to fear, doubt, or namecalling-at the sentence or span level @xcite . others have focused on classifying entire articles or highlighting binary instances of propagandistic content. while effective for technique recognition, these approaches fall short when propaganda is embedded through more subtle means, such as the strategic portrayal of specific named entities across a narrative. recent work has begun to shift toward understanding propaganda at the entity level, recognizing that how named entities are framed across a narrative can shape readers' perceptions. while earlier shared tasks and studies emphasized detecting rhetorical techniques at the sentence or span level, they did not require models to assess the narrative role of specific entities. the semeval 2025 task 10 builds on these foundations by introducing a more fine-grained challenge: subtask 1 @xcite . in this task, systems are provided with a news article and a list of named entity (ne) mentions, and must assign one or more roles to each mention using a predefined taxonomy. these roles fall into three overarching categories-protagonist, antagonist, and innocent-each with further finegrained subtypes such as saboteur or conspirator, making this a multi-label, multi-class, span-level classification problem. in this work, we explore two primary approaches to tackling this classification task: (1) prompting large language models (llms) such as gpt, leveraging their zero-shot capabilities for classification; and (2) fine-tuning transformer-based models, where we train one model for main category classification and three specialized models for subclassification within each category. while llms provide adaptability and require minimal task-specific training, our experiments indicate that fine-tuned models significantly outperform them in accuracy and reliability. this underscores the limitations of generic prompting in highly structured classification tasks and highlights the continued relevance of task-specific supervised learning. by benchmarking different approaches to hierarchical propaganda detection, we aim to contribute insights that will inform future developments in automated media narrative analysis. ## related work early efforts in propaganda detection focused primarily on identifying rhetorical techniques within news articles. for example, semeval-2020 task 11 introduced two subtasks: span identification and technique classification. these tasks aimed to detect specific propaganda techniques, such as "loaded language" or "appeal to fear", within textual spans, using models such as bert for improved accuracy @xcite . building upon this foundation, recent studies have explored the capabilities of large language models (llms) in propaganda detection. an investigation was carried out on the performance of gpt-4 in identifying propagandist content. the findings indicated that, while llms show promise, they often perform underperformance compared to fine-tuned models, especially in tasks that require a nuanced understanding of context and subtle linguistic cues @xcite . advancing the field further, there was an introduction of a multilingual hierarchical corpus specifically designed for entity framing and role portrayal in news articles. the dataset categorizes entities into fine-grained roles nested within three main categories: protagonist, antagonist, and innocent. this taxonomy facilitates a more detailed analysis of how entities are portrayed in different narratives and languages @xcite . ## methodology our approach involved a structured pipeline comprising three main stages: data preprocessing, llm-based classification using gpt, and finetuning a transformer-based model for hierarchical classification. each stage was designed to efficiently extract entity roles from news articles and classify them into protagonist, antagonist, or innocent, along with their respective subcategories. ## results to compare the performance of gpt-4o, gpt-3.5turbo, and gpt-3.5-turbo-1106, we evaluated each model's ability to classify news articles into the main categories (protagonist, antagonist, innocent) and their corresponding subcategories. the accuracy of each model was calculated based on its ability to correctly assign labels to a test set using prompt-based classification. initially, prompts that included a highlighted target word resulted in 0% accuracy across all models. subsequent tests removed the explicit target word, which led to modest improvements in performance. ## conclusion in this work, we explored two approaches for hierarchical propaganda classification in semeval 2025 task 10, subtask 1: llm-based prompting and fine-tuned transformer models. our results demonstrated that while gpt models offered a flexible, zero-shot solution, they struggled with hierarchical dependencies and inconsistent subcategory classification. in contrast, fine-tuned transformer models, particularly roberta and bert-base, significantly outperformed llms, achieving higher accuracy for both main category and subcategory classification. these findings highlight the importance of taskspecific supervised learning in structured classification tasks. future work could explore hybrid approaches that combine llms for generalization with fine-tuned models for precision. additionally, integrating external knowledge sources or multi-task learning frameworks could further improve classification accuracy. our study contributes to the development of automated propaganda analysis and provides a foundation for more advanced methods in entity framing detection within news media.