G ate NLP at S em E val-2025 Task 10: Hierarchical Three-Step Prompting for Multilingual Narrative Classification (2025)

Abstract: The proliferation of online news and the increasing spread of misinformation necessitate robust methods for automated narrative classification. This paper presents our approach to SemEval 2025 Task 10 Subtask 2, which aims to classify news articles into a predefined two-level taxonomy of main narratives and sub-narratives across multiple languages. We propose Hierarchical Three-Step Prompting (H3Prompt) for multilingual narrative classification. Our methodology follows a three-step prompting strategy, where the model first categorises an article into one of two domains (Ukraine-Russia War or Climate Change), then identifies the most relevant main narratives, and finally assigns sub-narratives. Our approach secured the top position on the English test set among 28 competing teams worldwide. This result highlights the effectiveness of our method in improving narrative classification performance over the baselines.

## introduction the rapid dissemination of information online has significantly influenced public discourse, making it crucial to detect and classify narratives accurately @xcite . narrative classification plays a key role in understanding how different perspectives shape public opinion and in identifying potential misinformation campaigns @xcite . to advance research in this area, the semeval 2025 shared task 10 @xcite presents multilingual characterisation and extraction of narratives from online news providers. a narrative is defined as a structured presentation of information that conveys a specific message or viewpoint, often forming a cohesive storyline foot_0 . the task provides a benchmark for evaluating and developing narrative classification models @xcite , helping researchers analyse how narratives emerge and propagate across different languages. as part of this challenge, subtask 2 @xcite focuses on assigning appropriate subnarrative labels to a given news article based on a two-level taxonomy foot_1 @xcite , where each narrative is further divided into subnarratives. this is a multi-label, multi-class document classification task involving news articles from two key domains: the ukraine-russia war and climate change. the dataset comprises articles, collected between 2022 and mid-2024, in five languages: bulgarian, english, hindi, portuguese, and russian. a significant portion of these articles have been flagged by fact-checkers as potentially spreading misinformation @xcite . previous work has focused on fine-grained narrative classification across various domains, including climate change @xcite @xcite , the ukraine-russia war @xcite , health misinformation @xcite , and the covid-19 infodemic @xcite @xcite . these studies have proposed models to identify narratives, aiding in the analysis of misinformation and public discourse within these critical topics. given the complexity and multilingual nature of this task, this paper proposes a novel hierarchical . in this, we fine-tune a large language model (llm) from the llama 3.2 family using h3prompt by leveraging both training data and synthetically generated data. our method follows a three-step prompting framework, ensuring a structured and hierarchical classification process. this approach enhances the model's ability to accurately distinguish between narratives and sub-narratives, improving classification performance across multiple languages. moreover, it also allows analysts to gain deeper insights into emerging narratives. ## hierarchical three-step prompting (h3prompt) our approach to narrative classification follows a hierarchical three-step prompting mechanism. we first describe the dataset and synthetic data generation process (section 2.1). next, we outline fine-tuning details (section 2.2). finally, we detail the prompt structure for refining predictions across classification levels (section 2.3). ## results and discussion in zero-shot models, gpt-4o achieves the highest f1-score for coarse-grained classification and gpt-4o-mini gives the highest f1-score for finegrained classification. on the other hand, zero-shot models, such as llama-3.2-3b-instruct and gemma-2-2b-it, perform significantly worse than trained models, indicating that domain-specific fine-tuning is crucial for improving the narrative classification performance. among trained models, logistic regression and lightgbm achieve moderate performance, but transformer-based models such as roberta-base and llama-3.2 h3prompt outperforms them. notably, our hierarchical three-step prompting approach (llama-3.2 h3prompt) achieves an f1 macro coarse score of 0.577 and an f1 samples fine score of 0.482, demonstrating the effectiveness of structured classification. the results also indicate that incorporating synthetic data during training improves performance, as models trained solely on the provided training data (denoted by w/o synth) perform worse than those that incorporate additional synthetic data. for instance, for llama-3.2 h3prompt, training with synthetic data improves the fine-grained f1 score by 23% (improvement from 0.392 to 0.482), while for roberta-base, it leads to a 10% improvement (improvement from 0.397 to 0.439). in addition, binary classification models (denoted by b) showed a slight decrease in performance compared to hierarchical prompting models, reinforcing the importance of a structured threestep classification approach. to further improve classification, we use the bestperforming model (i.e., llama-3.2 h3prompt) to experiment with a bagging ensemble @xcite to reduce variance from individual models. specifically, we train three different models on separate subsets of the dataset and then combine their predictions. we use three different strategies to aggregate the predictions: (1) union-based, where a sub-narrative is selected if any model predicts it; (2) majority-vote, where a sub-narrative is selected if at least two of the models predict it; and among the ensemble methods, we find that llama-3.2 h3prompt (ensemble -union) is the best-performing model. it achieved the highest scores, with 0.623 for narratives and 0.516 for subnarratives, showcasing the advantage of ensemble methods in improving classification robustness. furthermore, we submitted our best-performing run, llama-3.2 h3prompt (ensemble -union), for evaluation on the test set. we submitted our test predictions for bulgarian, english, hindi, portuguese, and russian. for all non-english articles, we first machine-translated foot_2 them into english and then used the translated text for inference. as shown on the test leaderboard foot_3 , our gatenlp submission secured 1st place for english, portuguese, and russian. for bulgarian and hindi, it ranked 3rd and 5th, respectively. these results highlight the potential of our method for fine-grained narrative classification across multiple languages and misinformation domains. ## conclusion in this paper, we introduced hierarchical three-step prompting (h3prompt) for multilingual narrative classification as part of semeval 2025 task 10 subtask 2. our approach fine-tuned llama 3.2 using both annotated training data and synthetically generated news articles to enhance classification robustness. our method secured the top position on the english test set among 28 competing teams worldwide, demonstrating the effectiveness of our approach for fine-grained narrative classification. experimental results showed that h3prompt outperforms baseline methods and zero-shot models, achieving state-of-the-art performance in narrative and sub-narrative classification. we further demonstrated that incorporating synthetic data during training significantly improves model performance. additionally, ensemble methods provided further enhancements, achieving the highest scores across multiple languages.