COGNAC at S em E val-2025 Task 10: Multi-level Narrative Classification with Summarization and Hierarchical Prompting (2025)

Abstract: We present our approach to solving the Narrative Classification portion of the Multilingual Characterization and Extraction of Narratives SemEval-2025 challenge (Task 10, Subtask 2). This task is a multi-label, multi-class document classification task, where the classes were defined via natural language titles, descriptions, short examples, and annotator instructions, with only a few (and sometime no) labeled examples for training. Our approach leverages a text-summarization, binary relevance with zero-shot prompts, and hierarchical prompting using Large Language Models (LLM) to identify the narratives and subnarratives in the provided news articles. Notably, we did not use the labeled examples to train the system. Our approach well outperforms the official baseline and achieves an F1 score of 0.55 (narratives) and 0.43 (subnarratives), and placed 2nd in the test-set leaderboard at the system submission deadline. We provide an in-depth analysis of the construction and effectiveness of our approach using both open-source (LLaMA 3.1-8B-Instruct) and proprietary (GPT 4o-mini) Large Language Models under different prompting setups.

## introduction disinformation, misinformation, propaganda, and foreign malign influence (fmi) have become serious problems in the modern information environment. one commonality amongst them is the use of narrative to drive their effects. a narrative can be defined as a concise, concrete description of a set of events involving a small number of actors, often supporting an evaluative judgment. the ability to automatically identify narratives in textual materials (for example, news or social media) would be of great use to tracking, understanding, and mitigating pernicious influence. task 10 at semeval-2025, multilingual characterization and extraction of narratives from online news @xcite , focuses on automatic identification of different types of narratives and subnarratives, as well as identifying the roles of the relevant entities in news articles. the task is divided into three subtasks-entity framing, narrative classification, and narrative extractionspanning five languages: bulgarian, english, hindi, portuguese, and russian. we work on the subtask 2: narrative classification for english, where we develop a prompt-based approach to identify narratives and their subtypes in news data. subtask 2 defines two domains (climate change @xcite and ukraine-russia war [urw]), for which the task creators have defined a set of top-level narratives, each having specific subnarratives. each news article associated with the domains is labeled with some number of top-level narratives and subnarratives, with no restriction on the number of labels. each top-level narrative and subnarrative is defined by a title (e.g., criticism of climate policies), plus a longer definition, instructions, and zero or more examples. thus, subtask 2 is a multi-label, multi-class document classification task. our approach has three stages. first, we apply a summarization step that condenses the target document (i.e., a news article) into a uniform length, information-dense representation. second, we apply class-specific zero-shot prompts using a binary relevance strategy @xcite to classify each document as to its top-level narrative category, aggregating results to generate multi-label outputs. third, we use hierarchical prompting @xcite to produce subnarrative labels for each narrative found in the articles. we experiment with both open-source (llama 3.1-8b-instruct) (meta, 2024) and proprietary models (gpt-4o-mini) (ope-nai, 2024) . notably, our approach does not use any of the labeled training data for fine-tuning or other model optimizations (barring the experiment comparing zero-shot vs. few-shot setup, where we find that the zero-shot approach performs bet-ter overall). our system achieved an f 1 score of 0.55 for narratives and 0.43 for sub-narratives, placing 2 nd in the official leaderboard for english (the leading system obtained scores of 0.59 and 0.44, respectively). it is notable that our approach was competitive despite the absence of computationally expensive model training. the remainder of the paper is structured as follows. we first provide background on the topic of narrative classification and prompt-based solutions in general ( §2). we next describe the data and task definition provided by the task organizers ( §3). we then elaborate on our methodology and experimental set-up ( §4), and report the result from the official submission along with some additional experiments ( §5). finally, we enumerate our contributions and discuss our findings, limitations, and scope for future improvements ( §6). ## related work multi-label document classification is the task of assigning multiple relevant labels or categories to a text, as opposed to a single label @xcite . traditional machine learning (ml) and natural language processing (nlp) approaches have developed various methods to tackle this problem. one common approach is binary relevance, where the task is decomposed into independent binary classification problems for each label @xcite . another is classifier chains, which extends binary relevance by linking classifiers in a chain, allowing label predictions to influence one another and capture label dependencies @xcite . a third method, label power-set, treats each unique combination of labels as a separate class, transforming the problem into a mutually exclusive multi-class classification task @xcite . with the development of large language models (llms) @xcite @xcite , a new paradigm for text classification has emerged: instead of training a model specifically for a classification task, it is now possible to prompt a pretrained llm to classify text by describing the task in natural language. this approach has gained widespread popularity as llms have shown strong performance in new classification tasks through in-context learning (icl) with just a few prompt examples @xcite . in scenarios where little to no training data is available, this approach is especially attractive. @xcite showed how llms can use class definition to produce multi-label classification with zero-shot prompting. this label-bylabel prompting in an one-vs-rest manner is simple and allows the model to focus on a binary question each time, potentially improving reliability for each label. this approach is more computationally expensive due to requiring multiple queries for each input. an alternative prompting strategy to increase efficiency is to prompt the llm to produce all desired labels in one pass. however, this approach increases the classification complexity, which can lead to reduced performance. @xcite additionally, classification involving multiple classes require more sophisticated prompts and reasoning steps to guide the model, which may also increase format deviation in a model's output, compared to binary classification with simple yes/no format @xcite . the english training data comprised 399 articles, with 176 from the climate change domain and 223 from the ukraine-russia war domain. additionally, a development dataset of 41 labeled articles (24: cc, 17: urw) and a test dataset of 101 unlabeled articles (48: cc, 53: urw) were released by the task organizers. each article in the training and development dataset is annotated with one or more high-level narrative(s) as well as corresponding finer-grained subnarrative(s). in the case where specific narrative or subnarrative label could not be assigned, the "other" pseudo-label was used. ## approach our approach for the narrative classification subtask has three steps: (1) summarization that makes the articles more uniform in length and style ( §4.1); in this task, the number of class and subclass labels was large compared to the available labeled data. figure 1 shows the distribution of the number of available labeled samples per subnarrative class. notably, some subnarrative classes never appear in the english training data. this rendered approaches like supervised learning and fine-tuning problematic for those classes. therefore, we opted for a prompt-based approach using pre-trained llms. for our experiments, we chose one open-source model (llama 3.1-8b-instruct) and one proprietary model (gpt-4o-mini). it is worth mentioning that, in the english training data, the domain of each input article is indicated in the filename, so we assume knowledge of the domain for all experiments. however, we also conducted an auxiliary experiment that showed llms could automatically identify 1 the domain of the input texts in 99% of cases (435 out of 440 articles) across the training and validation datasets. ## results for the final submission, we chose the best performing model based on our experiments on the development set, which was gpt-4o-model with zero-shot prompts using summarized articles as input in both narrative and subnarrative classification step. our experiments showed that summarization significantly enhanced classification performance. we attribute this to summarization making the input more information-dense and uniformly structured. we also found that breaking down a complex multilabel classification problem into a series of simpler binary classifications led to better performance, which is consistent with the observations of many other researchers that llms are better at performing simple tasks with a clear goal compared to complex tasks with more sophisticated instructions. these effects were much more pronounced with the smaller model, which is consistent with the assumption that more advanced llms are better at handling complex tasks. it also indicates that using a larger, state-of-the-art model may further improve the performance of our approach. ## conclusion in this paper, we proposed a prompt-based approach to the multi-label multi-class narrative classification problem introduced at semeval-2025 (task 10, subtask 2) for the english language. we leveraged text-summarization, binary relevance with large language models (llms), and hierarchical prompting technique to label broad narratives as well as fine-grained subnarratives in news articles. we developed zero-shot prompts for each narrative and subnarrative class solely from the provided taxonomy, avoiding the need for training data and expensive model fine-tuning. this approach achieved competitive performance, placing 2 nd in the leaderboard. despite promising results, there is much room for improvement. our approach does not consider the possibility that some labels may be more likely to co-occur together. while we take advantage of class-specific prompts in a binary relevance approach, further refinement of these prompts with help of domain experts could help address specific weaknesses in individual narrative classifications. additionally, we only applied our approach for the english dataset, which leaves its multilingual capability an open question for future studies.