Narrlangen at S em E val-2025 Task 10: Comparing (mostly) simple multilingual approaches to narrative classification (2025)

Abstract: Our team focused on Subtask 2 (narrative classification) and tried several conceptually straightforward approaches: (1) prompt engineering of LLMs, (2) a zero-shot approach based on sentence similarities, (3) direct classification of fine-grained labels using SetFit, (4) fine-tuning encoder models on fine-grained labels, and (5) hierarchical classification using encoder models with two different classification heads. We list results for all systems on the development set, which show that the best approach was to fine-tune a pre-trained multilingual model, XLM-RoBERTa, with two additional linear layers and a softmax as classification head.

## introduction & background narratives shape how information is conveyed and understood, influencing public discourse and decision-making processes. since both corporate and state actors are actively seeking to influence public discourse on topics such as climate change, vaccinations, migration, or the war in ukraine by pushing their own narratives, one of the greatest challenges of contemporary democracies is to identify and counteract such campaigns while upholding the ideal of freedom of expression. identifying narratives in texts, e.g. online news or social media, is a key part of this. more generally, robust methods for identifying and classifying narratives would also provide important support for large-scale analysis of textual data, allowing researchers to track the evolution of discourses across time. despite significant advancements in nlp, predicting narratives remains a complex challenge due to their abstract, multi-layered nature. traditional classification methods struggle with the implicit and evolving structures of narratives, which often span multiple sentences or paragraphs. recent ap-proaches, including zero-shot learning and finetuning of transformer models, have demonstrated promise in capturing nuanced narrative patterns without requiring extensive labelled datasets (see e.g. heinrich et al., 2024). the task is additionally complicated by terminological uncertainty. over the years, scholars have proposed various interpretations for the very term narrative itself, reflecting the difficulty in reaching a consensus (see e.g. santana et al., 2023). chatman (1980) e.g. offers a structuralist perspective, defining narratives as comprising a story (a chain of events and characters) and discourse (how the content is communicated). @xcite riedl and young (2010) see narratives and storytelling as cognitive tools for making sense of the world. broader definitions highlight that narratives are sequences of events that form a cohesive whole, with significance derived from the relationship between events. consequently, detailed annotation of narratives usually comprises key features such as participants, events, and time (silvano et al., 2021). in subtask 2 of semeval-2025 task 10 (piskorski et al., 2025) narrative annotation is provided as coarse-and fine-grained labels given to whole news articles. the provided data covers news in five different languages, namely english, portuguese, bulgarian, russian, and hindi, and the task is to automatically annotate texts with all (sub-)narratives in a multi-label fashion. the narratives belong to two macro topics: climate change and the war in ukraine, both prime domains for fake news and disinformation intended to mislead the public. in our contribution, we compare a variety of state-of-theart approaches; all our code is publicly available. 2 2 data & labels ## system overview machine learning (ml) baseline we initially ran simple multi-label classification experiments with classical machine learning algorithms for all languages in order to get stronger baselines than the ones based on random guessing provided by the organisers. we use logistic regression (lr) and support vector machines (svm) on bags of words for this purpose. the baselines are computed separately for each language. prompt engineering llms (prompteng) we also implemented a structured approach of prompt engineering large language models (llms), namely gpt-4o (openai, 2024) and deepseek r1 -32b (guo et al., 2025) in a step-by-step fashion. the llms were tasked to proceed level by level and output the response in json format. ## experimental setup all our models make use of the combined training data during training and are evaluated on the development data. note that since most of our models were trained to predict fine-grained narratives, we inferred the corresponding coarse-grained narratives from the fine-grained labels using regular expressions. where applicable, we used multilingual models, which enables us to combine the full training data for all five languages to train a single model, thereby addressing the problem of data scarcity in fine-grained labels. @xcite ml we implement logistic-regression and support vector machine with a tf.idf-weighted feature matrix based on uni-and bigrams using scikit-learn (pedregosa et al., 2011). prompteng for prompt engineering, the llms are provided with the multi-level labels and prompted to choose a narrative first and only choose the subnarrative within that particular narrative second. we used a few-shot learning framework within this methodological sequence by providing some examples in english. sentsim for the zero-shot experiments, we split texts by double new lines into paragraphs and treat each paragraph as a sentence; for subnarrative description, we experiment with two approaches: using subnarrative descriptions as a whole, and splitting them into smaller segments (which mostly correspond to single sentences, but can also comprise two or three sentences). we then calculate sentence embeddings using four different multilingual sbert (reimers and gurevych, 2019) models available off-the-shelf foot_8 from hugging face, namely ## results we list results on the development set for all our systems in ## discussion & perspectives the results of our experiments suggest that traditional machine learning approaches provide little utility in the given task, as their performance remains significantly below that of more advanced methods. in contrast, our sentence-similarity based zero-shot approach proves to be a viable alternative, delivering competitive results without the need for domain-specific training. nonetheless, while the zero-shot method offers a cheap alternative, finetuning masked llms remains the most effective approach, given a large amount of labelled data as in the task here. we also note that incorporating multiple languages into a single training set is a reasonable strategy, likely due to shared linguistic patterns across languages. for future work, several directions could enhance classification performance. firstly, since many annotated texts contain lengthy narratives including irrelevant sections, preprocessing techniques that extract relevant portions before model training could reduce noise and improve classification accuracy for all models. similarly, setfit could benefit from individually labelled paragraphs; it might also perform better as an ensemble of several models to predict fine-grained labels for each previously predicted coarse-grained label. for sentsim, an iterative refinement process for narrative descriptions could enhance model interpretability and predictive accuracy by incrementally improving label definitions and annotations. while prompt engineering llms was not a successful strategy here, chain-of-thought prompting could improve the results (wei et al., 2023). findings from previous work (jiang et al., 2024; qi et al., 2024; he et al., 2024) suggest that llms have difficulty following complex instructions. given that our task requires multi-label classification at various levels and a strict json output, this approach might not be sufficient to address the task's complexity. applying the narrative attention layer to the multi-label hierarchical model might strengthen the relationship between coarse-and fine-grained hierarchies. this enhancement can improve the model's ability to predict subnarratives which correspond to the classified narratives. another possibility would be to experiment with the hierarchical loss by minimising the subnarrative loss only. we assume that focusing on subnarratives might improve the performance of our approaches. finally, after a thorough error analysis of the individual models presented here, it would be straightforward to use an ensemble model which could potentially improve overall performance.