UNEDT eam at S em E val-2025 Task 10: Zero-Shot Narrative Classification (2025)

Abstract: In this paper we present our participation in Subtask 2 of SemEval-2025 Task 10, focusing on the identification and classification of narratives in news of multiple languages, on climate change and the Ukraine-Russia war. To address this task, we employed a Zero-Shot approach using a generative Large Language Model without prior training on the dataset. Our classification strategy is based on two steps: first, the system classifies the topic of each news item; subsequently, it identifies the sub-narratives directly at the finer granularity. We present a detailed analysis of the performance of our system compared to the best ranked systems on the leaderboard, highlighting the strengths and limitations of our approach.

## introduction the characterisation and extraction of narratives from news texts is an area of growing interest in natural language processing (nlp), with applications in discourse analysis, bias detection and the understanding of social and political dynamics. in this context, semeval-2025 task 10, entitled 'multilingual characterization and extraction of narratives from online news' @xcite , seeks to advance the identification and classification of narratives in news stories in multiple languages. this task is structured in three main subtasks: subtask 1: entity framing, which consists of assigning one or more roles to each named entity mention within a news article, using a predefined taxonomy of roles; subtask 2: narrative classification, where each news article must be assigned all relevant sub-narrative labels within a hierarchical taxonomy of narratives in a specific domain; and subtask 3: narrative extraction, which requires generating a free-text explanation justifying the selection of an article's dominant narrative, based on fragments of text supporting that choice. recent advancements in large language models (llms) have significantly improved the ability to detect and analyze narratives and propaganda techniques in textual data. leveraging their contextual understanding and capacity to generalize from large-scale datasets, llms have been applied to identify persuasive strategies, ideological framing, and coordinated messaging in political discourse and media @xcite . our work focuses exclusively on subtask 2. the task presents multiple challenges, including linguistic diversity, subjectivity in categorising narratives and the limited availability of labelled data in multiple languages. addressing these problems requires robust approaches that can generalise well across languages and domains. to this end, we opt for a zero-shot approach, in which we leverage the knowledge of an llm without performing specific training on the task data. since some llms have been trained with varying amounts of data in the five languages of the task and sota models often performs better in english, we performed a machine translation into english using opus-mt models prior to inference. this paper describes in detail our methodology, the results obtained in comparison with the best competing models and a critical analysis of the performance. finally, we discuss the limitations of our approach and propose future directions for improving automatic narrative classification in multilingual contexts. ## subtask description semeval-2025 task 10 subtask 2 focuses on the identification and classification of narratives in news articles covering multiple languages and subject domains. the task is multilingual in five different languages: english, bulgarian, portuguese, hindi and russian. having a large number of languages avoids linguistic and cultural biases in the classification models, allowing the systems to be more robust and adaptable to different contexts. moreover, the presence of languages rarely used in llm training, such as bulgarian or hindi, introduces additional challenges in identifying and structuring narratives and enriches the evaluation of the performance of multilingual and multicultural models. also, the subtask focuses on two thematic domains of great current relevance such as the war between ukraine and russia and climate change. both topics generate a large amount of content on social networks and in the media, which allows us to analyse the propagation of narratives in contexts of high political, economic and social impact. moreover, by covering both a geopolitical conflict and a global environmental crisis, different types of narratives are covered: some focused on politics and war, and others on science, economics and sustainability. similarly, including news in russian is particularly relevant for the analysis of the ukraine-russia conflict. russian media often offer a different perspective than western media, which makes it possible to study how narratives are constructed and disseminated within russia and internationally. ## methodology in this research, we have decided to use a zero-shot approach to news classification, without using pre-trained examples. this decision responds to the difficulty of finding sufficiently large datasets labelled by human annotators to train a model when we move to real world scenarios. by employing a zero-shot approach, we take advantage of the power of llms, which have been trained on a large amount of data and are able to generalise to new tasks without the need for prior examples. to overcome the linguistic limitations that llms may have, we have translated all news items into english using machine translation models based on opus-mt (tiedemann), an initiative of the university of helsinki that provides multilingual machine translation models, facilitating translation across languages. this decision is justified by the fact that not all available llms have been trained with a large amount of data in the five languages of interest. by translating the articles into english, we seek to maximise the accuracy of classification and narrative detection, taking advantage of the knowledge of the models in this language. in addition, this methodology allows for a more homogeneous comparison of the news items, regardless of the original language but adding possible errors in the translator models. ## results to evaluate the performance of our zero-shot approach, we performed an initial evaluation on the training and development sets, which allowed us to analyse the model's ability to correctly classify both the topic and the subnarratives. we then applied the model to the test set, generating the final inferences. ## discussion the obtained results show a variable performance of the model depending on the language, with significant differences between the dev set and the test set. in the evaluation on dev, it is observed that the model achieves a high f1 topic in all languages, with values above 87%, indicating that topic classification (urw or cc) is relatively straightforward for the model. however, f1 other is much lower, especially in portuguese (pt), where the model did not correctly identify any news items in the other category, suggesting that the model's ability to detect news items that do not fit the predefined narratives varies by language. as for the classification of sub-narratives, f1 samples values show moderate performance, with bulgarian (bg) as the best performing language (0.4248), while hindi (hi) and portuguese (pt) show the lowest values (0.2305 and 0.2257, respectively). analysing the results on the test set, a generalised decrease in ranking metrics is observed with respect to the dev set, indicating that the distribution of labels in the test set might be slightly different from the dev set. comparison with the best model reveals noticeable performance differences. for example, in russian (ru), our model obtains an f1 coarse of 0.513, while the best model achieves 0.709, which represents a difference of 0.196 points. similarly, in portuguese (pt), the difference is 0.127 points. one aspect to note is that the difference in f1 samples (classification of subnarratives) is larger than in f1 coarse, indicating that the identification of subnarratives remains a greater challenge than the classification of narratives. furthermore, the standard deviation of our model and the best ranked model across all languages remains relatively high, suggesting considerable variability in the quality of predictions. this behaviour is especially visible in english (en) and hindi (hi), where the standard deviation values are the highest, suggesting that the model is less consistent in these languages. to better interpret these results, it would have been useful to have a measure of the inter-annotator agreement when constructing the dataset. knowing the inter-annotator agreement would allow contextualising the f1 values and standard deviations, providing a reference on the intrinsic difficulty of the task. if inter-annotator agreement were low, this would indicate that even for humans the classification of certain news items into specific narratives is ambiguous, which would help to establish a reasonable threshold for evaluating the model's performance. on the other hand, if the agreement were high, the observed variability in the model's predictions could be attributed mainly to limitations in its generalisability. this analysis would be particularly relevant in languages with higher variability in f1 and high standard deviations, as it would allow distinguishing between problems arising from annotation ambiguity and model-inherent errors. ## conclusion and future work in this research, we have presented a llm-based system using a zero-shot approach for the se-meval 2025 task 10 subtask 2, together with a machine translation into english, focused on the classification of narratives in news stories from five different languages. our system approached this task without using training data, using only the ability of the pre-trained model to identify the gen the results obtained show that subnarrative classification remains a challenge with low performance and high variability in predictions. comparison with the best model of the semeval task 10 subtask 2 shared task shows that our system performs worse with noticeable differences, especially in languages such as russian and portuguese. these results suggest some directions for future work. first, an analysis of the importance of machine translation would allow us to quantify the degree of error introduced and its effect on classification. it would be interesting to explore direct classification without translation in those languages with sufficient coverage in the base models. in addition, to obtain a more reliable estimate of model performance, it would be necessary to perform multiple runs on the test set and employ ensemble techniques, combining predictions from several model runs to reduce variability and improve the robustness of the system. finally, the data provided in the training and development set could be used to test supervised approaches such as few-shot or fine-tuning.