IRNLP at S em E val-2025 Task 10: Multilingual Narrative Characterization and Classification (2025)

Abstract: Our system approach for multilingual narrative classification is basically based on XLM-RoBERTa Large and other bert-based models(e.g DeepPavlov, Neuralmind BERT), fine-tuned on different language datasets. To improve generalization and ensure robust performance across languages, we employed a repeated k-fold cross-validation strategy. This allowed us to maximize the use of available training data while mitigating potential overfitting issues. Our preprocessing pipeline included (1) language-specific tokenization, (2) hierarchical label structuring, and (3) dynamic batch sampling to balance label distributions. We optimized the model using the F1 macro and F1 samples metrics ,ensuring that the system’s predictions were well-calibrated for fine-grained multilingual classification. The results demonstrated that our approach effectively leveraged transformer-based architectures to model complex narrative structures across languages, with strong performance gains due to repeated k-fold evaluation.

## introduction narratives play a pivotal role in shaping public opinion and framing news reporting, often embedding persuasive messaging or ideological intent. automatically detecting such narratives is a complex task, particularly in multilingual settings, due to semantic ambiguity, class imbalance, and crosslinguistic variability. subtask 2 of semeval-2025 task 10 addresses this challenge by focusing on the classification of narratives and subnarratives in news articles across five languages. this paper presents the irnlp system, developed to address this challenge using multilingual transformer-based models. the system was trained on hindi, english, and russian datasets, leveraging repeated stratified k-fold validation to ensure robust evaluation. unlike standard approaches that rely on single-split validation, the use of repeated k-fold increases generalizability and minimizes overfitting. this work also contributes insights through error analysis and controlled ablation studies. ## related work previous work on fine-grained propaganda detection by @xcite introduced structured annotation strategies for identifying persuasive techniques. this laid the groundwork for related tasks such as narrative extraction and classification. multilingual transformer models like bert @xcite , roberta @xcite , and xlm-roberta @xcite have demonstrated strong performance across tasks such as sentiment analysis, named entity recognition, and text classification. hugging face's transformers library @xcite provides scalable implementations of these models and facilitates multilingual fine-tuning. few studies have focused explicitly on narrative modeling in multilingual contexts. work in stance detection and argument mining has highlighted the importance of modeling discursive structures, but the integration of coarse and fine narrative labels in low-resource settings remains under-explored. the irnlp system aims to fill this gap using repeated validation and tailored preprocessing for each language. ## ablation study the impact of major design choices was quantified in ablation experiments. the ablation results highlight the importance of repeated k-fold validation in preventing overfitting. removing this step led to a drop in f1 samples (from 0.515 to 0.472 in hindi). similarly, unbalanced mini-batches had a negative impact, likely due to dominance of majority labels during optimization. these findings confirm that both evaluation strategy and training stability significantly influence model effectiveness in low-resource settings. ## error analysis the most common source of error was label imbalance, which led the model to favor dominant narrative types while underpredicting rare ones. this was particularly evident in the hindi and russian datasets, where certain subnarratives appeared infrequently. additionally, semantic overlap between similar categories-such as foreign conspiracy and global threat-confused the model, often resulting in misclassification between conceptually adjacent classes. another recurring issue stemmed from the multilabel nature of the task. in some cases, the model correctly identified a coarse-grained narrative but failed to capture accompanying subnarratives, reducing f1 samples scores. this was especially noticeable in russian, where instance-level prediction was more challenging despite strong macro-level performance. these findings suggest that future iterations of the system may benefit from more balanced sampling strategies, label smoothing, and architectures that better capture inter-label dependencies. ## conclusion this paper presented the irnlp system for subtask 2 of semeval-2025 task 10. the system combined transformer models with repeated k-fold validation and language-sensitive preprocessing. results demonstrated robust generalization in multilingual narrative classification. future directions include incorporating contrastive loss, data augmentation for low-resource languages, and exploring semisupervised training.