Comparaison des approches basées sur BERT et sur l’agent LLM pour la classification hiérarchique de narratifs dans les articles de presse multilingues (2025)

Abstract: Nous présentons une étude comparative de deux paradigmes de classification hiérarchique multi-labels de texte dans le contexte de l’extraction des narratifs d’articles de presse. La première approche utilise un cadre séquentiel basé sur BERT qui identifie les narratifs et leurs sous-narratifs correspondants. La seconde utilise des agents LLM spécialisés, chacun effectuant une classification binaire pour des catégories narratives spécifiques. En évaluant les deux approches sur l’ensemble de données SemEval-2025 Task 10 dans cinq langues, nous constatons que l’approche basée sur BERT offre une efficacité de calcul et des performances interlinguistiques cohérentes (moyenne F1 macro : 0,475), tandis que la méthode basée sur les agents démontre une meilleure gestion des narratifs nuancés et de meilleures performances sur les données en anglais (F1 macro : 0,513). Notre analyse révèle des forces complémentaires entre ces paradigmes. Nous discutons des implications pratiques et proposons des orientations pour des systèmes hybrides potentiels.

## introduction the computational analysis of narratives in text has emerged as a critical area in natural language processing (nlp), with applications ranging from misinformation detection to media analysis and social science research. in this context, narratives refer to coherent interpretive frameworks that organize information around specific perspectives, claims, or themes, creating systematic patterns in how events and issues are presented @xcite . unlike simple topics or categories, narratives encompass complex interrelations of actors, events, and evaluative framing, making their automatic identification particularly challenging. media narratives play a crucial role in shaping public discourse, especially in domains such as climate change reporting and geopolitical conflict coverage. automatically identifying these narratives enables researchers to track narrative patterns at scale, offering insights into media ecosystems and information flows that would be impossible to analyze manually. the semeval-2025 task 10 @xcite launches a challenge of automatic detection and classification of narratives in media. it defines a two-level hierarchical taxonomy of narratives (broad categories) and subnarratives (finer-grained distinctions) in multilingual news articles about climate change (cc) and the ukraine-russia war (urw). this task presents distinct computational challenges that extend beyond conventional text classification : two distinct paradigms have emerged for addressing such challenges. traditional approaches relied on supervised learning with feature engineering, while more recent methods leverage transformer architectures like @xcite . concurrently, the emergence of large language models (llms) has enabled new paradigms based on specialized agents that can perform targeted classification tasks through carefully crafted prompts. these approaches represent fundamentally different strategies : bert models require extensive supervised training but offer efficient inference, while llm-based approaches promise zero-shot capabilities but potentially with higher computational demands. in this paper, we present and compare these two approaches in the context of the subtask 2 of @xcite , our contributions include : our findings reveal an important trade-off : while the bert-based approach offers more consistent performance across languages (average f1 macro : 0.475), the agent-based approach excels at capturing nuanced narrative distinctions and demonstrates superior performance on specific languages, particularly english (f1 macro : 0.513). notably, we find that language-specific performance variations correlate with the semantic complexity of narrative distinctions rather than simply with resource availability, suggesting different underlying capabilities between the two paradigms. these results highlight the complementary nature of the approaches and suggest potential avenues for hybrid systems that combine their strengths. the remainder of this paper is organized as follows : section 2 discusses related work in narrative classification and transformer-based text classification. section 3 formally defines the problem. section 4 details our methodologies. in section 5 we report our results presents results and discussion, followed by conclusions in section 7. ## related work computational narrative analysis has emerged as a critical research area, particularly in the context of news media and information environments. unlike traditional topic classification, narrative classification focuses on identifying interpretive frameworks that organize information around specific perspectives, claims, or thematic structures @xcite . early work in this domain focused primarily on identifying narrative structures in literary texts @xcite . however, recent research has shifted toward analyzing narratives in news media, particularly in the context of misinformation detection (gruppi et al., 2020) and political discourse analysis @xcite . the semeval shared tasks have been instrumental in advancing computational narrative analysis. semeval-2019 task 4 introduced news detection (kiesel et al., 2019), while semeval-2021 task 6 focused on detection of persuasive techniques in texts (dimitrov et al., 2021). building on this foundation, semeval-2025 task 10 (piskorski et al., 2025) proposes tasks to specifically address hierarchical narrative classification in multilingual news articles, presenting unique challenges in cross-lingual narrative understanding. we use this task to validate our approaches. ## problem definition we address the semeval-2025 task 10 challenge of automatically identifying and classifying narratives in multilingual news articles. narratives, in this context, refer to coherent interpretive frameworks that organize and present information through specific perspectives, claims, or thematic structures. unlike simple topic categorization, narrative classification captures how events are framed and which claims are emphasized. the task is formulated as a multi-label, multi-class hierarchical text classification problem with two distinct levels : ## methodology in this section, first, we present our two complementary approaches for hierarchical narrative classification : a bert-based supervised model and an llm-based agentic framework. we detail the rationale behind each approach and how they address different aspects of the multilingual narrative classification challenge. ## results and discussion this section presents a comparative analysis of our bert-based and agent-based approaches for hierarchical narrative classification. we evaluate performance on the semeval-2025 task 10 benchmark across five languages, examining both macro-level and sample-level f1 scores. these metrics are the ones officially used by the challenge organisers. statistical analysis reveals that performance differences are primarily attributed to linguistic distance from the source training data and translation quality. portuguese, despite being typologically distant from the training languages, achieves reasonable performance (0.491), suggesting effective cross-lingual transfer through translation-based augmentation. the agent-based system demonstrates a stark performance disparity between english and other languages. while achieving competitive results on english (0.513, securing third place in the competition), performance deteriorates substantially for portuguese (0.285) and russian (0.247), representing performance drops of 44% and 52% respectively. translation-induced semantic shifts significantly impact the agents' reasoning capabilities. despite operating on translated text, the agents' pattern recognition was substantially weaker for non-english content, indicating that narrative framing markers are partially lost during translation. ## conclusion this study provides a comprehensive empirical comparison of bert-based supervised learning and llm-based agent approaches for multilingual narrative classification. through systematic evaluation across five languages using the semeval-2025 task 10 benchmark.our analysis demonstrates that the bert-based approach achieves consistent cross-lingual performance with computational efficiency , while the agent-based approach provides superior english performance and interpretability at significantly higher computational cost. the main differences between bert-based supervised methods and proxy zero-shot large language model (llm) methods are summarized in this study established a comparative experimental framework for multilingual text classification, incorporating both traditional supervised learning algorithms and deep learning methods based on large language models to analyze and evaluate the classification performance of different technical approaches in cross-lingual scenarios. identifying specific scenarios where each approach demonstrates clear advantages. future research should focus on hybrid architectures that leverage the complementary strengths identified in this analysis, particularly addressing the computational efficiency limitations of agent-based approaches while maintaining their interpretability advantages. dimension bert-based method agentic zero-shot llm method training requirements requires extensive labeled data and fine-tuning no training needed ; relies on taskspecific prompts label expansion new labels necessitate model retraining easily extendable by updating prompts and adding new agents flexibility limited adaptability due to fixed model architecture highly flexible due to modular agent structure multilingual support requires preprocessing translation native multilingual handling without preprocessing imbalanced class handling bias toward frequent classes reduced bias as each agent manages its class independently scalability resource-intensive when training multiple classifiers scalable by adding additional agents