Irapuarani at S em E val-2025 Task 10: Evaluating Strategies Combining Small and Large Language Models for Multilingual Narrative Detection (2025)

Abstract: This paper presents the Irapuarani team’s participation in SemEval-2025 Task 10, Subtask 2, which focuses on hierarchical multi-label classification of narratives from online news articles. We explored three distinct strategies: (1) a direct classification approach using a multilingual Small Language Model (SLM), disregarding the hierarchical structure; (2) a translation-based strategy where texts from multiple languages were translated into a single language using a Large Language Model (LLM), followed by classification with a monolingual SLM; and (3) a hybrid strategy leveraging an SLM to filter domains and an LLM to assign labels while accounting for the hierarchy. We conducted experiments on datasets in all available languages, namely Bulgarian, English, Hindi, Portuguese and Russian. Our results show that Strategy 2 is the most generalizable across languages, achieving test set rankings of 21st in English, 9th in Portuguese and Russian, 7th in Bulgarian, and 10th in Hindi.

## introduction trusting online content has become increasingly difficult due to the rise of misinformation, disinformation, deceptive content, and deliberate attempts at manipulation @xcite . not only is it more challenging to distinguish between credible information and fake news, but the sophisticated techniques used to shape perceptions can intensify conflicts and influence political opinions, potentially swaying voter behavior @xcite . the vast amount of online disinformation highlights the urgent need for automated tools to identify such content @xcite . our research is centered on semeval-2025 task 10 @xcite , which addresses the multilingual characterization and extraction of narratives from online news. specifically, we focus on subtask 2, which involves classifying narratives and sub-narratives within a two-level taxonomy. the primary objective of the task is to foster the development of classification methodologies capable of identifying narratives designed to manipulate readers, instantiated this year in the domains of climate change and the ukraine-russia war. additionally, the task provides resources and enables participants to work in at least one of five languages: bulgarian (bg), english (en), hindi (hi), portuguese (pt), and russian (ru). our team, however, has chosen to evaluate our approaches across all available languages, aiming to achieve a multilingual analysis. we evaluated three distinct methodologies leveraging both small language models (slms) and large language models (llms) to address this task. moreover, we intend to assess whether the strategies exhibit generalizability across different languages, pursuing a general framework rather than a language-specific strategy. the approaches are as follows: (1) a direct classification method employing a multilingual slm; (2) a translationbased approach, where texts in multiple languages were translated into a single target language using an llm, followed by classification with a monolingual slm; and (3) a hybrid strategy that integrated the strengths of both model types, utilizing an slm for domain filtering and an llm for hierarchical label assignment. our experiments on the development set show that strategy 2 is the most generalizable across languages among the approaches we evaluated, ranking 8th in bulgarian, 9th in portuguese, 10th in russian, 11th in hindi and 22st in english on the test set 1 . creasingly leveraged the advanced capabilities of language models. encoder-based slms have been successfully employed in narrative classification tasks. for instance, @xcite explored combining the roberta model @xcite with the traditional machine learning algorithm logistic regression, utilizing a taxonomy within the climate change domain. similarly, @xcite , working in the context of covid-19, reported success with a fine-tuned bert @xcite model for classifying narratives. in addition to encoder-based slms, llms have also been applied in analyzing narratives and propaganda @xcite . @xcite experimented with gpt-4 for annotating spans of propaganda in arabic news articles, highlighting the model's potential when provided with additional contextual information. jones (2024) evaluated gpt-3.5-turbo's performance in identifying up to 18 possible persuasion techniques in news articles, reporting promising results while noting that the model's ability to detect these techniques varied across some categories. furthermore, @xcite compared the performance of roberta with gpt-3 and gpt-4 for propaganda detection, emphasizing that gpt-4 ranked among the best-performing models, alongside roberta. our work aims to evaluate strategies that integrate both slms and llms for classifying narratives in news articles within a multilingual context. moreover, we aim to determine whether the strategies exhibit generalizability across different languages, pursuing a general approach over a narrowly specialized one. more details are in the sections below. ## background the data utilized in this work was provided by the semeval-2025 task 10, which comprises a multilingual corpus of news articles. the corpus spans articles collected between 2022 and mid-2024, focusing on two primary topics: the ukraine-russia war and climate change. in the context of the addressed subtask, namely subtask 2, the data also includes labels associated with narrative classification, structured into a two-level hierarchy dataset based on the provided annotations @xcite . the ukraine-russia war (urw) domain includes 11 narrative labels and 38 sub-narratives. in contrast, the climate change (cc) domain has 10 narrative labels and 36 subnarratives. the label other can be used at the narrative level to indicate that a narrative does not match any available labels. it can also be paired with a narrative label to indicate that the corresponding sub-narrative does not fit the predefined categories. finally, the task organizers pre-divided the set as outlined in ## system overview this section outlines the methodology employed for the multilabel classification of narratives within the task's two-level hierarchy. based on the data characteristics previously described, we evaluated three strategies to ensure comprehensive label and language coverage. the (a) single-model strategy, following @xcite , uses a multilingual slm to classify narratives without considering their hierarchical structure. the (b) translation strategy involves translating texts into a single target language with an llm, followed by classification using a monolingual slm. lastly, the (c) hierarchical strategy applies a hybrid approach: an slm first classifies texts into urw or cc domains, guiding an llm to assign the final label based on the hierarchy. ## experimental setup implementation details the proposed solution was developed utilizing the hugging face transformers @xcite and scikit-learn @xcite libraries, using the multil-abelbinarizer approach. all experiments were conducted on two nvidia rtx 4090 gpus, each featuring 24gb of vram. models hyperparameters for the classification with the deberta models, the following hyperparameters were employed: batch @xmath0, number of @xmath1, maximum sequence @xmath2, learning @xmath3, and weight @xmath4.01. predictions were made using a threshold of 0.8 for logits, considering the number of labels and the multi-label classification setup. for classification with gpt4o-mini, the configuration included a temperature of 0.7, top-@xmath5.95, and maximum completion @xmath6. lastly, for translation, the parameters were set as follows: max new @xmath7, do @xmath8, @xmath9.8, and top-@xmath10.95, lastly, all random seeds were set to 42 wherever applicable. the training and inference parameters were selected based on a 5-fold cross-validation performed on the training set. evaluation metrics the evaluation metrics employed are based on the sample-level f1 score, with an emphasis on f1 samples , which focuses on subnarratives, rather than f1 coarse , which targets the narratives-level, in alignment with the official task directives. ## results this section aims to analyze the proposed strategies. first, all proposed strategies outperform the baseline. notably, the most basic approach -the single model strategy -achieves its best results in english and bulgarian, while ranking among the least effective solutions for the other languages. this pattern may indicate that the need for a single classifier to adapt simultaneously to language representation and classification itself during training may hinder its ability to generalize performance across languages. on the other hand, the translation-based strategy yields the best results among the proposed approaches for portuguese (with translations generated by the aya model), as well as bulgarian and hindi (with translations generated by the gpt-4o-mini model). additionally, the aya-translated approach secures second place in hindi, while the gpt-based variant achieves this same ranking for portuguese and russian. notably, translationbased strategies do not exhibit abysmal performance in any language -unlike, for example, the single model strategy in hindi. this observation suggests that translation-based approaches may enhance generalization by allowing the final classifier to focus solely on the classification, rather than concurrently handling multilingual representation. regarding the performance differences between the aya and gpt-4o-mini translations, a more detailed analysis of their pre-training corpora could offer valuable insights. however, such resources are not publicly available for the gpt model. despite achieving the best performance in english and russian, the hierarchical strategy demonstrated poor portuguese results and was outperformed by the translation-based strategies in other languages. an analysis of the first stage of the hierarchical strategy on the validation set (as test labels are not available) suggests that the results may be influenced by a specific characteristic in por-tuguese data. the table also shows performance for russian, which, despite exhibiting a higher number of absolute classification errors compared to portuguese, yielded better results in the final classification stage, as shown in consequently, aiming to evaluate the most generalizable approach, our final submission was based on the translation-based strategy utilizing the gpt-4o-mini model. in the test set, the submitted approach once again outperformed the baseline across all languages, consistently avoiding any notably poor results in any of them. this further reinforces its potential for generalization. future research may also consider a qualitative evaluation of the translations, which we regard as beyond the scope of the current work. ## conclusion this work addresses semeval-2025 task 10 and evaluates three distinct strategies for multilabel classification within a two-level taxonomy of narratives and sub-narratives in online news. our results indicate that the approach relying solely on a multilingual slm to classify texts in multiple languages failed to generalize its strong performance in languages such as english to other linguistic contexts. similarly, the strategy that employed a multilingual slm as a domain filter and an llm to assign the final labels achieved the best result on the development set for english but performed poorly in languages such as portuguese, also highlighting a generalization gap. conversely, the approach that translated all texts into english and utilized a monolingual slm for classification demonstrated more consistent and generalizable results across languages, frequently ranking as the best or secondbest performing strategy among those we analyzed. future work may explore the evaluation of additional combinations of slms and llms to identify the most effective pairings for this task. furthermore, the translation-based approach, which demonstrated robust generalization, could be extended by translating texts into target languages other than english. this would enable a more comprehensive analysis of the impact of the translation step on classification performance across diverse linguistic contexts.