i L ost T he C ode at S em E val-2025 Task 10: Bottom-up Multilevel Classification of Narrative Taxonomies (2025)

Abstract: This paper describes the approach used to address the task of narrative classification, which has been proposed as a subtask of Task 10 on Multilingual Characterization and Extraction of Narratives from Online News at the SemEval 2025 campaign. The task consists precisely in assigning all relevant sub-narrative labels from a two-level taxonomy to a given news article in multiple languages (i.e., Bulgarian, English, Hindi, Portuguese and Russian). This involves performing both multi-label and multi-class classification. The model developed for this purpose uses multiple pretrained BERT-based models to create contextualized embeddings that are concatenated and then fed into a simple neural network to compute classification probabilities. Results on the official test set, evaluated using samples $F_1$, range from $0.15$ in Hindi (rank #9) to $0.41$ in Russian (rank #3). Besides an overview of the system and the results obtained in the task, the paper also includes some additional experiments carried out after the evaluation phase along with a brief discussion of the observed errors.

## introduction online news is a primary source of information and has a major role in shaping public discourse and influencing perceptions. identifying the narratives embedded within news articles is crucial for critically analyzing their perspectives, biases, and underlying messages. for instance, this can be relevant in contexts where harmful or misleading content is present: recognizing the dominant narratives can facilitate the construction of counter-narratives @xcite , in view of promoting a more constructive and less toxic online debate. furthermore, given the abundance of information available online-from both mainstream and alternative media sources-understanding the stance underlying different narratives can be useful when navigating digital content. this requires not only recognizing the explicit claims made in a given article or social media post, but also understanding how such claims align with broader thematic views. a critical approach to narratives can help the interested reader distinguish between different perspectives and engage with news content in a more informed way. from a theoretical perspective, providing a shared framework for the definition and categorization of narratives and sub-narratives is essential @xcite for more systematic analyses and comparisons across different texts and media sources. in turn, automatic systems can build upon these theoretical foundations to detect and classify narratives with the help of natural language processing techniques @xcite . our work lays on these premises and it focuses on the task of narrative classification, proposed as part of semeval-2025 task 10 on multilingual characterization and extraction of narratives from online news @xcite . more in particular, the team participated in subtask 2 on narrative classification, which consists precisely in assigning one or more sub-narrative labels to a given news article in one among five languages (bulgarian, english, european portuguese, hindi and russian), and resorting to predefined narrative taxonomies. the news articles are centered around two main topics, ukraine-russia war and climate change, and each topic has its own taxonomy of narratives and corresponding sub-narratives. 1 this challenge aligns with prior research in text classification, particularly in multi-label and hierarchical classification tasks, such as task 4 from semeval 2024, which shares similarities @xcite . to address this task, our system combines a simple neural network with a concatenation of contextual embeddings generated using multiple bertbased models, each already fine-tuned for different tasks, though not specifically for this one. language models such as bert and its variations have demonstrated strong performance in various nlp tasks, using contextual embeddings to have good classification accuracy even without fine-tuning @xcite . furthermore, approaches resorting to vector concatenation to address similar tasks were implemented in the recent past, obtaining good results @xcite ; with similar foundations, we designed a lightweight neural architecture that aims to balance prediction accuracy with computational demand. the remainder of the paper describes the system architecture along with the experiment setup adopted for this task and an overview of the results obtained. ## system overview the model is a straightforward combination of several independent language models, previously finetuned for different tasks, whose results are then concatenated and fed into a neural network, as outlined in figure 1 . this approach has been structured into several steps and modules that are described below. data pre-processing each sentence of an article was extracted and treated as an individual data point (or sample), with the requirement that every sentence from the same article shares the full set of classes attributed to the entire article. this is motivated by the fact that bert models have a constrained token limit per sentence. treating each sentence independently, instead of using the article as a whole, allows to manage this limitation effectively, as each sentence can be processed within the model's token constraints, while still preserving the information on the associated narratives of the whole article. this approach also ensures more consistent results when using a voting system, which is indeed an additional component of the system, as explained later in this overview. embedding extraction every file undergoes an embedding extraction process. several extraction modes were tested to include a context window of previous content. after preliminary experiments, two configurations were ultimately employed for the purposes of this task, i.e., one without a context window and one with a window covering the two preceding sentences. various fine-tuned models based on bert architecture @xcite and available on huggingface were used. these models were employed only to extract textual embeddings, specifically the cls token embedding for each sentence. multiple instances of these models were used in parallel. it is worth pointing out that these models were selected because they were already fine-tuned for different (though relevant) tasks, but they were not retrained on this competition's dataset. ## dataset the dataset consists of different news articles and different type of texts about two main topics: climate change denial claims (abbreviated with the cc label) and propaganda in ukraine-russia war (abbreviated with the urw label). each article is linked to a set of exclusive narratives based on its topic, and each narrative is further associated with a group of sub-narratives. the dataset has been made available in five languages (with approximately 400-450 articles with golden labels and 100 unlabeled articles to submit per language), with the same taxonomy for each language, except for russian language where the cc topic was not present and there were far less articles (approximately 250 articles with golden labels and 60 unlabeled articles to submit). in general, 30-40 elements were used for development for each language. for further details on the dataset development and composition we refer the reader to the task report provided by the organizers @xcite . ## experiment setup the experiments were run on a laptop with an in-tel® core™ i7-1065g7 @ 1.30ghz cpu, 36 gb ram, cpu only. as described in section 2, a preliminary preprocessing step involved splitting each article into individual sentences. furthermore, all non-english datasets of the task were then automatically translated into english using microsoft translate api. the four models selected for feature extraction are roberta-large fine-tuned on the tweetemotion dataset for the emotion classification task foot_1 , @xcite deberta-v3-small foot_2 @xcite and deberta-v3-base foot_3 @xcite , both fine-tuned for nli tasks, and a distilbert model fine-tuned for named entity recognition foot_4 @xcite . on average, extracting the entire dataset for one language composed approximately of 400 samples takes 1-2 hours, which is why the extracted embeddings are saved and later used for experiments (this processing was not done in batch). concerning the development of the neural network, local experiments were conducted to determine its optimal configuration. the number of layers and neurons per layer was determined through extensive trial-and-error experimentation. in general, a single relu layer with a size 5-30 times smaller than the total number of input features-combined with a dropout rate of 0.3-0.4-was found to be sufficient, with the optimal layer size also depending on the language. increasing the number of layers often led to overfitting and slower training, and deviations from these values generally resulted in poorer performance. for example, using a different number of neurons caused the loss function on the development dataset to decrease more slowly and converge at higher values compared to the "optimal" configuration, while using too few neurons resulted in underfitting. overall, the development approach was to carefully adjust these parameters to optimize the neural network's classification performance on the loss function (binary cross entropy on the one hot encoding of classes of the sample), the objective was to reduce the loss on the validation dataset, or to decrease the training loss without causing an increase in the validation loss. the training process was divided into two phases. the first phase involved using a batch size of 128 sentences and training for 10 epochs, using the adam optimizer with keras default settings for rapid convergence. the second phase employed stochastic gradient descent with momentum (sgd) with default parameters, using a batch size equal to the entire dataset and a varying number of epochs (ranging from 1000 to 4000) with early stopping on loss function (to avoid increases on validation loss). this phased approach was chosen because adam allows the network to learn quickly but tends to overfit after too many epochs. in contrast, sgd with a large batch size learns more slowly but continues to improve the validation loss without overfitting as rapidly on the training dataset. no distinction between sentences of different articles were made as a random shuffle of all sentences was performed before the training phase. the final threshold was selected by running the system multiple times on the development set and choosing the threshold value that maximized the result according to the official ranking metric, which was samples f 1 score (further explained below). for what concerns the evaluation metrics adopted for subtask 2, submitted results were computed considering two measures: coarse f 1 , which reports how well the model predicts the narratives (without considering the sub-narratives), and samples f 1 , which instead measures how well the model predicts both narratives and sub-narratives, meaning that both aspects should be correct for the prediction to be considered correct. for both coarse and samples f 1 the values are first averaged at document level and then across all the documents of the set. standard deviation is finally included, in order to measure how much the scores vary across the documents. ## results in this section, we present the results of our experiments across all phases of the campaign, i.e. development, evaluation and post-evaluation, where the leaderboard was made available for participants to continue testing their models following the competition. ## discussion and error analysis while the bulgarian and russian datasets performed better than the other languages, despite being translated into english, surprisingly enough, the same model produced worse results on the original english dataset. upon the re-opening of the submissions, comparing the confusion matrices generated for the results on each development set (see appendix a) reveal that the model tends to produce a high number of false positives in certain classes. however, these classes also show a higher correct prediction rate, which suggests a significant class imbalance in the dataset. in contrast, some other classes are consistently missed, resulting in a high number of false negatives distributed across various classes, however the false negatives are low within each individual class, further confirming the imbalance. relatively very few cases have completely correct predictions. ## conclusions and future work the approach described in this paper consisted in feeding a simple neural network with a concatenation of multiple contextual embeddings from different fine-tuned bert-based models to tackle the challenges deriving from a multi-class and multilabel classification task. quantitatively, the model performed reasonably well in some languages, but underperformed in others, as the model seems to struggle to find optimal values of thresholding and it is sensitive to class definitions. in terms of possible improvements over this approach, we observe that the russian dataset-that only included urw-related topics-performed better than other languages, suggesting that developing separate classifiers for each topic (cc vs urw) might further improve results. additionally, using a single-language merged dataset approach could also yield better performance. another unexplored approach is a top-down hierarchical strategy. however, given that the narrative/coarse score was not particularly low across different languages, this approach may not be necessary.