Team INSA ntive at S em E val-2025 Task 10: Hierarchical Text Classification using BERT (2025)

Abstract: In this paper, we propose a BERT-based hierarchical text classification framework to address the challenges of training multi-level classification tasks. As part of the SemEval-2025 Task 10 challenge (Subtask 2), the framework performs fine-grained text classification by training dedicated sub-category classifiers for each top-level category. Experimental results demonstrate the feasibility of the proposed approach in multi-class text classification tasks.

## introduction with the rapid development of natural language processing (nlp), extracting narratives from online news has attracted widespread attention in both academia and industry @xcite . a deep understanding of how entities are presented in news articles and the identification of underlying narratives are crucial for media analysis, misinformation detection, and socio-political research @xcite . this study focuses on the cross-lingual, multilabel, and multi-category document classification task, which involves automatically identifying and assigning narrative labels and sub-narrative labels to news articles based on a two-level narrative labeling system within a specific domain. specifically, each article may contain one or more narrative labels, with each narrative further subdivided into sub-narrative labels. the main objective of this study is to accurately assign all applicable narrative labels and their corresponding sub-narrative labels to each article. the task covers five languages (bulgarian, english, hindi, portuguese, and russian) and aims to evaluate narrative classification performance under cross-lingual conditions. semeval-2025 task 10 @xcite focused on multilingual characterization and narrative extraction from online news, divided into three independent subtasks. we participated in the subtask 2 on the narrative classification. in this task, our model achieved a mid-to-upper range ranking on the final leaderboard, outperforming the baseline systems while still leaving room for further improvement. our approach combines large-scale pre-trained language models, a hierarchical classification strategy, and entity framing analysis to automatically identify and classify narratives across different languages and topics, providing a solution for narrative extraction from multilingual news texts. in this study, we first perform translation-based data augmentation on the raw text data to ensure label consistency and accuracy across multiple languages. we then fine-tune a transformer-based language model on the augmented dataset. to identify sub-narrative labels, we train separate classification models for each sub-narrative label and combine these models into an ensemble model. after completing the narrative label classification, we calculate the probability distribution of each subnarrative label under its corresponding main narrative label and select the sub-narrative label with the highest probability as the classification result. ## background narrative classification is of great importance for extracting and identifying narrative structures from various types of texts, and it plays a crucial role in fields such as computational linguistics, nlp, and information retrieval (ir). in recent years, researchers have shifted their focus from traditional personal narratives @xcite to more diverse text types, especially informational texts (e.g., news reports, meeting minutes, and case analyses), and have made significant progress in developing computational methods for identifying narratives. classical narrative theory was initially proposed by @xcite . building on this work, swanson et al. @xcite manually annotated texts related to personal stories, categorizing clauses into three narrative types (orientation, evaluation, and action) and developed corresponding feature-based models. this endeavor laid an important foundation for subsequent research in narrative classification. as the demand for automatic recognition of narrative structures continues to grow, there has been increasing interest in integrating narrative theory with machine learning models, aiming at achieving more efficient and accurate narrative classification across a variety of text types. saldias and roy (saldias and roy, 2020) employed convolutional neural networks (cnns) to classify sentences in personalstory texts, automatically labeling each sentence according to the three narrative types proposed by @xcite introduced neat (narrative elements an-notation), which uses multiple supervised learning models to distinguish highly interrelated narrative categories. hatavara et al. @xcite further developed a rule-based and computational approach to systematically extract narratives from parliamentary records and oral history interviews, demonstrating its feasibility on large datasets. recent surveys have provided comprehensive overviews of current methods and challenges in narrative extraction @xcite . meanwhile, large-scale pre-trained language models, especially bert @xcite with their pre-training on bidirectional language models @xcite , are capable of better understanding contextual information within sentences and have thus been widely applied to narrative classification tasks @xcite @xcite . however, challenges still persist due to data scarcity and the inherent complexity of narrative structures, motivating researchers to explore the use of large language models (llms) for data augmentation @xcite . although both gpt @xcite and bert are large-scale pretrained language models, gpt, as a generative model, has the ability to produce coherent and contextually relevant text. thus, it offers novel solutions to address data insufficiency, particularly in the generation of high-quality narrative texts @xcite and translating narrative records @xcite . ## experimental setup dataset description. the semeval-2025 task 10 subtask 2 dataset consists of news articles in five different languages: english (en), portuguese (po), russian (ru), bulgarian (bu), and hindi (hi). each article is annotated with one or more narrative labels from a predefined set of 21 top-level narratives, and each narrative is further associated with one or more sub-narratives from a total of 91 possible sub-narratives. for our experiments, we used the multilingual corpus with varying document distributions across languages (see figures 1 2 ). prior to data augmentation, the training set contained 399 en articles, 400 po articles, 133 ru articles, 401 bu articles, and 366 hi articles. the development set consisted of 41 en articles, 35 po articles, 32 ru articles, 35 bu articles, and 35 hi articles. our test set included 101 en articles, 100 po articles, 60 ru articles, 100 bu articles, and 99 hi articles. the dataset exhibits significant class imbalance at both narrative and sub-narrative levels, as shown in figures1 and 2. some narratives like "criticism of institutions and authorities" and "discrediting ukraine" appear frequently, while others like "climate change is beneficial" and "controversy about green technologies" are rarely represented. this imbalance presents a challenge for classification models, particularly in identifying and correctly classifying minority classes. to address the imbalance in training data across languages, particularly the limited number of ru articles, we implemented a translation-based data augmentation strategy. after augmentation, each language in the training set contained 1,699 articles, creating a balanced training corpus across all five languages. this augmentation approach enabled our model to learn more robust cross-lingual patterns and improved overall performance, especially for languages with fewer original training samples. implementation details. we implemented our model using pytorch (1.10.0) and the hugging face transformers library (4.16.2). for the text translation framework, we employed openai's gpt-4o via the official api. documents were segmented into chunks of approximately 1,000 tokens to stay within api limits while maintaining context coherence. for our augmentation process, we translated non-english articles to english and vice versa to ensure balanced representation across languages. bert-based multi-label classification model: (a) pre-trained model: bert-base-uncased, (b) maximum sequence length: 128 tokens, (c) batch size: 8 (narrative classifier) / 3 (sub-narrative classifiers), (d) learning rate: 2e-5 with adamw optimizer, (e) weight decay: 0.01, (f) training epochs: 30 (early stopping with patience of 5) for narrative classifier / 3 for sub-narrative classifiers, (g) dropout rate: 0.1, (h) classification threshold: 0.2 (optimized on validation set) for narrative classifier. to validate the effectiveness of our data augmentation approach, we conducted experiments both with and without the augmented data. the results demonstrate significant performance improvements when using the augmented dataset, particularly for languages with fewer original training samples like russian (see the training process for the narrative and subnarrative classifiers was structured as follows. we first trained the top-level narrative classifier on the full augmented dataset of 1,699 articles per language. for each narrative category, we filtered the dataset to include only articles with that narrative label. we then trained a dedicated sub-narrative classifier for each narrative category using a singlelabel classification approach with labelencoder. during inference, we first predicted the narrative labels using the top-level classifier, then employed the corresponding sub-narrative classifiers to predict the fine-grained labels. for the sub-narrative classification, we organized articles by their top-level narrative labels and trained separate bert-based models for each toplevel category. unlike the multi-label approach used for narrative classification, each sub-narrative classifier was trained as a standard single-label classification model using cross-entropy loss. this approach was chosen due to the hierarchical nature of the labels and to reduce complexity in the subnarrative prediction task. all sub-narrative models were saved with their corresponding tokenizer and label encoder to enable efficient inference. during prediction, once the top-level narrative was identified, the corresponding sub-narrative model was loaded to predict the specific sub-category. this hierarchical approach allowed us to effectively handle the large number of potential sub-narratives while maintaining computational efficiency. to address the class imbalance issue in the toplevel narrative classification, we implemented class weighting in the loss function, where weights were figure 2: distribution subnarrative labels inversely proportional to class frequencies in the training set. this approach improved the model's ability to identify minority classes without significantly degrading performance on majority classes. evaluation metrics. following the official semeval-2025 task 10 evaluation criteria, we measured our model's performance using the following metrics: (a) f1 macro: the unweighted mean of f1 scores for each class, giving equal importance to all classes regardless of their frequency. this metric is particularly important for evaluating performance on imbalanced datasets, as it prevents the model from being overly biased toward majority classes. (b) f1 samples: the f1 score calculated for each instance and then averaged, which accounts for the multi-label nature of the task. this metric provides insights into the model's ability to correctly predict all relevant labels for each document. additionally, we report the standard deviation (st.dev) for both metrics to analyze the stability of our model's performance across different classes and samples. a lower standard deviation indicates more consistent performance across all categories, which is desirable for robust classification systems. ## conclusion in the multi-label setting, the framework integrates a bert-based text classification method, using automated data processing, optimized training workflows, and memory management strategies. our proposed framework additionally provides a range of functional modules (segmentation, automated translation, and standardized output) that facilitate the generation of high-quality multilingual data for subsequent classification and semantic analysis. experimental results show that our method performs well in handling large-scale, multilingual text data and achieves high accuracy in hierarchical classification tasks. future research directions include: (1) further optimizing parallel processing strategies to improve overall training efficiency; (2) enhancing the accuracy of sub-category classification; and (3) exploring more powerful multilingual pre-trained models to strengthen system robustness and generalization capabilities.