HTU at S em E val-2025 Task 11: Divide and Conquer - Multi-Label emotion classification using 6 D ziri BERT s submodels with Label-fused Iterative Mask Filling technique for low-resource data augmentation. (2025)

Abstract: In this paper, the authors address the challenges of multi-label emotion detection in the Algerian dialect by proposing a novel Label-fused Iterative Mask Filling (L-IMF) data augmentation technique combined with a multi-model architecture. The approach leverages DziriBERT, a BERT variant pre-trained on Algerian text, to generate contextually and label-sensitive aug- mented data, mitigating class imbalance while preserving label consistency. The proposed method uses six independent classifiers, each trained on balanced datasets for dedicated la- bel, to improve performance. The results show significant improvement on mutli-label classification task using Deep Learning, with an F1 macro score of 0.536 on the validation dataset and 0.486 on the test dataset, the sys- tem ranked 28/41 on the Algerian dialect score- board; which is more than 7% higher than the task baseline using RemBERT.

## introduction dialectical arabic, like any low-resource language, offers many challenges in the areas of natural language processing. lack of high-quality annotated datasets for tasks makes it particularly difficult to train machine and deep learning models on downstream tasks like sentiment analysis, machine translation, named entity recognition, and emotion detection. other challenges when dealing with dialectic arabic include increased morphological complexity, and variety of dialects,and variation within dialect itself, in a fairly close geographical space, making it considerably harder to develop a generalizable model @xcite . hence, when dealing with a certain arabic dialect, such as the algerian dialect, one ought to be resourceful in both data pre-processing and model selection, leveraging any data pre-processing tool that might increase data size without degrading data quality, which could enhance model performance and generalizability. furthermore, leveraging certain systems and model architecture techniques might alleviate bottlenecks in tasks like multi-label classification @xcite . the proposed data augmentation strategy, labelfused iterative mask filling (l-imf), is a resourceful and contextually-based data augmentation technique that uses dziribert, a bert model variant pre-trained on a large algerian corpus @xcite . besides the proposed augmentation strategy, the usage of 6 dziriberts is also proposed to train on the given task by breaking down the multi-label technique into a set of binary classification tasks, with each dziribert dedicated to resolving one. the proposed system architecture addresses common problems associated with deep learning applications in multi-label classification tasks, primarily imbalanced data. despite limited time for optimization, the proposed system achieved strong results. this success was driven by a novel data augmentation technique, which expanded the original 901 training instances into over 15,500 samples, and by a model architecture that addressed multi-label classification using a binary relevance problem transformation. two sampling strategies were applied. in the undersampling approach, a subset of the augmented samples was strategically selected to mitigate the dataset's intrinsic class imbalance, contributing to improved performance. in the oversampling approach, data from the minority class was increased to achieve class balance. the remainder of this paper is structured as follows: section 2 provides a background overview of the task and related work. section 3 describes the proposed system. section 4 presents the experimental setup, followed by results in section 5. section 6 concludes the paper and outlines potential future work. finally, section 7 discusses ethical considerations, while section 8 includes acknowledgments. ## background the proposed system was designed for track a: multi-label emotion detection @xcite , which predicts multiple emotions from text snippets. each input is assigned a binary label (1 or 0) for six emotions; joy, sadness, fear, anger, surprise, and disgust; indicating their presence or absence. the algerian dialect dataset includes 901 labeled training instances with gold-standard annotations, 100 validation instances, and 902 test instances @xcite . multi-label classification extends beyond singlelabel approaches, allowing multiple classes to be assigned to the same input, sometimes with varying intensity levels @xcite . this approach is widely used across fields, including healthcare, document classification, and emotion recognition. various approaches have been proposed to address challenges in multi-label learning. traditional problem transformation techniques include binary relevance, classifier chain, and label powerset. in binary relevance, the multi-label problem is divided into multiple binary classification tasks, where each class is predicted independently and later combined into the final multi-label output. while simple to implement, this method ignores label dependencies, co-occurances, and correlations. classifier chain addresses these limitations by modeling binary label predictions sequentially, allowing the model to learn relationships from earlier predictions. label powerset treats each unique label combination as a distinct class, meaning a six-class binary output results in 64 possible label combinations. however, both classifier chain and label powerset are computationally expensive and struggle to capture high-order label correlations @xcite . with the rise of deep learning, researchers have explored neural architectures to bypass these traditional transformations. @xcite used a lstm layer as a decoder in multi-label document classification tasks where the lstm layer produces labels sequentially and predicts the upcoming label from previous ones, thus allowing the high order capturing of label relationships. transformer based model also were used in multi-label classification. despite deep learning models' ability to over-come the need for traditional problem transformation, many issues have been noticed in the usage of deep learning systems in multi-learning classification tasks. one of those limitations include difficulty in capturing high-order label dependencies, where it has been noted that deep learning has yet to overcome that inability to effectively address more that two labels simultaneously. there is also an issue with difficulty in addressing class imbalance in multi-label classification tasks. it has been noted that, compared to traditional problem transformations' approaches to resolve class imbalance, deep learning approaches are still underdeveloped @xcite . several studies have attempted to address these challenges in arabic multi-label text classification. @xcite used three bert models, marbert, arabert and arabicbert for embedding extraction of a preprocessed arabic 2018 se-meval multi-label dataset where the word embeddings are then concatenated then passed to a meta learner made of a bi-lstm layer that produces the multi-label outputs. the system was trained using a custom hybrid loss function that leverages label correlation matrix, contrastive learning, and class weighting, the use of three bert models as extraction embeddings enhanced generalization, and the use of the custom hybrid loss function reduced to the negative effects of class imbalance present in the data along with promoting label dependency and correlation present in the data. proposed systems for arabic multi-label classification using deep learning include those highlighted by al-smadi. (al-smadi, 2024) proposed deberta-bilstm, which is a deberta model with a bilstm layer that receives the hidden state of the pre-trained deberta model for the purpose of labeling faq covid-19 multi-label dataset released from arabic digital health platform altibbi. the author trained the model using a binary cross-entropy with logits loss as a loss function, thus treating each label independently as a binary classification problem. contrary to the belief that deep learning forgoes the need for traditional problem transformation, @xcite proposed a deep learning system that also successfully integrate traditional problem transformation; yang and emmert-streib developed br-cnn(binaryrelevance convolutional neural network) with a custom weight scaling factor in the binary cross-entropy loss function, br-cnn achieves stateof-the-art performance on aapd and mimic-iii datasets, also outperforming models that leverage label dependencies. while deep learning advancements have improved multi-label classification, challenges such as data scarcity and class imbalance persist. to address these issues, data augmentation techniques are employed to synthetically expand datasets, enhancing model performance, generalizability, and robustness while reducing overfitting. leveraging pre-trained language models has also been noted to be an effective technique that enhances model performance; hence, it was successfully used in english classification tasks. one of the successful usages of pre-trained langauge models for english data augmentation is outlined by (kesgin and amasyali, 2023). kesign and amasyali proposed iterative mask filling, a bert augmentation technique that iteratively replaces words using masked language modeling to produce a single final augmented sentence for each input sentence. the proposed method outperforms traditional data augmentation approaches like synonym replacement, back-translation, and random modifications in topic classification. by using probabilistic word replacements and confidence based filtering, iterative mask filling significantly improves model robustness. it was noted by the authors that the approach limited effectiveness in sentiment analysis, due to some words being critical in determining sentiment, when masked, allowing for the chance for the filled word to change the entire sentiment of the subsequent fully augmented sentence. another english data augmentation technique through language models is sketched by @xcite . @xcite conditional bert (c-bert), which is a fine-tuned version of bert that performs context-aware word filling of randomly masked tokens. c-bert outperforms other data augmentation techniques mentioned like random synonym replacement from wordnet and contextual augmentation proposed by @xcite . arabic literature, comparatively, has limitedly explored data augmentation techniques in the field of arabic text classification. @xcite employed methods such as modified easy data augmentation (eda), back-translation, and word embedding substitution for arabic named entity recognition (ner) tasks, achieving positive but varying results. similarly, (abuzayed and al-khalifa, 2021) observed substantial improvements in classification performance through label augmentation. @xcite were one of the few researchers to use transformer-based data augmentation in arabic text classification, they leveraged aragpt-2 and arabert for generative data augmentation, demonstrating significant performance gains in arabic sentiment analysis. additionally, @xcite utilized generative adversarial networks (gans) to generate dialectal arabic datasets for sentiment analysis, reporting enhanced model performance when training on the augmented dataset. ## experimental setup during training, the proposed model processed six tokenized text inputs, each corresponding to a label from the six datasets. each submodel employed a separate cross-entropy loss function and was optimized using the adam optimizer. most layers in the submodels were frozen, except for the pooler layer and the mlp head. dropout probabilities of 0.1, 0.2, and 0.3, alongside learning rates of 1.5 × 10 -5 , 2.5 × 10 -5 , and 3.5 × 10 -5 were evaluated; however, the batch size was set at 32. training was conducted for 15 epochs, with performance evaluated on the dedicated test set at the end of each epoch. the epoch with the best macro f1 score was selected. a grid search hyperparameter optimization algorithm was used to tune the hyperparameter for each model. the baseline single dziribert model was trained on the original data with no augmentation. despite almost the same training hyperparameters as the proposed model, the only difference was that binary cross entropy with logits loss was set as the loss function. this loss function enables simultaneous prediction of multiple non-mutually exclusive labels by applying a sigmoid activation to each output logit. a detailed breakdown of the computational resources used is provided in appendix b. ## results the proposed system, combining the model with the l-imf augmentation technique, achieved its highest performance with a macro f1 score of 51.2% when trained on an undersampled dataset generated using the l-imf simple prompt, while using oversampling on the same l-imf simple prompt augmented data achieved 41.6%. under alternative settings, training on an undersampled dataset produced via the l-imf elaborate prompt resulted in a macro f1 score of 42.1% for comparison, the task's baseline, rembert, attained a macro f1 score of 41.4%, while the standalone dziribert model, that provided a contrast to the proposed problem transformation, achieved only 25.1% under its best hyperparameters. while the proposed problem transformation proved effective, the same could not be said for the l-imf augmentation technique. when the model was trained on data generated by the simple data augmentation techniques, the model achieved a macro f1 score of 53.4%. nonetheless, the proposed system, incorporating the problem transformation and l-imf, achieved an improvement of nearly 10% over the task's baseline model and over 26% compared to the dziribert model. it ranked 28th out of 41 teams on the algerian dialect task leaderboard with an official score of 48.6%.a more detailed breakdown of the scores along with the tuned hyperparameters for the best performing models is provided in appendix c. ## conclusion this study introduced the label-fused iterative mask filling (l-imf) augmentation technique alongside a multi-model approach for multi-label classification in the track a: multi-label emotion detection challenge for algerian dialect. the approach addressed key challenges in multi-label emotion detection, including label dependencies, class imbalance, and limited linguistic resources. by leveraging l-imf, contextually and labelsensitive augmented data were generated, mitigating class imbalance while maintaining label consistency. to further tackle the challenges of deep learning in multi-label classification, a system of six independent classifiers was implemented, with each dziribert-based sub-model specializing in predicting a single emotion. this design allowed the reduction of class imbalance by creating six balanced datasets for model training. moreover, undersampling and oversampling helped ensure a more equitable distribution of classes, preventing the dominance of majority classes. results from the conducted experiments demonstrated the effectiveness of the proposed methodology in enhancing emotion classification performance in low-resource language settings. the in-tegration of a pre-trained algerian dialect model, l-imf augmentation, and independent classifiers contributed to strong performance in multi-label emotion detection for dialectal arabic that more than doubled the macro f1 score obtained by using a single model via a powerset problem transformation. the experimental results indicate that the proposed model and binary problem transformation had the greatest impact on performance. specifically, using six dziribert models as binary classifiers doubled the macro f1 score compared to a single dziribert model with a output vector of dimension 6. in contrast, the l-imf technique had a slightly negative effect on the model's performance when compared to a simpler augmentation method. however, the authors still argue that l-imf holds significant potential for future improvements and remains a promising area for further research. future studies may investigate the effectiveness of the l-imf technique in augmenting datasets of different types, such as topic-based or sentiment-based datasets. future work could also explore how the type and positioning of prompts influence performance on downstream tasks.