Dianchi at S em E val-2025 Task 11: Multilabel Emotion Recognition via Orthogonal Knowledge Distillation (2025)

Abstract: This paper presents KDBERT-MLDistill, a novel framework for multi-label emotion recognition developed for SemEval-2025 Task 11. Addressing challenges of fine-grained emotion misdetection and small-data overfitting, the method synergizes BERT-based text encoding with orthogonal knowledge distillation. Key innovations include: (1) Orthogonal regularization on classifier weights to minimize redundant feature correlations, coupled with dynamic pseudo-labeling for periodic data augmentation; (2) A hierarchical distillation mechanism where dual teacher-student models iteratively exchange parameters to balance knowledge retention and exploration.

## introduction multi-label text classification advances beyond the semantic unidimensionality limitation of traditional single-label classification by assigning multiple interrelated labels to a single text, thereby effectively capturing the complexity of coexisting emotions in real-world scenarios @xcite . as a pivotal technology for revealing human cognitive states, emotion recognition (er) demonstrates significant application value across diverse domains @xcite , including consumer behavior analysis @xcite and community mental health monitoring @xcite . however, existing methods often suffer from misdetection of fine-grained emotional features due to insufficient long-range semantic modeling capabilities, while also facing overfitting risks in small-scale annotated data scenarios. to address these challenges, this study proposes an innovative knowledge distillation framework: ## background in the field of natural language processing, bert-a transformer-based pre-trained language model @xcite -has significantly enhanced textual representation capabilities through its pre-training paradigm of masked language modeling and next-sentence prediction. however, the substantial parameter size of bert models incurs high computational costs, limiting their industrial deployment. to address this, knowledge distillation (kd) has been introduced for model lightweighting. original kd @xcite operates by transferring implicit knowledge-such as output-layer probability distributions and intermediate-layer attention weights-from complex teacher models to compact student models. in the context of bert compression, sanh @xcite developed distilbert, which reduces the model size by 40% while retaining 97% of the original performance through layer reduction and a teacher-student attention alignment loss function. subsequent work has extended this framework: jiao et al. @xcite proposed tinybert, which incorporates attention matrix mapping and hidden state adaptation to enable layer-wise knowledge transfer, achieving an accuracy gap of merely 3% compared to bertbase on glue benchmarks. notably, knowledge distillation applications in bert optimization have evolved beyond single-model compression to innovative directions such as multimodal pre-training @xcite and dynamic architecture pruning, demonstrating its enduring potential to balance model efficiency and performance. ## system overview in this section, we delineate our methodological framework. our approach leverages the bert pre-trained model for text sequence processing and contextual representation learning. we synergistically integrate knowledge distillation (kd) with advanced textual data augmentation strategies to enhance generalization performance. ## experimental setup this section details the configuration of subtask a, including dataset structure and training strategies. the task data originates from the official csv-formatted dataset released for semeval 2024, comprising three splits: a training set, a development (dev) set, and a test set, each containing 2,768 annotated samples. each sample follows a "text + sentiment label" structure: the text field contains the input sentence for classification, while the sentiment labels cover five fine-grained categories (anger, fear, joy, sadness, surprise) under a multi-label annotation scheme. through parameter tuning experiments, we identified optimal performance when training the model for 90 total epochs with a 15-epoch learning rate warm-up phase. the batch sizes were set to 32 for the training set and 128 for the validation set. during the warm-up phase (first 15 epochs), the initial learning rate was 3e-5, which decayed by 10% every 10 epochs. additionally, a learning rate scheduler was implemented to facilitate model convergence and enhance performance. ## conclusions this paper details our participation in semeval 2025 task 11: bridging the gap in text-based emotion detection, specifically subtask a (english). our approach employs a bert-based pretrained model to encode textual cls token representations as features, which are then passed through a linear layer to generate logits for multilabel classification via sigmoid thresholding. we innovatively enhanced the cross-entropy (ce) loss by introducing orthogonal regularization on the fully connected layer's weight matrix (using an inverse distance penalty between weight vectors) and incorporated knowledge distillation during later training stages with a kl divergence constraint between the teacher and student model outputs. throughout the training, validation set predictions were dynamically integrated into the training data every 15 epochs, alongside alternating parameter updates between the teacher and student models for iterative refinement. this