DUT _ IR at S em E val-2025 Task 11: Enhancing Multi-Label Emotion Classification with an Ensemble of Pre-trained Language Models and Large Language Models (2025)

Abstract: In this work, we tackle the challenge of multi-label emotion classification, where a sentence can simultaneously express multiple emotions. This task is particularly difficult due to the overlapping nature of emotions and the limited context available in short texts. To address these challenges, we propose an ensemble approach that integrates Pre-trained Language Models (BERT-based models) and Large Language Models, each capturing distinct emotional cues within the text. The predictions from these models are aggregated through a voting mechanism, enhancing classification accuracy. Additionally, we incorporate threshold optimization and class weighting techniques to mitigate class imbalance. Our method demonstrates substantial improvements over baseline models. Our approach ranked 4th out of 90 on the English leaderboard and exhibited strong performance in English in SemEval-2025 Task 11 Track A.

## introduction emotion classification is crucial in various natural language processing (nlp) applications, including customer feedback analysis, mental health monitoring, and social media sentiment tracking. unlike traditional sentiment analysis, which categorizes text into positive, negative, or neutral sentiments, multi-label emotion classification is more complex, as a single sentence can express multiple emotions, such as joy, anger, and sadness @xcite , as shown in figure 1 .this complexity arises from the subjective nature of emotions, their overlapping characteristics, and the ambiguity in short texts. although transformer-based models, particularly bert and its variants, have shown promising results in capturing semantic features and contextual dependencies @xcite , challenges per-* corresponding author sist, including class imbalance, difficulties in distinguishing subtle emotional expressions, and the need for better generalization across languages @xcite . in this study, we focus on multi-label emotion classification as defined in semeval-2025 task 11 track a @xcite , which aims to evaluate nlp systems' ability to identify multiple emotions in a given text. we propose an ensemble approach, integrating multiple bert-based pretrained language models (plms) (such as bert, roberta @xcite , and other variants) along with large language models (llms) to capture diverse emotional cues @xcite . the predictions from these models are aggregated using a voting mechanism, which enhances robustness and accuracy. by leveraging both pretrained transformers and llms, our approach effectively captures the complex and overlapping nature of emotions, improving the generalization across varied emotional expressions. in addition to the ensemble strategy, we incorporate threshold optimization and class weighting to address class imbalance and improve decision boundaries. these techniques ensure that underrepresented emotions are adequately considered, leading to significant performance improvements over baseline models and enhancing our system's effectiveness in multi-label emotion classification. ## related work the fundamental challenge in multi-label emotion classification lies in detecting non-exclusive emotional states within textual expressions. early methodologies predominantly employed lexiconbased systems combined with statistical classifiers like svms (mohammad and turney, 2013), utilizing hand-engineered features such as emotion-word counts and syntactic patterns. while effective for coarse-grained analysis, these approaches exhibited limitations in handling three critical aspects: currently, pre-trained language models, especially bert and its variants, have performed well in sentiment multi-label classification tasks. these models effectively capture contextual information through a bidirectional transformer architecture, improving classification accuracy. studies have shown that plms generally outperform traditional methods and early deep learning models. in multilabel prediction, the binary cross entropy loss function is widely used to deal with the independence of each label @xcite . at the same time, a weighted loss function is used to adjust the label weights to address the label imbalance problem. in addition, some studies have further improved the classification effect by modeling the dependencies between labels through graph neural networks (gnns) or conditional random fields (crfs) @xcite . in general, plms perform significantly better than traditional methods in this task and have achieved good results on multiple standard datasets. in the task of sentiment multi-label classification, large language models have performed well, especially in capturing the complex sentiment in text and the relationship between labels. llms usually perform label prediction through generative or sequence-to-sequence (seq2seq) methods, and mine the pre-trained knowledge of the model by designing appropriate prompts. in addition, similar to plms, llms also use weighted loss functions to solve the label imbalance problem and combine multi-task learning to further improve the classification effect @xcite . although llms have achieved excellent results in sentiment multilabel classification, their huge computational re-quirements remain a challenge. ## system overview as shown in figure 2 , our proposed system is composed of two main stages. in the first stage, we train and fine-tune three transformer-based models, bert, roberta, and deberta @xcite , employing strategies such as automatic threshold search, class weight allocation, and data augmentation to address challenges like data imbalance and overfitting. additionally, we explore advanced large models, including qwen2.5 @xcite and llama3.1, to further enhance performance. in the second stage, we improve model robustness and accuracy by integrating predictions from multiple models (roberta, de-berta, qwen2.5 and llama3.1), using a hard voting strategy and cross-validation, ensuring better generalization and complementary feature learning. ## results finally, we adopted the full model integration (covering language models and large language models) as the ultimate solution of the system, and submitted the prediction result file of the final model on the test set. the official ranking is shown in ## conclusion this paper introduces the system we designed in track a of semeval-2025 task 11, which aims to solve the problem of unbalanced class distribution that is common in multi-class label classification tasks. by combining methods such as automatic threshold search and class weight assignment, we effectively alleviate the model's excessive focus on high-frequency emotions and reduce its tendency to ignore low-frequency emotions. based on this, we further adopt a model integration strategy to optimize the shortcomings of a single model in capturing complex label relationships and semantic features in text, and significantly improve the robustness and generalization ability of the model. overall, our system performs outstandingly in the task of multi-label emotion classification, especially on the english test set of track a, where it achieved an excellent score of third place, verifying the effectiveness and advantages of our method.