Angeliki Linardatou at S em E val-2025 Task 11: Multi-label Emotion Detection (2025)

Abstract: This study, competing in SemEval 2025 Task 11 - Track A, detects anger, surprise, joy, fear, and sadness. We propose a hybrid approach combining fine-tuned BERT transformers, TF-IDF for lexical analysis, and a Voting Classifier (Logistic Regression, Random Forest, SVM, KNN, XG-Boost, LightGBM, CatBoost), with grid search optimizing thresholds. Our model achieves a macro F1-score of0.6864. Challenges include irony, ambiguity, and label imbalance. Future work will explore larger transformers, data augmentation, and cross-lingual adaptation. This research underscores the benefits of hybrid models, showing that combining deep learning with traditional NLP improves multi-label emotion detection.

## introduction sentiment analysis (sa) and emotion detection are key nlp tasks for analyzing emotional tone in text, essential for understanding human behavior. while sa classifies text as positive, negative, or neutral, emotion detection identifies specific emotions (joy, anger, sadness) but faces challenges like sarcasm, ambiguity, and overlapping emotions. transformer models such as bert and xl-net have greatly improved sentiment classification, with hybrid approaches emerging. @xcite combined bert with random forest (ens-rf-bert), while @xcite proposed bert-xlnet for long-text classification. however, multi-label emotion detection remains underexplored. semeval 2025 task 11 addresses this by providing a benchmark for detecting anger, surprise, joy, fear, and sadness. in our study, we develop a finetuned bert model, enhanced with tf-idf for lexical analysis, and a voting classifier. we also explore roberta-large for better predictions. our model achieves an f1-score of 0.68, demonstrating the effectiveness of combining transformers with traditional ml techniques. future improvements include data augmentation, cross-lingual adaptation, and larger transformer models to enhance performance. ## related work and background sentiment analysis (sa) and emotion detection are key nlp tasks, but traditional methods rely on single-label classification, failing to capture multiple coexisting emotions. semeval 2025 task 11 -track a addresses this by requiring models to detect multiple emotions per text. multi-label classification is essential, as texts can express mixed emotions (e.g., joy and surprise, sadness and fear). existing single-label methods struggle with this complexity, necessitating models that learn emotional dependencies. this task involves five core emotions: joy, sadness, anger, surprise, and fear. effective multilabel classification requires techniques like binary relevance, classifier chains, and deep learning to improve detection accuracy. ## previous studies several studies have explored multi-label sentiment analysis using ml and deep learning approaches: jin and lai (2020) (jin and lai, 2020) proposed a hybrid model combining bert's contextual embeddings with a modified tf-idf weighting technique. their approach enhanced key term importance while leveraging deep contextual representations, leading to a 4.2% f1-score improvement over traditional tf-idf and standalone bert models. @xcite introduced a sentiment correlation modeling approach to capture interdependencies between emotions. their correlation-aware mechanism refined sentiment prediction, improving f1-score by 3.8% compared to conventional multi-label models. these studies highlight the benefits of hybrid and correlation-aware approaches in multi-label sentiment classification. ## comparison with the proposed model unlike prior studies relying solely on transformers or traditional ml, our approach combines both for improved accuracy. we fine-tune bert for contextual understanding while integrating tf-idf for lexical features. a voting classifier (logistic regression, random forest, svm, knn, xgboost) optimizes classification via ensemble learning. grid search refines probability thresholds, balancing precision and recall. to handle overlapping emotions and class imbalance, we fine-tune decision boundaries. training efficiency is improved through reduced learning rates, minimal epochs, and cross-validation, ensuring robust predictions. ## innovative aspects of our approach our proposed method differentiates itself from previous studies in several key ways: our approach stands out by integrating deep learning with traditional nlp techniques for multi-label emotion detection. by combining bert for contextual understanding with tf-idf for lexical emphasis, we ensure that important words are not downweighted, enhancing interpretability and classification accuracy. additionally, we incorporate ensemble learning through a voting classifier, which combines multiple models to improve stability and reduce variance. to further refine classification performance, we apply grid search for optimal probability threshold selection, maximizing the f1-score. this hybrid methodology effectively balances semantic understanding and lexical precision, leading to a more robust and accurate emotion detection system. ## dataset and examples the dataset used in our experiments consists of english textual data annotated with multiple emotional labels. the data is provided by the se-meval 2025 task 11 competition (track a), which focuses specifically on english-language emotion detection. below, we present representative examples from the dataset along with their detected emotions: these examples demonstrate the necessity of multi-label classification, as a single sentence can express multiple emotions simultaneously (refer to fig. 2 ). capturing such nuances is crucial for improving the accuracy of emotion detection models. our approach introduces an effective fusion of modern nlp techniques and traditional text analysis, demonstrating the potential of hybrid methodologies for improving multi-label emotion detection. the following figure illustrates the frequency distribution of detected emotions in the dataset, highlighting which emotions appear more frequently in the text samples. ## system overview our system for multi-label emotion detection combines state-of-the-art transformer models with traditional feature engineering and ensemble learning to maximize classification performance. below, we describe the key components and methodological choices that define our approach. ## key algorithms and modeling decisions our hybrid approach integrates deep learning, feature engineering, and ensemble learning for multilabel emotion detection. we fine-tune bert and roberta for contextual embeddings, complemented by tf-idf for lexical features. a soft voting ensemble (logistic regression, random forest, svm, knn, xgboost, lightgbm, catboost) enhances classification, leveraging each model's strengths. grid search-based threshold optimization refines decision boundaries, maximizing the f1-score. to prevent overfitting, we use a reduced learning rate and minimal training epochs for bert fine-tuning. ## data preprocessing and feature extraction the dataset consists of english text snippets labeled with five emotions: anger, fear, joy, sadness, and surprise. some data samples are multilabeled, while others contain only one label. the dataset is split into 80% training and 20% testing. text is preprocessed using the bert tokenizer for subword embeddings with padding and truncation. tf-idf (vocabulary size: 8000) ex-tracts numerical features, which are concatenated with bert embeddings to form an enriched feature space. lemmatization, stemming, and stopword removal are not applied. ## model training the transformer model is fine-tuned for two epochs using the adamw optimizer and binary cross-entropy loss.this choice was made to minimize overfitting and ensure efficient training. to support this decision, we conducted additional experiments by training bert for 1, 2, 3, and 4 epochs and monitoring the macro f1-score. the model achieved its highest performance at epoch 2, with f1-scores plateauing or slightly decreasing afterward, indicating potential overfitting.this empirical finding aligns with jin and lai (2021) (jin and lai, 2020), who observed that 2-3 epochs were optimal when fine-tuning bert on emotion-labeled datasets of similar size. after training, sentence embeddings are extracted from the final hidden layer of the bert model. these are then concatenated with tf-idf features to form a hybrid representation, which is used to train an ensemble classifier. binary cross-entropy loss was selected as it is well-suited for multi-label classification, treating each label as an independent binary task. it effectively handles overlapping labels and produces well-calibrated probability estimates. finally, we perform grid-based hyperparameter tuning to adjust the number of estimators, learning rates, and maximum depths of the ensemble classifiers (e.g., random forest, xgboost, lightgbm, catboost), ensuring the best possible configuration for robust performance. ## threshold optimization since our task involves multi-label classification, we needed to decide how to convert the model's probabilistic outputs into binary predictions for each emotion. to do this, we optimized the probability thresholds using a grid search approach. we tested a range of threshold values from 0.30 to 0.70, increasing by 0.05 each time. this process was carried out separately for each emotion, allowing the model to adjust its sensitivity depending on the characteristics of each label -which is especially helpful in cases of class imbalance. for example, emotions like surprise or fear, which appear less frequently in the data, may benefit from a lower threshold. our goal was to find the com-bination of thresholds that would maximize the macro f1-score across all five emotions. to make the results more reliable, we used 5-fold crossvalidation on the training set. the best thresholds were then applied to the test predictions. this per-label tuning led to a better balance between precision and recall, especially for texts expressing subtle or overlapping emotions. by avoiding a "one-size-fits-all" threshold (like the default 0.5), we were able to make the model more adaptable to the specific behavior of each emotion category. ## model evaluation our final model achieves: the results confirm that combining deep learning with traditional machine learning improves multi-label emotion classification. lightgbm and catboost further boost performance alongside xgboost. the model's low accuracy (0.3827) reflects the difficulty of multi-label classification, where texts express multiple emotions. the macro f1-score (0.6864) suggests an imbalance between precision (0.6814) and recall (0.7094), likely due to class imbalance, overlapping emotions, or suboptimal threshold selection. additionally, language subjectivity and ambiguity pose challenges, as the same phrase can convey different emotions depending on context. ## experimental setup our experimental setup consists of carefully designed steps to ensure reliable model training and evaluation. below, we describe the dataset splitting strategy, preprocessing steps, hyperparameter tuning, and external tools utilized. ## results in this section, we analyze the performance of our model based on official evaluation metrics, competition ranking, and additional qualitative observations. ## conclusion we proposed a hybrid approach for multi-label emotion detection, combining transformer models (bert, roberta) with tf-idf and ensemble learning. this method leveraged deep embeddings and classical ml models, achieving an f1-score of 0.6864. challenges like ambiguous text, sarcasm, and label imbalance persist. future work includes larger transformers (e.g., roberta-large) with potential f1-score improvements to 0.74, alongside emotion lexicons, sentiment-aware embeddings, and advanced augmentation. our results highlight the benefits of integrating deep learning with traditional nlp, with further refinements promising enhanced performance.