U niversity of I ndonesia at S em E val-2025 Task 11: Evaluating State-of-the-Art Encoders for Multi-Label Emotion Detection (2025)

Abstract: This paper presents our approach for SemEval 2025 Task 11 Track A, focusing on multilabel emotion classification across 28 languages. We explore two main strategies: fully fine-tuning transformer models and classifier-only training, evaluating different settings such as fine-tuning strategies, model architectures, loss functions, encoders, and classifiers. Our findings suggest that training a classifier on top of prompt-based encoders such as mE5 and BGE yields significantly better results than fully fine-tuning XLMR and mBERT. Our best-performing model on the final leaderboard is an ensemble combining multiple BGE models, where CatBoost serves as the classifier, with different configurations. This ensemble achieves an average F1-macro score of 56.58 across all languages.

## introduction this paper presents the university of indonesia's multi-label emotion classification system for all 28 languages included in semeval 2025 task 11 track a @xcite . the task focuses on recognizing multiple emotions expressed in text across diverse linguistic and cultural contexts. language is a rich and complex medium for conveying emotions @xcite . however, emotional expression and interpretation vary widely across individuals, even within the same cultural or social background. this variability introduces inherent uncertainty in accurately inferring emotions from textual cues. emotion recognition is a challenging task that involves multiple subproblems, such as identifying the speaker's emotional state, detecting emotions embedded in text, and analyzing the emotional impact on readers @xcite . addressing these challenges requires models that can handle multiple emotional labels accurately. to address this problem, we explore both classifier-only training and end-to-end fine-tuning strategies. our approach leverages state-of-the-art encoder-based architectures, including jina, bge, and multilingual-e5 (me5) @xcite @xcite . these models are pretrained to generate high-quality embeddings, improving classification performance. we experiment with both pre-trained embeddings combined with machine learning classifiers and finetuning transformer-based models with specialized loss functions such as focal loss and asymmetric loss to mitigate class imbalance @xcite . our key findings indicate that embedding-based methods with tree-based classifiers, where we freeze the classifier, particularly bge combined with catboost, outperform fine-tuning approaches for multi-label emotion classification. specifically, employing separate prompts for each emotion in bge leads to a improvement in f1-macro scores. finally, ensembling enhances the model's robustness, as reflected in our final submission, which shows an improvement compared to using a single model. ## related works this task focuses on multilingual multilabel emotion classification using the brighter dataset @xcite , which includes predominantly low-resource languages from africa, asia, eastern europe, and latin america. these instances, annotated by fluent speakers, span multiple domains, presenting unique challenges due to both multilinguality and the complexity of multilabel classification. recent advancements in decoder-based models such as llama, @xcite @xcite @xcite , alongside the widespread use of the bert family of models @xcite @xcite , have demonstrated strong performance in multilingual natural language processing (nlp) tasks. prior research @xcite has leveraged these architectures for emotion classification, yet the exploration of advanced encoder-based models like jina, bge, and me5 @xcite @xcite remains limited. these models have performed exceptionally well in embedding benchmarks such as mteb @xcite , suggesting their potential for our task. multilabel classification poses distinct methodological challenges. a traditional approach is binary relevance (br), where separate models are trained for each label @xcite . more recent strategies leverage bert-based architectures to enable multi-output classification, predicting multiple labels simultaneously @xcite . another technique incorporates the [sep] token to convert multilabel classification into a sequence-labeling task, effectively treating it as a single-label problem @xcite . a persistent challenge in multilabel classification is class imbalance @xcite . unlike standard classification tasks, conventional stratification techniques do not naturally extend to multilabel settings. iterative stratification methods @xcite offer a partial solution, while alternative techniques such as weighted loss functions @xcite , focal loss, and asymmetric loss @xcite help mitigate imbalance in deep learning models. linguistic diversity further complicates multilingual emotion classification. given the authors' limited language proficiency, exhaustive linguistic analysis across all dataset languages is infeasible. to address this, we explore two approaches, training models separately for each language or collectively across all languages, following prior work @xcite . our work builds on these foundations by investigating the underexplored potential of advanced encoder-based models in multilingual multilabel emotion classification. by combining these models with effective imbalance-handling techniques and leveraging external linguistic resources, we aim to advance the state of multilingual emotion classification beyond existing methodologies. ## limitations the limitation of our study is the lack of extensive qualitative analysis due to limited language proficiency. since we do not fully understand many of the languages in the dataset, our analysis primarily relies on quantitative methods. ## conclusion our study demonstrates that classifier-based approaches with prompt-based encoders, particularly bge and multilingual-e5 (me5), outperform fully fine-tuned transformer models for multilingual multi-label emotion classification. our best-performing model, bge with catboost and emotion-specific prompting, achieved the highest average f1-macro scores across languages in our experiment. additionally, an ensemble of multiple bge-based models further improved performance, significantly surpassing the best decoderbased model from prior work. these results highlight the strength of high-quality embeddings combined with tree-based classifiers for emotion detection tasks. our findings also show that multilingual training does not provide a clear advantage over monolingual models. furthermore, minor prompt modifications led to measurable gains, emphasizing the importance of prompt engineering. overall, our study suggests that leveraging strong embedding models with efficient classifiers is a more effective strategy than full transformer fine-tuning for multi-label emotion classification across diverse languages. m.a. hearst, s.t. dumais, e. osuna, j. platt, and b. scholkopf. 1998. support vector machines. ieee intelligent systems and their applications, 13(4):18-28. tollef jørgensen. 2024. pear at semeval-2024 task 1: pair encoding with augmented re-sampling for semantic textual relatedness. in proceedings of the 18th international workshop on semantic evaluation (semeval-2024), pages 1405-1411, mexico city, mexico. association for computational linguistics. yova kementchedjhieva and ilias chalkidis. 2023. an exploration of encoder-decoder approaches to multilabel classification for legal and biomedical text. in findings of the association for computational linguistics: acl 2023, pages 5828-5843, toronto, canada. association for computational linguistics. tsung-yi lin, priya goyal, ross girshick, kaiming he, and piotr dollár. 2017. focal loss for dense object detection. in proceedings of the ieee international conference on computer vision, pages 2980-2988.