A Mapping on Current Classifying Categories of Emotions Used in Multimodal Models for Emotion Recognition (2024)

Abstract: In Emotion Detection within Natural Language Processing and related multimodal research, the growth of datasets and models has led to a challenge: disparities in emotion classification methods. The lack of commonly agreed upon conventions on the classification of emotions creates boundaries for model comparisons and dataset adaptation. In this paper, we compare the current classification methods in recent models and datasets and propose a valid method to combine different emotion categories. Our proposal arises from experiments across models, psychological theories, and human evaluations, and we examined the effect of proposed mapping on models.

## introduction emotion recognition, as an essential ability for good interpersonal relations @xcite , has long been a major subject in psychology, and for the last two decades has received increasing attention from the field of computer science, especially artificial intelligence @xcite . yet in this process a divergence has emerged from newly published datasets and models -the misalignment between different categories of emotions. to resolve such disparity between emotion datasets, we propose a psychologybased solution for computer scientists to solve the problem of misalignment in emotion classification datasets, which is caused by the independent nature of emotion classification theories. in the field of psychology, there are many different theories on how to classify emotions focusing on different aspects. various theories classify emotions based on different factors: ekman's theory focuses on universal facial expressions @xcite , comparing the facial expressions of westerners and aboriginal residents of new guinea; plutchik's evolutionary perspective categorizes emotions into 8 primary emotions with 3 lev-els of intensity @xcite based on the communication function of emotions; barrett's @xcite biological approach studies brain responses to emotions through intepreting eegs and physiological changes @xcite ; and emphasizing cultural influence, the constructionist theory adds social and linguistic elements to emotion understanding @xcite . these theories are independent but sometimes interconnected, providing a foundation for potential integration. different theories are mostly considered to be independent theories of emotion, yet these classification approaches are often interconnected and sometimes built upon each other, providing a basis to connect them. however, few studies explore ways to connect or combine these different categorizations. in computer science, researchers face the challenge of choosing an emotion theory when building datasets for emotion detection. recent work in emotion classification has shifted towards using multimodal data sources like audio, video, and text @xcite , and some even explore incorporating additional factors like personality and social connections to leverage more information for deep learning models @xcite . due to varying annotation methods and mismatch in the set of labels, a model typically selects a single dataset for experiments, although more data could improve its performance. a significant issue arises from the lack of alignment in labeling schemas across datasets, making it challenging for models to leverage multiple datasets in supervised learning @xcite . this disparity results in a lack of cohesion in the literature, hinders direct performance comparisons, and complicates dataset combination and training. since annotating such datasets is costly and timeconsuming, a mapping method that can unify existing datasets could benefit the community. currently, little research in both psychology and com-puter science explores the relationship between different emotion categories. while there are studies mapping categorical emotions onto dimensional models @xcite and recent work inproviding more grounded emotion categories in dutch @xcite , the mapping between multiple categorical emotions, which creates misalignment in emotion datasets for machine learning, remains unstudied. this paper aims to establish a valid mapping of emotion categories based on psychological theories and validated through machine learning models. we select the five most commonly used emotion classification methods in large emotion datasets, propose a valid mapping method rooted in psychological theory, verify it through human evaluation, and assess its impact on emotion recognition models. our mapping method is an initial effort to create a continuous mapping approach connecting these discrete emotion classification methods. ## mapping effects on ml models to analysis the effect of the proposed mapping on machine learning models, we set up an experiment to check the accuracy of emotion re-categorization after applying the mapping method in ## conclusion in this paper, we propose the first complete mapping that connects different emotion categories for multimodal emotion recognition studies, and pro-vide a study of the effect of using different emotion classification methods when training models. we are the first group of researchers attempting to bridge the different psychological emotion theories and lend them consistency in the computer science world. moreover, using our mapping allows researchers to obtain a larger and more flexible dataset for training and testing and to analyze the model's ability to differentiate emotions using different emotion categories, as well as identify the best model across all datasets.