Analyzing Key Factors Influencing Emotion Prediction Performance of VLLM s in Conversational Contexts (2024)

Abstract: Emotional intelligence (EI) in artificial intelligence (AI), which refers to the ability of an AI to understand and respond appropriately to human emotions, has emerged as a crucial research topic. Recent studies have shown that large language models (LLMs) and vision large language models (VLLMs) possess EI and the ability to understand emotional stimuli in the form of text and images, respectively. However, factors influencing the emotion prediction performance of VLLMs in real-world conversational contexts have not been sufficiently explored. This study aims to analyze the key elements affecting the emotion prediction performance of VLLMs in conversational contexts systematically. To achieve this, we reconstructed the MELD dataset, which is based on the popular TV series Friends, and conducted experiments through three sub-tasks: overall emotion tone prediction, character emotion prediction, and contextually appropriate emotion expression selection. We evaluated the performance differences based on various model architectures (e.g., image encoders, modality alignment, and LLMs) and image scopes (e.g., entire scene, person, and facial expression). In addition, we investigated the impact of providing persona information on the emotion prediction performance of the models and analyzed how personality traits and speaking styles influenced the emotion prediction process. We conducted an in-depth analysis of the impact of various other factors, such as gender and regional biases, on the emotion prediction performance of VLLMs. The results revealed that these factors significantly influenced the model performance.

## introduction emotional intelligence (ei) in artificial intelligence (ai), which refers to the ability of an ai to un-derstand and respond appropriately to human emotions, is a crucial topic in ai research. ei involves the ability to interpret and manage the emotions that are embedded in information and is essential for various cognitive tasks, from problem solving to behavior regulation @xcite . human emotions play a significant role in various domains such as academics, competitive sports, and daily life and are shaped by internal and external factors @xcite @xcite . equipping ai systems with ei enhances the quality of human-ai interactions, improves user experience, and enables more natural and effective communication based on emotional empathy. large language models (llms), which are considered a crucial step towards achieving artificial general intelligence, have exhibited exceptional performance in various fields @xcite . as a result, there has been growing interest in whether llms possess ei. @xcite evaluated the ei of llms through psychological measurements and discovered that gpt-4 achieved high eq scores. moreover, studies by @xcite and @xcite showed that llms can understand emotional stimuli in the form of text and images, perceiving emotions in a manner similar to humans. however, these studies have limitations as they focus on a single modality, whereas various factors such as verbal cues, visual cues, and contextual information interact in a complex manner in real-world conversational situations. vision large language models (vllms) have recently gained attention to overcome the above limitations. as vllms can process text and images simultaneously, they have the potential to solve more complex and multifaceted emotion prediction tasks. for example, vllms can infer emotional states by comprehensively analyzing the facial expressions and verbal cues of conversation participants or predict appropriate emotional responses considering the conversational context. however, despite their potential, the key factors influencing the emotion prediction of vllms in conversational situations have not yet been sufficiently explored. this study aimed to analyze the factors influencing the emotion prediction of vllms, such as the model architecture, persona information, and biases, systematically to explore means of improving emotion prediction performance in conversational situations. to achieve this, we reconstructed the meld dataset @xcite based on the popular tv series friends and augmented it by integrating various elements, such as images, conversational context, and persona information, to evaluate the performance of vllms comprehensively. we conducted an extensive assessment of the emotion understanding and expression performance of vllms through three sub-tasks: overall emotion tone prediction, character emotion prediction, and contextually appropriate emotion expression selection. the experimental results showed that differences in the model architecture had a distinct impact on the emotion prediction performance. this suggests that the structural characteristics of vllms, such as the method of integrating image and text information and the llm backbone, play crucial roles in emotion prediction performance. in addition, models that included persona information exhibited notable differences in the emotion prediction process. this implies that information on the personality traits and speaking styles of an individual significantly influences the emotion understanding and response performance of the model. we also conducted an in-depth analysis of the impact of various factors related to emotion prediction, such as gender and regional biases, on the emotion prediction performance of vllms. the analysis revealed that factors such as gender and regional biases significantly influenced the emotion prediction process of vllms, revealing the biases and limitations that may arise in this process. ## related work the rapid development of llms has led to substantial progress in language generation, knowledge utilization, and complex reasoning tasks. however, as these models are being integrated into various application domains, enhancing their ei and mitigating social biases have become increasingly important. @xcite explored the ei of llms using psychological methods, thereby laying the foundation for further research on how these models perceive and respond to emotional stimuli. building on this work, @xcite and @xcite investigated the ability of llms to understand emotional content and demonstrated that current models can react to emotional stimuli similarly to humans. paech (2023) introduced a new criterion for evaluating the ei of llms through eq-bench, which is a benchmark that measures the ability of a model to predict the emotional states of characters within conversations. @xcite proposed emobench, which is a benchmark that is designed to evaluate the ei of llms comprehensively by assessing not only emotion recognition, but also emotional regulation and the application of emotional understanding. along with research on the ei of llms, addressing the social biases that are inherent in these models is a crucial task for the development of ethical ai. @xcite and @xcite emphasized the importance of recognizing and mitigating gender stereotypes and other biases in the training data. @xcite measured stereotypical biases using the stereoset benchmark, while @xcite evaluated biases in question-answering tasks using the bbq dataset. these studies provided important insights into the ei and social biases of llms. building on this foundation, the present study aimed to analyze the key factors influencing the emotion prediction of vllms in conversational contexts systematically. specifically, we intended to investigate the impact of factors such as persona, gender, and regional biases on the emotion prediction processes of vllms in depth. ## dataset and task overview we reconstructed the meld dataset @xcite , which is based on popular tv series friends, to investigate the key factors influencing the emotion prediction performance of vllms in conversational contexts. the meld dataset provides full-scene images for each scene and the corresponding conversational context, along with the names of the characters who engage in the dialogue and the emotion and sentiment labels for the feelings of each character. the dataset includes emotion and sentiment labels for each utterance. emotions are categorized into seven types: "fear," "disgust," "joy," "sadness," "surprise," "anger," and "neutral," whereas sentiments are divided into three categories: "positive," "negative," and "neutral." appendix a presents the overall statistics. ## dataset construction we reconstructed the existing meld dataset to align it with the objectives of this study. the data reconstruction process consisted of three stages: 1) dialogue selection, 2) image scope reconstruction, and 3) incorrect sentence selection. using these stages, we constructed data that fit the purpose and removed unnecessary data. the entire process is illustrated in figure 1 . ## experiments and results we measured the performance using three different prompts, considering their influence. the detailed prompts can be found in appendix d. ## analysis we analyzed the performance for each emotion in the emotion prediction. as shown in figure 2 , all prompt types showed the highest performance in predicting the "joy" emotion, with the speaking styles prompt achieving the best result of 50.82%. this suggests that the tone and style of conversation play an important role in predicting positive emotions. the personality traits prompt also showed high performance in predicting "joy," at 49.98%, indicating that individual personality traits are crucial elements in understanding and expressing joy. these results demonstrate that the model can predict positive emotions more accurately based on the personality and speaking style of the speaker. in contrast, all prompt types showed relatively lower performance in predicting "fear" than other emotions. in the case of the speaking styles prompt, the performance for predicting the "fear" emotion was the lowest among all emotions, at 39.58%, and similar trends were observed for the other prompt types. this indicates that predicting negative emotions such as fear is challenging. fear may require complex and subtle contexts and the limitations of the model may be exposed when accurately predicting such emotions. in addition, the prediction of the "neutral" emotion showed relatively low performance across all prompt types, particularly in the personality traits prompt, which had the lowest performance at 39.44%. this suggests that individual personality traits may add complexity to the process of discerning emotional neutrality. neutral emotions are difficult to predict owing to the absence of clear positive or negative signals, indicating that additional research is required to improve the model performance to respond in situations in which clear emotional signals are lacking. ## conclusion this study has systematically analyzed the key factors influencing the emotion prediction performance of vllms. the experimental results showed that the model architecture and size, particularly the llm backbone, had the most significant impact. the integration of persona information and cot prompts exhibited varying effects depending on the model, and differences in the prediction performance were observed based on the image scope for each emotion. however, biases in emotion prediction performance based on gender and region were identified, indicating the need for efforts to mitigate these biases. future research should focus on developing emotionally intelligent vllms by minimizing data and model biases using advanced dataset composition and model training methods.