D eep P avlov at S em E val-2024 Task 3: Multimodal Large Language Models in Emotion Reasoning (2024)

Abstract: This paper presents the solution of the DeepPavlov team for the Multimodal Sentiment Cause Analysis competition in SemEval-2024 Task 3, Subtask 2 (Wang et al., 2024). In the evaluation leaderboard, our approach ranks 7th with an F1-score of 0.2132. Large Language Models (LLMs) are transformative in their ability to comprehend and generate human-like text. With recent advancements, Multimodal Large Language Models (MLLMs) have expanded LLM capabilities, integrating different modalities such as audio, vision, and language. Our work delves into the state-of-the-art MLLM Video-LLaMA, its associated modalities, and its application to the emotion reasoning downstream task, Multimodal Emotion Cause Analysis in Conversations (MECAC). We investigate the model’s performance in several modes: zero-shot, few-shot, individual embeddings, and fine-tuned, providing insights into their limits and potential enhancements for emotion understanding.

## introduction in the dynamic domain of artificial intelligence, the emergence of mllms has gained significant interest due to integrating input from different modalities, such as audio, vision and language, opens up in-depth perceptual and interpretive capabilities in dialogues instead of chat-based dialogue systems @xcite . these models exhibit impressive potential in resolving a lot of challenges and have been deployed across various sectors, including banking support systems, social services, and as adjuncts in psychological assistance. in these applications, the deciphering of user intent and emotions is crucial for generating pertinent responses. consequently, one of the most important domains within mllm research is emotion reasoning. this paper explores a specific facet of emotion reasoning: multimodal emotion cause analysis in conversations (mecac) @xcite . this task involves emotion recognition and matching emotional states with their causes in the context of a conversation, leveraging inputs from different modalities such as text, audio, video and more. despite the remarkable advancements in mllms, their capabilities and limitations in the high-potential area of emotion reasoning remain active topics for research. one of the seminal contributions to the investigation of mllms capabilities within the area of emotional reasoning is delineated in @xcite . the authors introduce a novel task, explainable multimodal emotion reasoning (emer), and proceed to evaluate the efficacy of modern multimodal models in addressing emer. their research focuses on the integration and interpretability of emotional cues across diverse modalities, thereby advancing the understanding of emotion reasoning. to address the described issue, we propose to continue research of the mllms capabilities in emotion reasoning by evaluating one of the most promising models, video-llama @xcite , also explored in @xcite , for mecac on the emotion-cause-in-friends dataset. the work evaluates the performance of the model in three modes: ## conclusion in this paper, we conduct an analysis of the mllm video-llama with an emphasis on its emotion understanding capabilities in mecac. our experiments show that multimodal models, in their current iteration, exhibit limitations in deciphering emotional states under zero-shot and few-shot modes. to enhance the capabilities of such models in emotion understanding, our findings indicate that task-specific dataset fine-tuning is an essential step. despite the challenges observed, the raw embeddings generated by the video-llama model show promising potential as a foundation for improving emotion recognition performance. the implications of this research highlight the necessity for continued development and refinement of multimodal learning frameworks. future work may concentrate on expanding the diversity of the datasets used for fine-tuning to include a broader spectrum of emotional expressions and cultural contexts. this could mitigate existing biases and enhance the model's generalizability across various demographics and scenarios. moreover, incorporating advanced techniques such as transfer learning and domain adaptation could further enhance the model's proficiency in interpreting nuanced emotional states @xcite .