Thesis Proposal: Detecting Empathy Using Multimodal Language Model (2024)

Abstract: Empathy is crucial in numerous social interactions, including human-robot, patient-doctor, teacher-student, and customer-call centre conversations. Despite its importance, empathy detection in videos continues to be a challenging task because of the subjective nature of empathy and often remains under-explored. Existing studies have relied on scripted or semi-scripted interactions in text-, audio-, or video-only settings that fail to capture the complexities and nuances of real-life interactions. This PhD research aims to fill these gaps by developing a multimodal language model (MMLM) that detects empathy in audiovisual data. In addition to leveraging existing datasets, the proposed study involves collecting real-life interaction video and audio. This study will leverage optimisation techniques like neural architecture search to deliver an optimised small-scale MMLM. Successful implementation of this project has significant implications in enhancing the quality of social interactions as it enables real-time measurement of empathy and thus provides potential avenues for training for better empathy in interactions.

## introduction the ability to understand and respond appropriately to the feelings, viewpoints, and beliefs of others is referred to as empathy @xcite . through engagement, this capacity can strengthen bonds among people and lessen stress and sadness. for instance, consider a situation where a family member falls ill, leading to personal distress. when sharing this sadness with a colleague, receiving genuine support can significantly mitigate unhappiness while enhancing the bond with that colleague. the significance of empathy is evident across a broad range of contexts, from socially assistive robots to human-to-human interactions @xcite . audio converted to text multimodal language model empathy existing frameworks proposed framework figure 1: comparison of our proposed approach with the literature. existing works on text-based empathy detection @xcite ) leverage pre-trained language models. there exist a few works on audioand video-based empathy detection @xcite , where audio features or visual features are separately used mostly in classical machine learning algorithms. in contrast, we propose to leverage both audio and video information in a multimodal language model. assessment of empathy levels in real-life empathy-seeking interactions is crucial for determining the quality of such interactions @xcite . empathy deficit often leads to conflicts and miscommunications, highlighting the importance of measuring empathy levels. evaluating empathy levels can help answer questions such as 'to what extent does a teacher exhibit empathy towards students?', 'how empathic is a caregiver towards patients?' and 'does an employer demonstrate empathy towards employees?', among others. empathy detection allows for a compre-hensive evaluation of empathy levels in various contexts and can facilitate the development of effective strategies to promote empathy in social interactions. the term empathy is itself subjective @xcite , so annotated video datasets of empathy are also scarce @xcite . to this front, we will collect and annotate audiovisual data of dyadic conversations. our annotation protocol will consider the subjective experience of the participants involved in the dyadic conversation. apart from our data, there are some public datasets (such as realtalk @xcite ) where the data closely aligns with the scope of this research but not the annotation. to this end, we propose re-annotating some samples in terms of empathy and training the model using semi-supervised learning @xcite . re-annotating these existing data can be challenging as we may no longer have the subjective experience of the study participants. empathy measurement is challenging for people @xcite , let alone automated systems. there are several works on empathy detection from textual contents @xcite @xcite . research in computational empathy from audio and audiovisual data is emerging with a few existing works @xcite . it is likely that a multimodal model with visual content -in addition to conventional textual content -can boost empathy detection performance because action and gesture play a crucial role in signalling the presence or absence of empathy. large language models, such as gpt-4 and palm, have shown excellent results in performing various complex tasks. recently, multimodal language models (mmlms), by integrating other modalities, such as images, demonstrate promising proof of concept in complex audiovisual recognition tasks @xcite . mmlm, therefore, seems an appropriate and stateof-the-art approach for empathy detection from audiovisual data, and hence, we propose to leverage mmlm as the backbone of our prediction system. however, training an mmlm requires huge computational budgets (e.g., multiple gpus or tpus for multiple days). with a limited computational budget, this research aims to experiment in two aspects: (1) using small-scale mmlms and (2) prompt engineering with large-scale mmlms. firstly, small-scale mmlms, such as minivlm @xcite and simvlm @xcite , or compressing comparatively large-scale mmlms through knowledge distillation @xcite will allow us to leverage mmlms in our experimental setup. one of the key aims of this research is to find an optimised and smallscale mmlm suitable for empathy detection. to this end, we will employ techniques such as neural architecture search, lottery ticket hypothesis or knowledge distillation. secondly, prompt engineering and few-shot learning (fine-tuning with few data samples) with large-scale mmlms shall reduce the huge computational requirement of full training and is thus considered appropriate for our low-computation and low-data scenario. ## preliminary experiments we have experimented with the newsempathy datasets @xcite , where we experimented with fine-tuning three plms (albert, distillbert, and bert). the dataset has numerical demographic information, which enhanced empathy detection because of the subjective nature of empathy. to this end, we constructed meaningful sentences from the numeric demographic information, which in fact, boosted empathy detection performance. as for data-centric improvement, we also leveraged t5based plms for text summarising and rephrasing @xcite . as a follow-up study, we leveraged gpt-3.5 llm to mitigate annotation noise in crowdsource datasets. crowdsourcing is a faster and cheaper way to collect data and annotation in computational social science research, such as empathy. however, crowdsourcing involves many different people who may undertake such jobs only for financial benefit, and thus it becomes difficult to maintain the quality of collected data and annotation @xcite . to this end, we proposed reannotating noisy and misleading annotations using gpt-3.5 llm and mixing these new annotations with human-provided good annotations. apart from this, we also leveraged gpt-3.5 in converting numerical demographic information into meaningful sentences and data augmentation through paraphrasing @xcite . ## conclusion the ability to detect and understand empathy is central to improving social interaction. this phd research proposes a multimodal framework by modelling empathy based on video and speech transcripts in an integrated manner. we start with public datasets and further collect data tailor-made for empathy detection, as most currently available corpora do not fully match the target objectives. with the recent success of multimodal language models (mmlms), this project aims to use cross-modality dynamics and joint representations of multimodal audiovisual data in an mmlm. to accommodate mmlm in a resource-constrained environment, this research aims to adopt optimisation techniques, such as neural architecture search, lottery ticket hypothesis, and knowledge distillation. the proposed mmlm can be used to detect empathy in various settings, such as in education, healthcare, and businesses.