COMMA - DEER : CO mmon-sense Aware Multimodal Multitask Approach for Detection of Emotion and Emotional Reasoning in Conversations (2022)

Abstract: Mental health is a critical component of the United Nations’ Sustainable Development Goals (SDGs), particularly Goal 3, which aims to provide “good health and well-being”. The present mental health treatment gap is exacerbated by stigma, lack of human resources, and lack of research capability for implementation and policy reform. We present and discuss a novel task of detecting emotional reasoning (ER) and accompanying emotions in conversations. In particular, we create a first-of-its-kind multimodal mental health conversational corpus that is manually annotated at the utterance level with emotional reasoning and related emotion. We develop a multimodal multitask framework with a novel multimodal feature fusion technique and a contextuality learning module to handle the two tasks. Leveraging multimodal sources of information, commonsense reasoning, and through a multitask framework, our proposed model produces strong results. We achieve performance gains of 6% accuracy and 4.62% F1 on the emotion detection task and 3.56% accuracy and 3.31% F1 on the ER detection task, when compared to the existing state-of-the-art model.

## introduction each year, mental illness costs the economy almost 12 billion working days. mental diseases are predicted to cost the global economy $16 trillion by 2030 @xcite . despite extensive evidence of the close association between mental health and nearly every major issue in development, people with mental illnesses are among those most at risk of being left out of development programmes. promoting mental health and preventing mental illness should be part of larger efforts to meet the united nations' sustainable development goals. unlike object identification tasks, emotion perception is significantly impacted by personal bias, cultural backgrounds, and contextual information (such as the environment). emotional information can function as an authoritative inner voice in those who suffer from anxiety, despair, guilt, or fear on a daily basis @xcite . normal individuals can suppress or remove the mood influence, while clinical patients cannot stop the inner voice from being viewed as authoritative. in spite of contradictory empirical facts, someone may draw the conclusion that something is true based only on their emotional response. this process is known as emotional reasoning @xcite . for example, while automatic emotion recognition in conversations can facilitate developing intelligent systems, such as empathetic chatbots, customers' feedback assessment, etc., automatic identification of emotional reasoning from conversations can help to perceive the mental states of the involved persons and understand latent vulnerability factors that often raise the likelihood of self-harm (such as suicide); thus helping to determine preventive efforts. a significant increase in tele-health usage seeking mental health services was observed during the peak of the covid-19 pandemic @xcite . despite advancements in certain nations, persons with mental illnesses frequently face human rights abuses, discrimination, and stigma. automated systems based on natural language processing (nlp) approaches can be incorporated into digital interventions, particularly web and smartphone apps (chatbot like woebot 1 , virtual assistant like ellie 2 , etc.) to provide early identification, take preventive measures and provide personalized healthcare, more so in developing countries with a high population and minimal healthcare facilities. the lack of suitable annotated data in the public domain is a serious impediment to mental health research utilising nlp methods. also, there are certain nuances in mental health conversations (shown in figure 1 foot_2 ), that makes them challenging to be addressed using the existent automated systems. the first set of images in the figure displays examples of how visual signals for current textual utterances can be found in past time steps. the second series of images depicts a patient's uneven bodily motions while discussing er experiences at various points in the discussions. furthermore, identifying er utterances is challenging since the automated system must have commonsense reasoning ability to distinguish factual information from others. to this end, we introduce a novel task of detection of emotion and emotional reasoning in conversations in a multitask setting. we develop a commonsense aware multimodal multitask approach for detection of emotion and emotional reasoning (comma-deer) in conversations at the utterance level. we create the deer corpus (detection of emotion and emotional reasoning), which is a multimodal doctor-patient conversational dataset involving various common mental illnesses annotated with emotion and er at the utterance level. we compare our proposed method to various existing state-of-the-art techniques to the presented dataset. empirical evaluation and qualitative analysis show strong performances by our approach compared to the prior works on both tasks. we intend to make the code and data available to facilitate future research in this domain. the main contributions of this work are: 1. we introduce a novel problem of joint detection of emotional reasoning and emotion in conversations exploiting the correlatedness between the two tasks. 2. this work introduces the first multimodal mental health conversational corpus, deer, manually annotated with emotion, presence of emotional reasoning, speaker information, start and end timestamps at the utterance level. 3. we propose comma-deer, a commonsense aware multimodal multitask system for detection of emotion and emotional reasoning utterances in a conversational setting. the rest of the paper is organised as follows. section 2 summarises some of the prior efforts in this subject. following that, in section 3, we discuss the dataset preparation in detail. in section 4, we discuss our proposed methodology for multimodal multitask experiments. we explain the experiments, results and their outcomes in section 5. finally, in section 6, we conclude our work and identify the scope of future work. the research community has focused heavily on utterance and document level emotion recognition @xcite . psychology research @xcite also points to the importance of considering multimodal information to build automated systems to perceive human emotion. given the significance of mental health and its growing influence on society, researchers @xcite are currently developing methods to precisely detect human emotion in the intention of developing mental health therapeutic solutions and better understand mental health issues. recent works @xcite have considered external knowledge resources (conceptnet @xcite , senticnet @xcite , etc.) in building systems for emotion detection in various domains (conversations, suicide notes, etc.). in mental health conversations such as doctor-patient interactions, any factual information or presence of any grounded knowledge, such as a doctor's description of a patient's mental condition or characteristics of a particular illness (examples shown in section 3.2.1), can also be exploited to infuse additional knowledge into the context of the dialogue. data for multi-modal emotion detection is challenging to collect and annotate, especially for lowresource emotions (e.g., disgust, surprise) that are encountered seldom in everyday life, which drives us to investigate this problem. building on existing studies and the limitations of the prior works on related topics, we presented a manually annotated multimodal mental health conversational dataset and devised an automated approach leveraging commonsense knowledge to detect emotions and emotional reasoning utterances in conversations. ## dataset in this section, we discuss the data collection and annotations for various attributes at the utterance level, such as speaker information, start and end timestamps, emotion, er, and factual utterances. ## methodology this section formalizes our task objective and discusses our proposed comma-deer approach. figure 3 illustrates the general architecture of our proposed approach. ## experiments and results in this section, we discuss the experiments performed, and present the results and analysis. ## conclusion this study presented a novel multimodal multitask system for the detection of emotion and er in conversations. it also contributes towards mitigating the problem of scarce availability of annotated corpora by introducing a manually annotated multimodal mental health conversational corpus, deer. empirical and qualitative analysis suggests that (1). most of the existing state-of-the-art systems for conversational data perform poorly when it comes to comprehending er in conversations; (2). the proposed system performs commendably well on both tasks and attains notable improvements from existing comparable methods; (3). performance on the er detection task is considerably improved when we simultaneously learn the correlated task of emotion recognition. manual observation of the dataset shows that the doctor's responses to the patients' er usually follow a pattern, such as responses of bewilderment and denouncement. we believe that knowing the responses of the doctor in the future time steps may significantly improve the performance for the detection of er, which subsequently would enable detection of early warning signs of more serious mental illnesses. in future work, we want to extend this study in the above-mentioned directions as knowledge of the cognitive biases induced by er may lead to novel therapeutic approaches.