We Care: Multimodal Depression Detection and Knowledge Infused Mental Health Therapeutic Response Generation (2024)

Abstract: The detection of depression through non-verbal cues has gained significant attention. Previous research predominantly centred on identifying depression within the confines of controlled laboratory environments, often with the supervision of psychologists or counsellors. Unfortunately, datasets generated in such controlled settings may struggle to account for individual behaviours in real-life situations. In response to this limitation, we present the Extended D-vlog dataset, encompassing a collection of 1,261 YouTube vlogs. Additionally, the emergence of large language models (LLMs) like GPT3.5, and GPT4 has sparked interest in their potential that LLMs can act like mental health professionals. Yet, the readiness of these LLM models to be used in real-life settings is still a concern as they can give wrong responses that can harm the users. We introduce a virtual agent serving as an initial contact for mental health patients, offering Cognitive Behavioral Therapy (CBT)-based responses. It comprises two core functions: 1. Identifying depression in individuals, and 2. Delivering CBT-based therapeutic responses. Our Mistral model achieved impressive scores of 70.1% and 30.9% for distortion assessment and classification, along with a Bert score of 88.7%. Moreover, utilizing the TVLT model on our Multimodal Extended D-vlog Dataset yielded outstanding results, with an impressive F1-score of 67.8%

## introduction depression is a prevalent and significant medical condition. it hurts one's emotional state, thought processes, and behaviour. it manifests as persistent feelings of sadness and diminished interest in previously enjoyed activities. this condition can give rise to various emotional and physical challenges, affecting one's ability to perform effectively both at work and in personal life. depression symptoms range from mild to severe and can include persistent sadness, loss of interest in once-enjoyable activities, appetite changes, sleep disturbances, fatigue, psychomotor changes, feelings of worthlessness, cognitive challenges, and, in severe cases, suicidal thoughts. symptom severity varies, requiring careful clinical evaluation for diagnosis and treatment (cleveland clinic). motivation: according to the statistics of the world health organisation (who) (world health organization) 3.8% of the world's population experience depression, including 5% of adults less than 60 years of age (4% of men and 6% of women) and 5.7% of adults above 60 years of age. approximately 280 million people have depression which depression is 50% more common in women than men. depression is 10% more in pregnant women and women who have just given birth @xcite . if the depression is left untreated, it can lead to several serious outcomes such as suicide @xcite . 1 currently, there is a lack of mental health practitioners globally, with a ratio of 1 : 10000 mental health professionals per patient. due to the shortage of mental health professionals, we aim to reduce the overreliance on them by automating the prediction of depression and providing therapeutic responses to users, which can mitigate distress to some extent. ## related work with the rise in mental health conditions, there is growing interest in detecting depression. however, there is a shortage of datasets for this purpose, largely due to privacy concerns, limiting public availability. among the few publicly accessible datasets, the daic-woz @xcite is notable, featuring clinical interviews in text, audio, and video formats, relying on self-reporting via the phq-8 questionnaire. another dataset, the pittsburgh dataset, primarily contains audio and video clinical interviews. despite its small size of 189 samples, the daic-woz @xcite remains valuable for research. the avid-corpus, used in avec 2013 @xcite and 2014 @xcite competitions, includes video recordings of various activities with self-reporting conducted in the presence of mental health professionals. while these datasets provide insights into depression patterns, their assembly in controlled environments may not fully represent typical behaviours of depressed individuals. the use of social media for depression detection is increasingly preferred over clinical interviews due to its ability to capture patient's authentic behaviour. unlike supervised interviews, social media datasets reveal atypical behaviors exhibited in daily life but fail to capture "in the wild" behavior-those day-to-day activities that laboratory settings miss. "in the wild" datasets for depression detection are often more useful than those collected in laboratory settings because they record natural behaviors and interactions, providing more accurate and applicable data. these datasets encompass a wide range of contexts and a more diverse population, enhancing the generalizability of the findings. additionally, "in the wild" datasets capture spontaneous and authentic responses, reducing the influence of artificial settings that can distort behavior in laboratory environments @xcite @xcite . in recent years, depression detection using text from social media has been focused on @xcite , @xcite , @xcite . various ap-proaches have emerged to detect depression using data from platforms like twitter, reddit, and facebook, focusing on textual-based features such as linguistic characteristics. for instance, @xcite ) utilized text and tags from micro-blogs in china to extract behavioural features for depression detection. however, there is a growing need to explore video data and multimodal fusion for more comprehensive detection methods. multimodal fusion combines various modalities to predict outcomes, and it's increasingly used for depression detection. @xcite utilized 3d facial expressions and spoken language features to detect depression. @xcite integrated text and video features, employing deep and shallow models for depression estimation. @xcite proposed an end-to-end deep neural network integrating speech, facial, and text features for emotional state estimation. although previous studies have explored depression detection using multimodalities, the combination of multimodal transformer with wav2vec2 features and spectrograms remains unexplored despite its potential for superior results. ## datasets the d-vlog dataset @xcite is a collection of depression vlogs of various people posted on youtube. the d-vlog @xcite dataset has 961 vlogs in total out of which 505 are categorized as depressive vlogs and 465 are categorized as non-depressive vlogs. however, the d-vlog dataset @xcite has some limitations, such as the dataset majorly having major depressive dis-order and lacking other disorders such as bipolar disorder, postmortem disorder, and anxiety with depression. which does make the dataset more generalized. so, we extended the d-vlog dataset @xcite by adding 300 more vlogs to the d-vlog dataset @xcite . the dataset now includes more vlogs on various depressive disorders from different age groups, specifically targeting individuals aged 40 and above, as well as a greater number of vlogs from males, which were previously underrepresented. refer figure 1 . for more details on data collection (appendix b). ## methodology the system is comprised of two stages: the first involves detecting depression, and the second focuses on generating a therapeutic response. detection: detection of depression where the video, audio and text are provided as input to the tvlt model for depression detection. generation: provide a therapeutic response to the depressed user. the utterance that was given previously to detect depression. the same text utterance will be fed to the virtual assistant to find the type of distortion classification and after that, we generate the responses. ## result and discussion in this section, we will cover the results on the extended d-vlog dataset @xcite , the clinical diac-woz dataset and results on distortion classification and response generation. ## summary, conclusion & future work this study introduced an extended d-vlog dataset with 1261 videos, including vlogs by individuals with depression (680 videos) and those without (590 videos). our goal is to detect depression in non-verbal and non-clinical vlogs using a tvlt @xcite model, a multimodal transformer. we utilized text, video, and audio data, with visual embeddings from the vit model and audio features from wav2vec2 @xcite and spectrograms. the tvlt @xcite model, incorporating all modalities, achieved an f1-score of 65.6, which improved to 66.3 with text word masking and 66.1 with frame masking. our tvlt model, along with wav2vec2 @xcite and spectrogram features, outperformed all baseline models on the d-vlog dataset and set a new benchmark on the extended d-vlog dataset. our mistral model achieved impressive f1 scores of 70.1 and 30.9 for distortion assessment and classification, along with a bert score of 88.7. in future, we are planning to create an llm-based psychologist agent to converse with the user.