Deciphering Cognitive Distortions in Patient-Doctor Mental Health Conversations: A Multimodal LLM -Based Detection and Reasoning Framework (2024)

Abstract: Cognitive distortion research holds increasing significance as it sheds light on pervasive errors in thinking patterns, providing crucial insights into mental health challenges and fostering the development of targeted interventions and therapies. This paper delves into the complex domain of cognitive distortions which are prevalent distortions in cognitive processes often associated with mental health issues. Focusing on patient-doctor dialogues, we introduce a pioneering method for detecting and reasoning about cognitive distortions utilizing Large Language Models (LLMs). Operating within a multimodal context encompassing audio, video, and textual data, our approach underscores the critical importance of integrating diverse modalities for a comprehensive understanding of cognitive distortions. By leveraging multimodal information, including audio, video, and textual data, our method offers a nuanced perspective that enhances the accuracy and depth of cognitive distortion detection and reasoning in a zero-shot manner. Our proposed hierarchical framework adeptly tackles both detection and reasoning tasks, showcasing significant performance enhancements compared to current methodologies. Through comprehensive analysis, we elucidate the efficacy of our approach, offering promising insights into the diagnosis and understanding of cognitive distortions in multimodal settings.The code and dataset can be found here: https://github.com/clang1234/ZS-CoDR.git

## introduction the pervasive impact of mental health disorders @xcite , particularly depression and anxiety, poses significant global challenges, with substantial economic costs and profound personal suffering. the world health organization (who) 1 estimates an annual productivity loss of $1 trillion due to these conditions. cognitive distortions, which are inaccurate thought patterns @xcite contributing to negative thinking, play a crucial role in the development and exacerbation of these disorders. while considerable research has focused on detecting cognitive distortions @xcite @xcite , merely identifying them does not provide a comprehensive understanding of the underlying psychological processes in conversations. it is essential to elucidate the origins and thought patterns that give rise to these distortions. in fig 1 , besides the cognitive distortion label, the reasoning includes the type of negative thinking pattern and the trigger, such as the patient's statement about others' comments. the explanation of cognitive distortions (cods) is vital for mental health and therapeutic practices. it enhances diagnosis by providing comprehensive insights into thought patterns and triggers, allowing for contextual analysis of a patient's mental state. this understanding helps therapists design personalized interventions and aids patients in recognizing negative thinking patterns, essential in cognitive-behavioral therapy (cbt). to equip natural language processing (nlp) systems to advance ai and automation, explanations build trust and transparency, encouraging the adoption of ai tools and ensuring decisions are ethically sound. moreover, explanations drive research and development, leading to improved models and interventions in cognitive distortions. this paper embarks on a pioneering endeavor by curating a high-quality dataset of multimedia doctor-patient conversations annotated with cognitive distortion labels and reasoning. despite challenges with dataset size and human interpretation variability, we have diligently curated a reliable, labeled dataset for reasoning. this crucial contribution supports advancements in detecting and reasoning about cognitive distortions in patient-doctor dialogues. by training our model using a zero-shot approach, we aim to enable it to independently recognize subtle cues in conversations and interpret the nuanced facial expressions of patients and doctors. this method allows the model to explain cognitive distortions on its own, using contextual and interactional understanding. our zero-shot model's improved performance over traditional methods highlights the effectiveness of this approach. the key contributions of our work are four-fold: ## related work cognitive distortion is a serious mental health disorder and often is a precursor to many other disorders. the authors in @xcite have compared different techniques, such as logistic regression, support vector machines, bert, and transformer, to detect cognitive distortion and further classify it. although the existing work @xcite has incorporated multimodal patient-doctor interactions to train a multitasking framework to detect cognitive distortion, it does not address the reasoning task. additionally, authors in @xcite ; @xcite have utilized patient-doctor interactions as a dataset for their models, emphasizing their importance for training. our detailed literature review suggests that on mental health disorders, research focusing on reasoning generation is very limited, and in the case of cognitive distortion, there are none to the best of our knowledge. the importance of generating reasoning for the model's detection is highlighted by @xcite ; @xcite . moreover, the importance of incorporating multi-modal input, such as video, and audio of the patient interaction is increasing @xcite ; @xcite ; @xcite ; @xcite as it enhances the performance of the model, thereby improving the diagnosis. hence, by addressing these limitations, we take a step forward to solve the novel task of detecting cognitive distortion from multimodal patientdoctor interaction and generate relevant reasoning for detecting cognitive distortion. to this end, we create a new dataset and propose an effective zeroshot learning approach to solve the task. ## methodology in this section, we first define the problem and then describe our proposed framework, zs-codr's pipeline, and its components.our primary objective is to classify whether a given text contains cognitive distortion or non-cognitive distortion, designated as y , in the k th utterance u @xmath0,1 , u k,2 , . . . , u k,n ), where n is no. of tokens in utterance. each utterance is associated with video v k , and audio a k features, all situated within the broader conversational context h @xmath1, v 1 , a 1 ), (u 2 , v 2 , a 2 ), . . . , (u k-1 , v k-1 , a k-1 )). furthermore, we consider the presence of emotion at the utterance level, denoted as e. our secondary task is to generate the reasoning for detecting cognitive distortion. ## results and analysis main result. in regarding cognitive distortion reasoning generation, we observe significant enhancements of 7.9 and 9.87 decrement( comparisons among different llms our approach is agnostic to any specific llm and aims to identify the most effective one among a range of options. in our study, we employed ten different llms: opt, llama, bloom, mpt, al-paca, vicuna, dolly, stable lm, xlnet, and t5. through rigorous experimentation, we discovered that llama 7b consistently outperformed all the other llms in terms of various evaluation metrics. the superior performance of llama 7b was evident across multiple tasks and datasets. this could be attributed to several factors, including the architecture, pre-training data, and fine-tuning strategy of llama 7b, which enabled it to better capture the complexities of the cognitive distortion reasoning task. consequently, for the purpose of our study, we selected llama as the reference llm for comparison with different baseline models. the detailed results and responses generated by different llms are provided in the appendix to highlight the variability in perplexity and performance. additionally, in figure 3 , we visually demonstrate that llama 7b consistently yields the most favorable results among all tested llms, further supporting our choice for comparison. human evaluation: to assess the quality of the generated reasoning by the zs-codr model, a human evaluation was conducted using a randomly selected sample of 250 instances from the test set. consistent with the experimental results (refer to case study: in figure 4 , we present case studies illustrating zero-shot reasoning segments from the dataset within the context of the cognitive distortion reasoning task. the figure demonstrates that within the dataset, the reasoning generated by our proposed zs-codr with llama-7b framework exhibit higher accuracy, fluency, and information content compared to the baseline zeronlg approach, closely aligning with the actual ground-truth reasoning. the baseline approach tends to produce shorter reasoning, resulting in the omission of context and vital information. it is evident that our proposed approach yields improved reasoning compared to the zeronlg approach and is on par with the gold-standard reasoning provided for the given dialogue instance. additionally, in fig 4 we compare reasoning generated by considering all three modalities and just text modality. the reasoning generated by the multimodal model is more clear and more accurate than the plain text model.since it is difficult to show multimodal features such as eye gaze, body language, e.t.c on the paper, the github link provided in the abstract contains the youtube links for the patient-doctor interactions in the dataset, which emphasize the importance of audio and visual cues. we also showcase different responses generated with different llms in the appendix. ## limitations in addition to the aforementioned points, it's crucial to acknowledge that the nature of patient-doctor dialogues is unique, often involving nuanced communication dynamics and specialized terminology. this specificity could potentially limit the effectiveness of the proposed method when applied to other types of conversations, such as those in legal or educational settings. moreover, the ethical considerations surrounding the use of multimodal data extend beyond mere technical implementation. in sensitive domains like mental health, where confidentiality and trust are paramount, the responsible handling of data becomes even more critical. issues such as the inadvertent disclosure of sensitive information or the potential for algorithmic biases to exacerbate existing disparities in healthcare access and treatment outcomes must be thoroughly addressed. furthermore, while the study may demonstrate promising results within its controlled environment, the real-world variability of conversational data poses challenges to generalization. factors such as diverse linguistic styles, cultural nuances, and contextual cues can significantly impact the performance of any automated system. therefore, ongoing validation efforts across a wide range of datasets and conversational contexts are essential to ensure the reliability and effectiveness of the proposed method in diverse real-world scenarios. ## ethical considerations the rigorous evaluation and review conducted by our institutional review board (irb) ensure that the study adheres to strict ethical standards and safeguards the rights and well-being of all involved parties. it's important to emphasize that the primary objective of this research is to enhance the capabilities of medical professionals in diagnosing and addressing medical health issues, ultimately leading to improved patient care and overall human wellbeing. by leveraging innovative technologies and methodologies, the study aims to empower healthcare providers with valuable insights and tools to enhance medical practice and outcomes. regarding the utilization of youtube videos in the dataset, it's worth noting that these videos are sourced responsibly and ethically. they are freely available online without any copyright restrictions, and their usage is solely for research and educational purposes. furthermore, the dissemination of these videos through various channels serves the overarching goal of advancing scientific knowledge and fostering educational initiatives within the medical community. this transparent approach ensures compliance with legal and ethical guidelines while promoting the exchange of information and collaboration in the pursuit of scientific advancement. ziheng zhang, weizhe lin, mingyu liu, and marwa mahmoud. 2020. multimodal deep learning framework for mental disorder recognition. in 2020 15th ieee international conference on automatic face and gesture recognition (fg 2020), pages 344-350. ieee. tiancheng zhao and maxine eskenazi. 2018. zero-shot dialog generation with cross-domain latent actions. arxiv preprint arxiv:1805.04803.