Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning (2025)

Abstract: Previous research has revealed the potential of large language models (LLMs) to support cognitive reframing therapy; however, their focus was primarily on text-based methods, often overlooking the importance of non-verbal evidence crucial in real-life therapy. To alleviate this gap, we extend the textual cognitive reframing to multimodality, incorporating visual clues. Specifically, we present a new dataset called Multi Modal-Cognitive Support Conversation (M2CoSC), which pairs each GPT-4-generated dialogue with an image that reflects the virtual client’s facial expressions.To better mirror real psychotherapy, where facial expressions lead to interpreting implicit emotional evidence, we propose a multi-hop psychotherapeutic reasoning approach that explicitly identifies and incorporates subtle evidence. Our comprehensive experiments with both LLMs and vision-language models (VLMs) demonstrate that the VLMs’ performance as psychotherapists is significantly improved with the M2CoSC dataset. Furthermore, the multi-hop psychotherapeutic reasoning method enables VLMs to provide more thoughtful and empathetic suggestions, outperforming standard prompting methods.

## introduction as a crucial part of cognitive behavioral therapy (cbt), cognitive reframing addresses lots of mental health issues rooted in deeply ingrained negative and distorted thought patterns @xcite @xcite @xcite @xcite . recently, studies attempted to utilize large language models (llms) in this task, highlighting their growing potential in the field of psychotherapy @xcite @xcite @xcite @xcite . conventionally, cognitive reframing has been explored * equal contribution with text-based sentence rewriting methods aimed at shifting negative viewpoints to positive ones @xcite @xcite . concerned that sentence-based cognitive reframing can feel unnaturally imposed, @xcite suggest a three-stage conversational approach with llms encouraging clients to engage more actively and form self-positive viewpoints. despite the promising results of llms in previous systems, non-verbal aspects of psychotherapeutic theory are often overlooked, creating a significant gap between real face-to-face therapy and the systems. in actual psychotherapy situations, recognizing non-verbal emotions is essential for effective communication and is a critical skill closely linked to the therapist's ability to provide effective therapy @xcite . in this study, we propose to extend the concept of cognitive reframing into multimodality, integrating visual clues into the therapy process. our aim is to provide a natural and effective cognitive reframing framework within a multimodal context, incorporating clients' non-verbal clues in the conversation (figure 1 ). to this end, we create a novel synthetic benchmark dataset, multi modal-cognitive support conversation (m2cosc), which pairs each synthetic dialogue with an image of the client's facial expression. to create m2cosc, we utilize the powerful roleplaying capabilities of llms. we leverage gpt-4 vision and gpt-4 @xcite to take on the roles of psychotherapist and client, respectively, simulating therapy sessions as shown in figure 2 . inspired by counseling theory's @xcite @xcite initial disclosure stage among the five stages, we add the introduction phase into our multimodal psychotherapy framework, extending the three-stage model proposed by @xcite : introduction, problem exploration, brainstorming, and suggestion. in addition, to provide explicit guidance based on the client's state, we introduce a multi-hop psychotherapeutic reasoning method. by exploring the implicit evidence necessary for the therapy and then generating responses based on this evidence, the ai therapist can offer adequate guidance after identifying the client's state. we evaluate our approach by conducting extensive experiments with two test scenarios, dialoguelevel evaluation and stage-level evaluation, using both llms and vision-language models (vlms). the results show that training with our m2cosc significantly enhances the counseling capabilities of vlms, surpassing those of existing llms. moreover, the multi-hop psychotherapeutic reasoning method allows vlms to offer more rational and empathetic suggestions, outperforming standard prompting methods. human evaluations further suggest that capturing the client's facial expressions in the system, as practical therapists do, can remarkably assist counseling. to sum up, our contributions are as follows: this work is a first step toward bringing multimodal cognitive reframing into ai-enhanced psychotherapy. by introducing a benchmark dataset and a structured reasoning approach, we hope to inspire future research on leveraging non-verbal cues for more effective therapeutic conversations. ## problem definition and goals in cognitive reframing therapy, a therapist must understand the client's states, which include their problematic situations, distorted thoughts, and thinking traps. the therapist then encourages the client to consider alternative possibilities. building rapport with the client by expressing empathy is also crucial @xcite . in the real-world psychotherapy procedure, these stages involve both verbal and non-verbal information, where the therapist has sufficient ability to understand the client's states. here, our goal is to enhance the abilities of an ai psychotherapist by leveraging non-verbal information, guiding it to focus on facial expressions and to comprehend the client's states. given the client's facial image and dialogue history, we aim to provide empathetic responses while maintaining a consistent focus on the client's issues throughout the procedure, offering rational interventions free from logical errors or contradictions. to achieve our goals, we created the m2cosc dataset founded on three key values-empathy, logical coherence, and guidance-which serve as evaluation criteria in a prior study @xcite . we also utilize overall scores encompassing all three items. (see appendix a for details.) ## results and discussions for reliable comparison, we compared the performance of both llms and vlms with two test scenarios: dialogue-level evaluation and stage-level evaluation. the dialogue-level testbed, which has been used in prior research, allows us to observe how interventions are carried out throughout conversations. however, relying solely on this testbed makes it difficult to accurately compare the ai therapists' abilities due to the variability of the ai client. to better assess interventions in terms of empathy, logical coherence, and rationality, we also conducted a stage-level evaluation. this approach enabled us to compare therapists' interventions more precisely by analyzing turn-level performance on the m2cosc test set, using consistent contextual input across the models. ## conclusion in this paper, we explore cognitive reframing therapy within a multimodal context. recognizing the gap between real face-to-face cognitive reframing therapy and prior research, as well as the potential benefits of ai in psychotherapy, we take an initial step toward enhancing the therapeutic capabilities of ai therapists by incorporating non-verbal cues, particularly facial expressions, into the intervention process. to address the challenge of restricted access to real client data in the field of psychology, which hinders research efforts, we synthetically create a novel multimodal conversational cognitive reframing dataset called m2cosc. our experiments across two test scenarios, dialogue-and stage-level evaluations, exhibit significant improvements in the therapeutic capabilities of vlms when using m2cosc. the proposed multi-hop psychotherapeutic reasoning strategy, which integrates facial expressions, thoughts, and thinking traps, demonstrates superior performance in providing clients with empathetic, logically coherent, and specific rational suggestions.