PATIENT - \psi : Using Large Language Models to Simulate Patients for Training Mental Health Professionals (2024)

Abstract: Mental illness remains one of the most critical public health issues. Despite its importance, many mental health professionals highlight a disconnect between their training and actual real-world patient practice. To help bridge this gap, we propose PATIENT- \psi , a novel patient simulation framework for cognitive behavior therapy (CBT) training. To build PATIENT- \psi , we construct diverse patient cognitive models based on CBT principles and use large language models (LLMs) programmed with these cognitive models to act as a simulated therapy patient. We propose an interactive training scheme, PATIENT- \psi -TRAINER, for mental health trainees to practice a key skill in CBT – formulating the cognitive model of the patient – through role-playing a therapy session with PATIENT- \psi . To evaluate PATIENT- \psi , we conducted a comprehensive user study of 13 mental health trainees and 20 experts. The results demonstrate that practice using PATIENT- \psi -TRAINER enhances the perceived skill acquisition and confidence of the trainees beyond existing forms of training such as textbooks, videos, and role-play with non-patients. Based on the experts’ perceptions, PATIENT- \psi is perceived to be closer to real patient interactions than GPT-4, and PATIENT- \psi -TRAINER holds strong promise to improve trainee competencies. Our code and data are released at https://github.com/ruiyiw/patient-psi .

## introduction one in eight people globally are living with mental health conditions (world health organization, 2023) 2 . however, there is a significant gap between the available mental health support and patient needs, with over half (54.7%) of adults with a mental illness receiving no treatment in the us 3 . train-*major contributors. see §a for individual contributions. ing mental health professionals requires extensive effort, yet many professionals highlight a disconnect between their training and the complexities of real patient interactions. to understand these training challenges, we conducted a formative study involving semi-structured interviews with twelve mental health experts and trainees. this diverse group comprised of clinical psychologists, licensed social workers, and current master's students in social work. the experts provided insights into the difficulties faced when transitioning from formal cbt training to real-world practice (details in appendix b). all experts noted that their training did not adequately prepare them for the unpredictable and multifaceted nature of real patient interactions. despite wanting more interactive experiences, they found role-playing exercises with peers, a common training method, to be unrealistic, as these exercises often do not reflect actual therapy sessions. there has been growing interest in developing llm-based methods for psychology @xcite . in @xcite , chatgpt and gpt-4 are able to solve some basic theory of mind tasks that generally require the ability to understand and attribute mental states to oneself and others. inspired by such promise, we propose to use llms to simulate patients to train mental health professionals, with the goal of bridging the gap between their existing training methods and the complexities of real patient interactions. however, two major challenges must be addressed to realize this idea: fidelity. how can we build simulated patients that closely resemble the communicative behaviors of real patients with mental health disorders? effectiveness. how can we design an effective training scheme that allows trainees to benefit from interacting with these simulated patients? in this work, we claim that integrating a patient's cognitive model with an llm can achieve high fidelity in simulating real patients with mental health disorders corresponding to that cognitive model. we implement this idea using the cognitive modeling framework in cbt @xcite , a popular paradigm in psychotherapy. we propose patient-ψ, a novel simulated patient agent that integrates cognitive modeling with llms. we collaborate with clinical psychologists to curate a dataset, patient-ψ-cm, which comprises 106 high-quality and diverse patient cognitive models. these cognitive models cover unhealthy cognitive structures embedded in multiple contexts, such as family issues, relationship problems, workplace challenges, and more. we then use these cognitive models to program an llm to act as the patient-ψ agent. to better resemble the complex dynamics of real patient communications within a therapy session, we also integrate six conversational styles into patient-ψ. these conversational styles were identified from our formative study with mental health domain experts. in cbt, formulating a patient's cognitive model is a crucial skill that therapists need to learn @xcite . our design of patient-ψ naturally incorporates a feedback mechanism for trainees to practice this skill without extensive need for supervisor intervention, which is a desired benefit of ai-based training. we propose patient-ψ-trainer, an interactive training framework for mental health trainees to practice cbt cognitive model formulation using patient-ψ. specifically, trainees converse with the simulated patient, patient-ψ, to formulate its cognitive model. afterward, the system displays the original cognitive model that was used to program the simulated patient as a reference, allowing trainees to compare their results as feedback. within this training framework, the effectiveness of the feedback theoretically depends on how accurately patient-ψ simulates a real patient with the corresponding cognitive model. figure 1 illustrates the overall idea of our framework. to evaluate the fidelity of patient-ψ and the effectiveness of patient-ψ-trainer, we conducted a thorough user study with 20 mental health experts and 13 trainees. evaluation results from the experts indicate that: (1) patient-ψ closely resembles real patients in terms of maladaptive cognitions, conversational styles, and emotional states; outperforming gpt-4. (2) practicing with patient-ψ-trainer is perceived to be highly beneficial for improving cbt formulation skills and better-preparing trainees for interactions with real patients. experts also highlighted several advantages of patient-ψ-trainer, including customized options to choose conversation styles and the diverse patient cognitive models. evaluation results from the trainees indicate that practicing with patient-ψ-trainer is perceived to improve skill and confidence, compared to current training methods. overall, experts and trainees prefer using patient-ψ-trainer over a strong gpt-4 baseline. we also demonstrate that automatic evaluations with llms fail to assess the simulated patient fidelity, indicating the challenge of our task. our contributions are summarized as follows: ## methodology in this section, we first describe the construction of patient-ψ in §2.1. we detail the integration of cognitive models with llms and the incorporation of conversational styles to accurately mimic real patient interactions. next, we explain the training framework, patient-ψ-trainer, in §2.2, which utilizes patient-ψ to create an interactive learning environment for practicing cbt formulation skills. figure 2 provides an overview of our method. 2.1 patient-ψ using cognitive models to simulate patients. cognitive models in mental health provide a structured framework for understanding how an individual's thoughts and beliefs are interconnected and influence emotions and behaviors. in established therapy paradigms like cbt (beck, 2020), formulating a patient's cognitive model is central for a therapist to understand and address the maladaptive cognitions maintaining distress and symptoms (hollon and beck, 2013; hofmann et al., 2012). the cognitive conceptualization diagram (ccd) (beck, 2020) is a commonly used representation of a patient's cognitive model in cbt. the left side of figure 2 depicts an example ccd-based cognitive model, illustrating eight key components. 1 ⃝ relevant history contains significant past events that contribute to an individual's mental state. 2 ⃝ core beliefs are deeply ingrained perceptions about oneself, others, and the world. 3 ⃝ intermediate beliefs are the underlying rules, attitudes, and assumptions derived from core beliefs and shape an individual's thought patterns. foot_0 ⃝ coping strategies are techniques used to manage negative emotions. an external event or context ( 5 ⃝ a situation) may trigger quick, evaluative thoughts without deliberation ( 6 ⃝ automatic thoughts) stemming from the beliefs, leading to responses in terms of 7 ⃝ emotions and 8 ## experiment setup we now present the experimental setup for evaluating patient-ψ and patient-ψ-trainer. we aim to answer the following research questions: rq 1 fidelity: does patient-ψ improve the fidelity of patient simulations compared to baselines? rq 2 accuracy: how closely does patient-ψ imitate the underlying cognitive model? rq 3 effectiveness: do experts and trainees perceive patient-ψ-trainer as an effective tool for cbt training? rq 4 autoeval: can we leverage existing methods, such as llms, to automatically evaluate the patient simulations? in §4, we answer the first three rqs through our user study with both trainees and experts. then, in §5, we show that current automatic evaluations cannot capture the nuances necessary for conducting highly technical, domain-specific assessments. this finding not only shows the importance of user study evaluations but also motivates future work on performant automatic evaluators. ## automatic evaluation results given the potential of using llms for evaluating text generation quality @xcite , we attempt to automatically assess the fidelity and accuracy of patient-ψ and the baseline using two state-of-the-art llms as evaluators: gpt-4 (openai, 2023) and llama 3 70b (ai@meta, 2024). we evaluate over the 40 conversation histories between the experts and patient-ψ in our user study. llm-based evaluators tend to underestimate patient-ψ's fidelity in favor of gpt-4 baseline. following rq 1 (fidelity), the llms are prompted to provide ratings on a 5-point likert scale assessing the fidelity of how closely the simulated patient resembles real patients following the same dimensions used in the user study. in figure 5 , paired t-tests show that the fidelity of patient-ψ, as evaluated by both llms (µ: 3.53; µ: 3.06, respectively), is consistently lower than expert evaluation (µ: 4.10; @xmath0.01 for the differences between expert and both llms), contrasting with the user study results. gpt-4 assigns the highest fidelity scores to the gpt-4 baseline. all fidelity dimensions demonstrate the same trend (see appendix f). ## related work our work is broadly related to the recent use of llms in psychology, education, and computational social science @xcite @xcite @xcite @xcite @xcite @xcite @xcite . in contrast to existing research on using llms for cbt, which focuses on cognitive distortion detection @xcite @xcite and negative thoughts reframing @xcite , our work aims to provide realistic and interactive scenarios for cbt professional development by simulating diverse patient types using llms. as a result, our work most closely relates to research that leverages llms for simulation-based training, particularly communication skill learning and emotion management grounded in dialectical behavioral therapy @xcite , social skill training @xcite , and clinical diagnosis @xcite . our work is the first to ground llmbased simulations in clinical psychology theory by leveraging cbt-based cogntive models to program llms, incorporate a natural curriculum and feedback mechanism in the training tool, and perform evaluation in context with mental health trainees and professionals rather than crowdworkers. ## conclusion in this paper, we introduce patient-ψ, a simulated patient that integrates cognitive models with an llm to accurately mimic the communicative behaviors of real patients. we propose patient-ψ-trainer, where trainees engage in role-playing therapy sessions with patient-ψ and attempt to formulate the underlying cognitive model. user studies with both mental health experts and trainees demonstrate the high fidelity of patient-ψ and the training effectiveness of patient-ψ-trainer, showing improvements over existing training methods and outperforming a gpt-4 baseline. our framework has the potential to transform mental health professional training and be generalized to broader training protocols and therapy paradigms.