Assessing Motivational Interviewing Sessions with AI -Generated Patient Simulations (2024)

Abstract: There is growing interest in utilizing large language models (LLMs) in the field of mental health, and this goes as far as suggesting automated LLM-based therapists. Evaluating such generative models in therapy sessions is essential, yet remains an ongoing and complex challenge. We suggest a novel approach: an LLMbased digital patient platform which generates digital patients that can engage in a text-based conversation with either automated or human therapists. Moreover, we show that LLMs can be used to rate the quality of such sessions by completing questionnaires originally designed for human patients. We demonstrate that the ratings are both statistically reliable and valid, indicating that they are consistent and capable of distinguishing among three levels of therapist expertise. In the present study, we focus on motivational interviewing, but we suggest that this platform can be adapted to facilitate other types of therapies. We plan to publish the digital patient platform and make it available to the research community, with the hope of contributing to the standardization of evaluating automated therapists.

## introduction the rapid advancements in large language models (llms) have created unprecedented opportunities for their application in clinical psychology. our study focuses on utilizing these models in the context of motivational interviewing to develop llm-based patients with varied and intricate patient characteristic profiles, aiming to emulate the dynamics of real-world therapeutic interactions. motivational interviewing (mi) is a psychotherapeutic technique designed to aid individuals in addressing their ambivalence toward behavioral change, employing a collaborative and clientcentered approach @xcite . this study seeks to replicate the complex interplay between patient and therapist using llms, thereby offering a new perspective on therapeutic communication, as well as a practical method for evaluating attempts at automating psychological counselors. traditionally, mi sessions are assessed by mental-health professionals using specific coding and evaluation frameworks, like the motivational interviewing skills code (misc) foot_0 and the motivational interviewing treatment integrity (miti). 2 these coding frameworks are designed to capture the nature of responses given by the therapist during their conversation with the patient. using these coding frameworks for evaluation is labor-intensive, as it requires professionals to read through the conversations and assign codes to each utterance. furthermore, randomized control trials intended to evaluate clinical protocols are exceedingly costly and time-consuming due to the human burden. given this context, llm-based evaluation appears timely. to find an automated method for evaluating a therapist's performance, one approach could be to use similarity metrics. these would compare the automatic therapist's responses with those of professionals in similar therapist-patient scenarios. this approach faces two major challenges: first, creating a comprehensive set of "gold-standard" conversations is difficult due to the extensive variability in potential scenarios; additionally, current text-similarity metrics are primarily tailored for comparing semantic similarity, rather than assessing how a response influences the overall objectives of the therapy. to address these challenges we take a different approach. we created digital patients using llms and explored their potential in evaluating the effectiveness of therapeutic sessions. in our experiments, we have created 96 patient characteristic profiles, each defined by specific characteristics such as targeted behavioral change, gender, initial level of motivation and so forth. the feasibility of using llm-based patients was assessed via three types of therapists, each represented by an llm. these therapists were configured with varying levels of therapeutic skills: poor, average and expert. here we evaluate whether the llm-based patient could assess the three types of llm-based therapists accordingly, in a controlled environment, and we aspire to extend this evaluation to real-life settings involving human therapists in future research. using the conversations conducted between the llms representing patients and therapists, we design a new evaluation metric, based on pre-existing self-report questionnaires intended for humans, to ensure a comprehensive assessment of the conversation's quality. for every conversation, a third llm-based agent was utilized for the questionnaire response. this agent is provided with the conversation between the therapist and the patient, as well as the questionnaire itself. through statistical analysis, including methods frequently used in self-report questionnaire analysis to test their reliability (e.g., cronbach's alpha) and validity, our study aims to shed light on the efficacy of llms in mimicking patient-like therapeutic communication. in the following sections, we will first provide some background and discuss related work. then, we will describe our methodology in detail. finally, we will summarize the results obtained from conducting several experiments. ## method in this section, we describe the methodology used to generate the conversations between the therapists and the patients, using llms. our approach involves creating distinct patient characteristic profiles through prompt engineering for an llm. for all the experiments reported in this paper we use openai's gpt-3.5-turbo-1106. all together, we constructed 96 unique patient characteristic profiles, varying across multiple parameters such as gender, age, targeted behavioral change (such as smoking or obesity), the duration of the habit, pre-vious attempts at managing it, and the level of cooperation in motivational sessions. to test the validity of our approach, the patients engaged with three types of therapists, each representing a different level of therapeutic skill: poor, average, and expert. the conversations between a therapist and a patient were crafted carefully, with each interaction generated utterance by utterance. 3 this approach ensures that every utterance not only logically followed the previous one but also stays true to the distinct patient characteristic profiles of the participants. importantly, the llms used for the therapist and patient in each interaction were different and independent, allowing for authentic responses in line with their predefined role in the conversation, i.e., patient and therapist, and characteristic traits. therefore, substituting the llm-based therapist with a human therapist who interacts through a chat console represents the logical progression and something we aim to explore in future work. in the prompts for both patient and therapist, we incorporated instructions on how to end the conversation. after a conversation ended, we recorded it and submitted it for evaluation. this evaluation was conducted within a fresh llm session which was tasked with answering two questionnaires regarding satisfaction with the session and the alliance between patient and therapist, essentially filling in questionnaires that are typically expected from human patients. in the following sections we provide details about the prompt we used for each agent. ## discussion there is growing interest in applying llms as automated therapists (e.g., @xcite , as well as attempts at commercial products. however, caution is required, especially as llms are not fully understood and can be unpredictable; it is crucial to develop robust, reliable and valid methods for measuring their quality. provided that existing similarity based metrics are probably not sufficient, we suggest using digital patients as an evaluation platform. in this study, we propose measuring the quality of a chat-based therapist, automated as per our research but applicable to human therapists as well, by engaging them in motivational interviews with our digital patients. we show that a digital patient, implemented using an llm, can fill in questionnaires related to such sessions, and that the ratings are both reliable and valid. in this study, we demonstrate that the model can distinguish among three levels of therapist expertise; future work will have to determine if the measurement can be further refined. the validity of the digital patients is enhanced by a human expert analysis; future work will need to involve more systematic blind evaluation by multiple experts. we note that the variance in the rating by the llm is very low. this is reflected in very high cronbach's alpha values, close to 1, whereas human studies rarely yield values over 0.9. thus, while we provide a wide range of digital characteristic profiles, we do not claim that our platform replaces a complete human population. increasing variance in the model responses, to obtain a better approximation of a human population can be achieved by simple statistical methods such as adding noise or model temperature. however, our goal in this study is not to replace human participants for psychological studies (as discussed by @xcite ); rather, our main goal is to allow for a standard, reliable and valid method for evaluating digital therapists. to that end, we intend to make the digital-patient platform available, and hope it can be further extended, explored, and utilized by the community to ensure responsible use of ai in mental health and clinical psychology. ## limitations a notable limitation of this study is its constrained scope of human evaluation, as the assessment of the sample sessions was conducted by only one expert. we hope to extend this evaluation with multiple human experts, which will facilitate systematically comparing human-human agreement vs human-ai agreement. additionally, our method provides an evaluation on a single dimension: session quality. while we consider this the most critical measurement and use two different questionnaires for it, it might be beneficial to broaden the method to encompass multiple evaluation dimensions and employ a more diverse range of questionnaires. furthermore, our quality measurement of the therapist is based on only three levels of quality, poor, average, and expert; a more refined scale may be desired. as mentioned in section 4.4, the digital patients exhibit a relatively narrow scope of advice. in a similar vein, the numeric ratings assigned to the sessions demonstrate limited variance, a point elaborated upon in the discussion section. finally, we are aware of the possibility of generating text which may be considered as problematic, particularly in sensitive domains such as mental health. although our experiments did not observe this issue, it is crucial to acknowledge that gpt-3.5, despite its implemented safeguards, may still sporadically generate inappropriate responses. this remains an area for continuous vigilance and improvement. these limitations are typical of current llms. the extent of their impact and the need for additional research will vary depending on the specific use case.