Roleplay-doh: Enabling Domain-Experts to Create LLM -simulated Patients via Eliciting and Adhering to Principles (2024)

Abstract: Recent works leverage LLMs to roleplay realistic social scenarios, aiding novices in practicing their social skills. However, simulating sensitive interactions, such as in the domain of mental health, is challenging. Privacy concerns restrict data access, and collecting expert feedback, although vital, is laborious. To address this, we develop Roleplay-doh, a novel human-LLM collaboration pipeline that elicits qualitative feedback from a domain-expert, which is transformed into a set of principles, or natural language rules, that govern an LLM-prompted roleplay. We apply this pipeline to enable senior mental health supporters to create customized AI patients as simulated practice partners for novice counselors. After uncovering issues with basic GPT-4 simulations not adhering to expert-defined principles, we also introduce a novel principle-adherence prompting pipeline which shows a 30% improvement in response quality and principle following for the downstream task. Through a user study with 25 counseling experts, we demonstrate that the pipeline makes it easy and effective to create AI patients that more faithfully resemble real patients, as judged by both creators and third-party counselors. We provide access to the code and data on our project website: https://roleplay-doh.github.io/.

## introduction the application of llms in simulations holds great potential for a variety of interactive applications, ranging from social skill training systems as ai practice partners @xcite to prototyping tools that use them as believable proxies of human behavior @xcite . however, achieving realistic and reliable simulations remains a significant challenge, due to issues such as caricature @xcite , bias, and limited domain knowledge. existing methods for improving llm contact emails: {rylouie, diyiy}@stanford.edu 1 https://roleplay-doh.github.io/ simulations such as finetuning @xcite can help, but such methods typically require the use of application-specific datasets. in sensitive application domains like mental health, privacy concerns with obtaining the required data can restrict the feasibility of such methods. this suggests that experts-in-the-loop may be a powerful alternative to guide the evaluation and refinement @xcite of llm-powered simulations. however, how to involve experts when improving simulations is an open challenge. collecting sufficient amounts of binary or preference data from experts for post-training @xcite can be tedious and expensive. experts can guide the prompting of llm simulations, directly by editing their own prompts or indirectly through testing and thinkaloud sessions. however each prompting method has its limitations: domain-experts may not know how to prompt simulations for desired behaviors @xcite ; and indirect methods are inefficient as it requires a designer or researcher to translate qualitative insights into prompt-design changes. as a focal example, we consider the problem of creating ai patients that serve as roleplay partners to enable varied and interactive practice opportunities for novice therapists and counselors @xcite . creating realistic simulations by finetuning on mental health data is infeasible because therapy transcripts with real patients is difficult to obtain due to privacy concerns. naively prompting llms fail to resemble typical behaviors of real-patients-for example, mental health experts report that patients use colloquial language and can show resistance to help @xcite . to date, no system supports counseling experts, who are familiar with real-patient behaviors but are unlikely to have the technical expertise to write effective prompts, to customize an ai patient themselves. to address this limitation, we aim to enable human-llm collaboration for realistic simulation by developing a novel interactive tool, called roleplay-doh, that empowers domain experts to directly guide the creation of simulations by providing qualitative feedback without any explicit prompting. our initial tool design adopts an intuitive and effective paradigm for user-driven chatbot assistant design @xcite : experts customize a set of principles, or rules written in natural language that govern its behavior @xcite -by (1) interactively critiquing responses in natural language that then (2) gets transformed by an llm into well-formulated principles describing how the llm simulation should act from now on for example, "respond to encouraging words with hesitation, doubting their significance" (fig 1 ). the principles are then used along with a persona description to generate roleplay responses. in our initial tests of the tool with expertcounselors, we found that even with expert refinement via principles, the llm-simulations had difficulty delivering high-quality responses consistently. our analysis of gpt-4 prompted simulation revealed that in 20% of responses, the simulation had difficulty adhering to multipart principles and misapplying those principles that are only applicable in specific contexts e.g., only when the therapist provides encouraging words. to resolve these issues, we introduce a novel principle-adherence pipeline in the final tool design. the first stage in the pipeline decomposes multipart and contextual principles into a set of yes/no questions that are easier to judge, and the second stage assesses the applicability of each simplified principle to the current scenario before self-refining @xcite the ai patient response as required. we conducted a detailed evaluation of roleplaydoh to assess its human-llm collaboration pipeline, focusing on how expert feedback helps develop more authentic ai patients for training. in a within-subjects study involving 25 expert counselors, participants created ai patients either by describing real-patient scenarios only or by additionally using roleplay-doh to define simulation principles. the results show that roleplay-doh enables counselors to produce ai patients that are more authentic, closely resemble real cases, and are better prepared for training use, as judged by creators and third-party counselors. further, thirdparty counselors ranked responses generated by roleplay-doh's full principle-adherence pipeline, as well as three ablations of the pipeline and a baseline that does not self-refine its output to any extent. the full pipeline achieves the highest principle following (35% wins i.e., ranked higher than the no-self-refine baseline; 5% losses i.e., ranked lower than the no-self-refine baseline) and dialogue consistency (35% wins; 10% losses) compared to all ablations. this work highlights the limitations of existing llm simulation systems in specialized, data-scarce domains like mental health counseling, and designs and validates a tool that enables expert counselors to directly customize llm simulations of ai patients. since roleplay-doh does not contain any technical components specifically tailored for mental health, we hypothesize that it can be used to build realistic llm social simulations for a variety of domains, with appropriate expert feedback systems tailored to the domain. ## designing for simulated roleplay we take a human-centered design approach to developing a tool for expert counselors to create and customize an ai patient for eventual use as a simulated training partner. after designing an initial version of our tool, we pilot test it with experienced peer counselors to understand any remaining challenges to effective human-llm collaboration when creating and customizing an ai patient. ## roleplay-doh roleplay-doh helps counseling experts create customized ai patients based on scenarios from their past experiences. roleplay-doh uses llms in two ways: principle elicitation and response generation with principle-adherence, which we describe in more detail below: principle elicitation roleplay-doh enables counselors to customize an ai patient to resemble a real-patient case by eliciting their qualitative feedback and transforming it into constitution principles that dictate behavior. we expect these principles to be internally consistent because domain-experts have the real-patient case from their memory to reference @xcite when evaluating the simulation responses. we provide some examples of principles defined by expert counselors in ## user study using roleplay-doh to evaluate how roleplay-doh can aid counseling experts in creating ai patients, we conducted a within-subjects study with 25 counseling experts, comparing: (1) a scenario-only dialogue simulation, where the counselor writes a patient scenario description, and (2) a scenario+expert-principles simulation, where the counselor uses roleplay-doh to define principles. see §g for full study setup. we evaluate the ai patients created by counselors on criteria inspired by prior work evaluating standardized patients, who are trained human actors, on their ability to roleplay a case @xcite . counselors rated the two ai patients based on 6 dimensions ( we recruit 25 counseling experts with real-world experience in mental health support to perform the evaluation, categorized by their primary expertise: 1) those who are pursuing or have completed degrees in counseling or clinical psychology with practicum experience; 2) those who provided online counseling to over 30 clients on the 7 cups platform; and 3) peer counselors who have provided in-person or virtual support. the ai patients created were grounded in real-patient cases that counselors had encountered in their past experience. these patients represented people with different severity of mental health disorders: 14 out of 25 ai patients were simulating people seeking counseling support for specific issues but without a diagnosed mental disorder (e.g., family issues); 10 out of 25 ai patients were simulating scenarios describing symptoms of mental disorders or previous diagnoses of mental health disorders; 1 out of 25 patients could be categorized as having serious mental illness (history of symptoms of psychosis, substance abuse, and severe impairment of life activities). ## evaluation of principle-adherence we now evaluate whether the principle-adherence pipeline improves the quality of responses for roleplay-doh, along with an ablation analysis showcasing the utility of its various components. specifically, we break down the evaluation of model responses along three metrics: m1) are they consistent with the patient description and conversation history? m2) do they exhibit an awkward style of speech? m3) do they adhere to the provided principles? we evaluate the performance of our principleadherence pipeline [full] over (1) gpt-4 response generation without our pipeline [no critique]; to analyze how the pipeline mitigates errors that arise in base gpt-4 generations, we select 40 conversation turns from our user study logs that fall into one of the error categories described in §3.2 as testcases. each testcase contains the scenario, conversation history up to that point, and the expertdefined principles for the ai patient. for each test case, responses are generated for all models and then ranked by expert counselors from 1 (best) to 5 (worst) for metrics m1 and m3, along with "yes" or "no" annotations for m2. finally, experts pro-vide an overall ranking , along with a brief textual explanation. we allow multiple responses to have the same rank and randomize order of responses to minimize positional bias (details in §l). we treat [no critique] as our baseline and compare all other models to it. we report preference results based on majority vote across 3 expert counselor annotations (fig 3 ). we find our @xcite method performs better than [no critique] on m1 (w: 35%; l 10%) and on m3 (w: 35%; l 5%), where it has the highest win/loss rates compared to all ablations. on overall rankings, it again has the strongest performance (w: 30%; l 15%). we find that the performance of [full] compared to [no critique] is weaker on overall than m1 and m3. this is because the annotators often used their own subjective judgements (e.g.,"although the middle response ranked third on principle following, it feels like the most realistic response in this scenario") to perform the overall ranking, resulting in unpredictable and subjective results. we also find that [naive] has a disproportionately high tie rate across metrics, indicating that it rarely produces better responses even for error cases. this highlights the importance of the principle-as-questions rewriter and automatic principle generator for improving responses. for m2, after majority voting, annotators report that 2.5% of responses are awkward for the [full] method, as compared to 15% for [no critique], 7.5% for [naive], 7.5% for [no principle rewrites] and 15% for [no autogenerated criteria]. therefore, our principle adherence pipeline substantially reduces the occurrence of awkward style in responses (by a margin of 12.5%). the 12.5% gap in percent-age of awkward responses between [full] and [no autogenerated criteria] also indicates the importance of the automatic principle generator for producing realistic rewrites. we repeat these experiments with 50 randomly picked conversation turns and report results in §k, along with krippendorff's α numbers. ## conclusions this paper introduces roleplay-doh, a tool that empowers domain experts to create llm simulations through the automatic conversion of expert feedback into natural language principles, and validates the tool for the task of creating ai patients that serve as roleplay partners for novice counselors. roleplay-doh's novel principle-adherence pipeline also addresses gaps in existing simulation methods by reducing the prevalence of responses that do not follow expert-defined principles or dialogue conventions. studies with mental health counselors creating and comparing ai patients demonstrate that roleplay-doh allows experts to refine llm simulators to be authentic and more ready as practice partners. roleplay-doh could be generalized to support domain-experts in creating realistic simulations in other social dialogue domains, such as roleplay practice for teaching, coaching, conflict resolution, and negotiations, as future work.