IMBUE : Improving Interpersonal Effectiveness through Simulation and Just-in-time Feedback with Human-Language Model Interaction (2024)

Abstract: Navigating certain communication situations can be challenging due to individuals’ lack of skills and the interference of strong emotions. However, effective learning opportunities are rarely accessible. In this work, we conduct a human-centered study that uses language models to simulate bespoke communication training and provide just-in-time feedback to support the practice and learning of interpersonal effectiveness skills. We apply the interpersonal effectiveness framework from Dialectical Behavioral Therapy (DBT), DEAR MAN, which focuses on both conversational and emotional skills. We present IMBUE, an interactive training system that provides feedback 28% more similar to experts’ feedback, compared to that generated by GPT-4. IMBUE is the first to focus on communication skills and emotion management simultaneously, incorporate experts’ domain knowledge in providing feedback, and be grounded in psychology theory. Through a randomized trial of 86 participants, we find that IMBUE’s simulation-only variant significantly improves participants’ self-efficacy (up to 17%) and reduces negative emotions (up to 25%). With IMBUE’s additional just-in-time feedback, participants demonstrate 17% improvement in skill mastery, along with greater enhancements in self-efficacy (27% more) and reduction of negative emotions (16% more) compared to simulation-only. The improvement in skill mastery is the only measure that is transferred to new and more difficult situations; situation-specific training is necessary for improving self-efficacy and emotion reduction.

## introduction some conversations can be challenging to navigate @xcite , whether they concern negotiating a salary increase with a boss, discussing healthcare options with an aging parent, or asking a friend to return the money they owe. various communication frameworks assist individuals in conducting such conversations by providing a set of skills to apply @xcite @xcite . however, psychology research highlights that a lack of communication skills is not the only obstacle to effective communication, particularly in emotionally charged situations @xcite . difficult conversations can evoke strong emotions that disrupt effective communication, even for individuals with solid communication skills @xcite . to successfully communicate during challenging situations, it is crucial to focus not only on communication skills but also on managing emotions. the popular dear man framework, from dialectical behavioral therapy (dbt), was originally developed for borderline personality disorder, but is widely used to teach conversational strategies and emotional regulation @xcite . it includes conversational strategies (describe, express, assert, reinforce, and negotiate) and a desired "state of mind" (mindful and confident) for productive conversations. remaining mindful and confident in challenging conversations helps speakers regulate difficult emotions so they can successfully exercise their conversational strategies ( §2;a). currently, dear man skills are mainly taught in therapy sessions and practiced either onsite in a roleplaying setting or at home with paper worksheets, which presents several challenges. access to a trained therapist may be limited due to the significant shortage of mental health professionals @xcite . outside of therapy sessions, static worksheets do not provide opportunities for interactive role-playing and just-in-time feedback necessary for effective learning @xcite @xcite . prior work in nlp has shown the ability of lms to simulate personas and social interactions @xcite @xcite . a few recent works leverage this capability by using lms to weak describe. stick to the fact, how many times is "constantly"? avoid judgmental language. mindfulness check: express frustration directly by using "i statement" can help you gain more mindfulness. confidence check: well done. ! " # $ dad, i want to talk to you about your healthcare. i think we should find an assistant to take care of you and help you with daily activities. kiddo, i appreciate your concern, but i don't need any help. i can take care of myself just fine! you can use the describe skill. describe the facts and express to your dad how the situation makes you feel. situation: my dad has difficulty remembering to take his medication. i want to hire a nurse to take care of him, but he disagrees and thinks he doesn't need help. goal: convince my dad to accept me hiring an assistant for him. in the past two weeks, you forgot about taking medications for 4 times. i am worried about your health. you constantly forget to take your meds and it is so bad for your health. help people improve interpersonal skills @xcite or conflict resolution skills @xcite , without considering emotional regulation. our work extends this literature, focusing on communication and emotional regulation skills simultaneously, incorporating expert domain knowledge into feedback, and grounding strategies in clinical psychology theory. we conduct a humancentered study and make three key contributions. first, we present a formative study and an expert annotated dataset on dear man skill use. we conduct a formative study to gain insights from psychology experts on best practices when simulating challenging conversations and providing finegrained feedback ( §2). to understand how clinicians provide feedback on dear man in their practice and to develop and evaluate our method on real situations, we collect a dataset from crowd workers consisting of difficult situations they encounter and simulated conversations within them (the crowd worker being paired with a role-playing lm partner). we then ask psychology experts specifically trained in teaching dbt skills to annotate these conversations, assessing skill use and offering suggestions for improvement ( §3). second, we develop computational methods to provide feedback using insights from the formative study and collected dataset ( §4). we propose a new prompting strategy, demonstrating contrasting pairs of strong and weak utterances, in addition to state-of-the-art prompting methods. our method improves the accuracy in skill use evaluation, outperforming gpt-4 by 24.8%, and more expert-like, specific, and actionable improvement suggestions. third, we build imbue, an interactive training system that simulates difficult conversations and provide just-in-time feedback backed by lms to support the practice and learning of dear man skills (figure 1 ). imbue can be used at an individual's convenience to practice both communication and emotional regulation. through a randomized trial with 86 participants, we evaluate imbue 's training outcomes on skill mastery, emotion reduction, and self-efficacy ( §6). we show that a simulation-only variant of imbue improves participants' self-efficacy towards having the conversation-boosting confidence (27%) and reducing worries (4%)-and emotion reduction towards the situations-reducing fear (16%) and sadness (12%)-while not improving skill mastery significantly. with the addition of just-in-time feedback, the participants' skill mastery significantly improved by 17.6%, with additional improvement in self-efficacy (confidence, 26.7%) and emotion reduction (fear, 15.7%). ## formative study to inform design to understand how dear man skills are taught in practice, we conduct a formative study with three clinical experts, summarizing crucial insights and corresponding design decisions below. further details on the study procedure are in appendix b. insight 1: guide clients to focus on facts instead of making judgmental comments when describing a situation. we refrain from asking the participants to describe the personality of the conversation part-ner, even though it may help lm simulate a more realistic conversation, and instead only focus on past behaviors that might influence the difficulty of the situation in imbue. insight 2: among the dear man skills, d, e, a, r, n are conversation strategies one can choose for each utterance. mindful and confident are the "state-of-mind". one should always stay mindful and confident throughout the entire conversation. therefore, in each turn, imbue gives participants the option to choose from five conversation strategies. conversation skills are evaluated only if they are selected for use, while mindfulness and confidence are assessed for each utterance ( §5; §6). insight 3: practicing simpler or less emotionally intense situations helps with harder situations. we collect difficulty levels from participants and use the less difficult situation in training ( §6). insight 4: training should prioritize emotion management. it is not considered a successful use of dear man skills if the client negotiates well but gets agitated. we evaluate training outcomes in three aspects: skill mastery, emotion reduction, and self-efficacy ( §6). insight 5: choosing strategies before writing helps learning. we adapt this design in data collection ( §3) and imbue. ## data collection we collect a dataset to understand how clinicians provide feedback on dear man in their practice and to develop and evaluate our method. data collection with crowdworkers. we intend for our system to be used by individuals without specialized knowledge, so we collect data from crowdsourcing platforms. we recruit 20 people from amazon mturk who provided 60 different situations and annotations. each worker provides three conversations, one from each of the following categories: family, social, and work. workers are asked to have conversations with our lm, which was instructed to roleplay as their partner during these conversations. each conversation needs to be at least 10 responses from the worker, or until the simulated conversation partner "agrees" with the worker, whichever comes first. in each utterance, workers need to select one or more strategies they want to use in the given utterances to encourage them to follow the dear man framework as much as they can ( §2). we include more ethics and safety details in §8;i. ## methodology imbue is an interactive training system that simulates bespoke communication situations and provides expert-like just-in-time feedback based on the dear man framework. imbue is backed by lms that perform two tasks: (a) next skill suggestion: before a user writes a message, imbue suggests a skill to apply ( §4.2), (b) feedback on skill use: after a user writes a message, imbue provides skill rating and improvement suggestions ( §4.1). we describe our methods for performing these tasks. ## evaluation of imbue with an expert-annotated dataset we evaluate imbue on the expert-annotated dataset ( §3) with cross-validation. we use imbue to generate skill use feedback ( §4.1) and next skill suggestions ( §4.2) and report the average performance across all conversations foot_1 . since there are no established methods for the proposed new tasks, we use gpt-4 and llama-2 variants as a baseline and conduct the following ablations to assess the impact of each component in imbue (tables 1&2): (1) without contrasting pairs component (retrieval-based in-context learning, with reasoning step and curated rubric), (2) without contrasting pairs or knn-retrieval component (random in-context examples), with reasoning step and curated rubric (3) without in-context examples, only reasoning step and curated rubric, (4) without in-context examples or reasoning, only curated rubrics. skill ratings. to maximize feedback opportunities, we prioritize identifying the distinction between strong vs. not strong skill use, which determines whether the model will provide improvement suggestions. as shown in ## evaluate imbue in a randomized trial we conduct a randomized trial with 86 participants and assess how imbue can help people improve interpersonal effectiveness. ## conclusion in this paper, we demonstrate how human-lm interaction can be used to facilitate interpersonal effectiveness skill learning and practice. we collect a dataset with crowd workers and clinical experts who are specifically trained in the practice of dbt. using this dataset, we develop methods to prompt lms to simulate bespoke communication scenarios and provide just-in-time feedback, grounded in psychotherapy theory. we build an interactive training system imbue, and conduct a randomized user study with 86 participants to assess the effectiveness of the simulation and feedback components of imbue. we find that simulation-only training is effective in improving self-efficacy and emotion reduction, and adding just-in-time feedback shows significantly more benefits in all of skill mastery, self-efficacy, and emotion reduction. the skill mastery can be acquired from practicing with a different situation, while emotion reduction and self-efficacy appear to only benefit from training specifically on the situation.