A Multi-Persona Chatbot for Hotline Counselor Training (2020)

Abstract: Suicide prevention hotline counselors aid individuals during difficult times through millions of calls and chats. A chatbot cannot safely replace a counselor, but we explore whether a chatbot can be developed to help train human counselors. Such a system needs to simulate intimate situations across multiple practice sessions. Open-domain dialogue systems frequently suffer from generic responses that do not characterize personal stories, so we look to infuse conversations with persona information by mimicking prototype conversations. Towards building a “Crisisbot” hotline visitor simulation, we propose a counseling strategy annotation scheme and a multi-task framework that leverages these counselor strategies to retrieve similar examples, generate diverse sub-utterances, and interleave prototype and generated sub-utterances into complex responses. We evaluate this framework with crowdworkers and experienced hotline counselors. The framework considerably increases response diversity and specificity, with limited impact to coherence. Our results also show a considerable discrepancy between crowdworker and counselor judgements, which emphasizes the importance of including target populations in system development and evaluation.

## introduction crisis hotlines provide the opportunity for individuals to reach out, be heard, and get help during intense times of need. they are invaluable, highly used resources, as evidenced by the national suicide prevention lifeline centers alone handling 2.2 million calls during 2018 @xcite . further, many helplines have expanded services to chat, in addition to voice calls, to support texting and increased demand. to help visitors who contact them, helplines rely on counselors trained in various active listening techniques and de-escalation strategies @xcite @xcite . as in other high-stakes settings, training hotline counselors suffers from the fundamental challenge that counselors must practice counseling skills without putting any distressed hotline visitors in danger. crisis counselors usually obtain experience by practicing with experienced counselors who role-play hotline visitors. role-playing has been shown to improve crisis intervention @xcite , but requires significant resources that can be challenging for hotlines to provide on tight budgets. for example, it requires experienced counselors' time that could instead be used to help hotline visitors in need. to provide a realistic, low-risk practice environment to augment crisis hotline counselor training, we seek to develop crisisbot: a dialogue agent that can simulate a hotline visitor and be used to train human counselors. a chatbot clearly cannot safely and thus ethically take on a counseling role, so we instead explore this direction of using a chatbot to support human counselors. in particular, we consider whether a chatbot can be used to simulate a visitor to better train human counselors. by chatting with crisisbot, novice counselors could safely practice counseling and build confidence before interacting with hotline visitors. in addition to returning relevant messages, crisisbot needs to simulate multiple visitor stories to begin to expose counselors to the astonishing breadth of scenarios that they may encounter. multiple stories are also needed to support multiple practice sessions, and these visitor stories must be intimate conversations with realistically complex messages (as in figure 1 ). however, collecting pre-defined personas in addition to example conversations burdens a limited pool of skilled workers, so we seek to develop stories by mimicking example conversations directly. developing personas for chatbots is challenging, as generic responses remain a shortfall of dialogue generation models @xcite . generic responses are especially problematic for characterizing distinct personas and for carrying engaging conversations. without some specificity in responses, different scenarios cannot be simulated for multiple training sessions. to simulate multiple, specific, and distinct personas, we propose a counselor strategy annotation scheme and a novel multi-task training framework that constructs persona-relevant responses from example transcripts rather than pre-defined personas. our framework leverages counselor strategies, a pre-trained generative model, utterance segmentation, and sub-utterance selection. it allows for interleaving generated and prototype text to construct complex responses, which are more realistic. we explore the utility of our strategy annotation scheme and multi-task framework in the context of hotline counseling. we use our scheme to annotate a corpus of realistic conversations and then use our framework to leverage these annotated conversations for response construction. we first test the trained model components through interactive chats with untrained crowdworkers to assess response quality and experience. we then test the framework with experienced hotline counselors to additionally assess how realistic and useful the system is perceived to be. the interactive conversations are evaluated with both subjective human judgements of user experience and automatically calculated measures of response diversity. we find that our approach considerably increases the diversity of and the presence of persona-relevant sub-utterances in chatbot responses. we also find discrepancy between crowdworker and counselor evaluation. this discrepancy stresses the importance of our two-fold evaluation approach and the general need for testing within a target setting, especially for specialized systems. while additional improvements are needed for conversational coherence between turns, our annotation scheme and modeling approach contribute to curating chat experiences with richer responses that characterize distinct personas. we study this framework in the context of counseling, but it could be used more broadly in applications that require simulating multiple scenarios. we make anonymized conversations available for research use foot_0 . ## related work our work builds on a growing interest in using natural language processing to address challenges in mental health care. unlike previous work that has predominantly focused on studying the quality of care provided or delivering mental health care with chatbots, we focus on training hotline counselors with a crisisbot. with this approach, we must address the persistent challenge of generic dialogue response generation @xcite to avoid tedious and unrealistic training experiences. we attempt to construct personas by mimicking personas from prototype conversations with a multi-task framework that enables constructing responses from pre-written and generated text. ## dataset to scaffold conversations and help the model identify similar contexts, we develop a counselor strategy annotation scheme. we use this scheme to annotate a dataset of realistic hotline training conversations that was previously collected with experienced hotline counselors @xcite . the counselors were asked to role-play hotline conversations, as during training, and to preserve the privacy of all hotline visitors. these conversations are representative of how counselors currently acquire experience during training. however, as they are synthetic, we are able to consider models that include repeating portions of text without violating the privacy of vulnerable individuals. conversations were required to be 40 total turns in length and allowed to be up to 60 total turns. additional dataset statistics can be found in ## method to simulate realistic hotline visitors, we seek to construct complex responses that characterize specific, consistent personas across multiple turns. towards persona consistency, we base each conversation on a prototype conversation from the corpus and try to mimic the visitor's persona in that conversation. as a result, we do not generate personas, but model conversations on individual training example conversations.to do this, we propose an end-toend multi-task modeling framework that leverages our annotation scheme and a large pre-trained language model. this framework fine-tunes a generative language model while learning two additional model components: a strategy detector to select similar conversational exchanges, i.e., messageresponse pairs, and a next sub-utterance classifier that can interleave generated sub-utterances and sub-utterances selected from the prototype conversation into a coherent response. an overview of using these components to construct responses is in previous output e.g., "i need help" counselor input e.g., "i'm here to help." 2. extract exchanges from prototype conversation that match last strategy 1. classify last input sub-utterance strategy 3. condition on exchanges to generate next sub-utterance candidates generated sub-utterances e.g., "i appreciate it" 0. select prototype conversation from corpus strategy e.g., <here-for-@xmath0." extracted exchanges from prototype e.g., ("we are here to help", "thank you.") dialog history ⨁ sub-utterance decoder for each response for each sub-utterance e examples here 24/7. v: thanks. prototype conversation 4. choose a generated or extracted sub-utterance all prototype sub-utterances strategy classifier exchange extraction next sub-utterance classifier figure 2: overview of process to construct responses to messages input by a counselor. the framework bases each conversation on a single prototype conversation and learns three components to construct responses: a counselor strategy classifier to select relevant exchanges to condition the generator on, a generator that decodes diverse sub-utterances, and a classifier that selects to append either a generated sub-utterance or one selected from the prototype conversation. figure 2. we combine generated and selected prototype sub-utterances so that the model can respond with diverse, specific responses and appropriately respond to unseen contexts. ## experimental setup to evaluate the proposed full framework and ablated versions, we consider human judgements of technical conversation quality and both automatic measures and human judgements of engaging persona development. we collect these measures first with untrained crowdworkers. then, based on crowdworker assessments, we select a subset of models to have counselors assess. not all of the models are evaluated by counselors, as a limited number of experienced hotline counselors were able to be recruited and participate. further, each model needs to be considered multiple times by each counselor in order for them to assess distinctness of personas between conversations. all human ratings used 5-point likert scales. this study was approved by the internal review board. ## evaluation results each of the four models was interactively evaluated by 30 crowdworkers and we calculated measures of model response diversity on all responses generated ( turn hotline counselor crisisbot 0 start hi, i'm michael and i kind of need somebody to talk to today. i'm feeling really down. 1 hi michael, sorry to hear you are feeling down, but i am so glad you reached out. what's up? i'm having suicidal thoughts lately. my life sucks. 2 sounds like things have been really rough, so much that you are thinking about suicide. do you feel like killing yourself today? sometimes i think about it. 3 thank you for sharing with me. have you done anything to harm yourself today? no. i'm feeling a bit better now. but i just want to talk to someone. and i don't know how to help my mom because it hurts me to see that she's in so much distress. 4 okay, i am definitely here to talk. i hear that you want to help your mom, and that you are worried about her. what is going on with your mom? she's been really depressed lately. my parents want to divorce and i'm so stressed that i can't focus on school. ## conclusion we explored developing a crisisbot to imitate hotline visitors' varied personal stories for better training human counselors. towards this goal, we developed a counselor strategy annotation scheme and proposed a multi-task framework. this framework bases conversations on personas from prototype conversations and interleaves generated text with text selected from the prototype. through two user studies, we observed increased response diversity overall, but a considerable difference be-tween crowdworker and counselor perceptions and inconsistency with how ratings reflect conversations. these results reinforce the known challenge of dialogue system evaluation and cautions the use of crowdworkers alone for developing system insights. the disparity in ratings between counselors and crowdworkers stresses that the involvement of specialized, intended users in system development and evaluation is crucial for success. while perhaps intuitive, this result is sometimes overlooked during system evaluation. counselor open feedback indicates that additional development is required for coherence across turns, but that our framework adds detail and variety in personas, which could enable better variety for training counselors.