Would you like to tell me more? Generating a corpus of psychotherapy dialogues (2021)

Abstract: The acquisition of a dialogue corpus is a key step in the process of training a dialogue model. In this context, corpora acquisitions have been designed either for open-domain information retrieval or slot-filling (e.g. restaurant booking) tasks. However, there has been scarce research in the problem of collecting personal conversations with users over a long period of time. In this paper we focus on the types of dialogues that are required for mental health applications. One of these types is the follow-up dialogue that a psychotherapist would initiate in reviewing the progress of a Cognitive Behavioral Therapy (CBT) intervention. The elicitation of the dialogues is achieved through textual stimuli presented to dialogue writers. We propose an automatic algorithm that generates textual stimuli from personal narratives collected during psychotherapy interventions. The automatically generated stimuli are presented as a seed to dialogue writers following principled guidelines. We analyze the linguistic quality of the collected corpus and compare the performances of psychotherapists and non-expert dialogue writers. Moreover, we report the human evaluation of a corpus-based response-selection model.

## introduction the idea of developing conversational agents as personal healthcare agents (pha) @xcite has gained growing attention in recent years for various domains including mental health @xcite @xcite . most of the conversational agents in the mental health domain are created using rulebased and simple predefined tree-based dialogue flows, resulting in limited understanding of the user input and repetitive responses by the agent. these limitations lead to shallow conversations and weak user engagement @xcite . the major reasons for such limitations are the complexity of conversations, the lack of dialogue data and domain knowledge. the conversations about mental state issues are very complex because they usually encompass personal feelings, user-specific situations, different spaces of entities, and emotions. in this domain, the state-of-the-art data-driven frameworks are not applicable and domain knowledge is very scarce. the two main approaches to collect dialogue data for the purpose of developing data-driven dialogue agents are either acquiring user interaction data via user simulators and hand-designed policies @xcite , or to collect large sets of human-human conversations in different user-agnostic settings @xcite @xcite . these approaches have been used for goal-oriented agents (e.g. reservations of restaurants) or open-domain agents answering questions about a finite set of topics (e.g. news, music, weather, games etc.). however, neither of the above approaches can address the need for personal conversations which include user-specific recollections of events, objects, entities and their relations. last but not least, state-of-the-art conversational agents cannot carry out engaging and appropriate singleuser multi-session conversations. however, personal conversations' requirements include the ability of carrying out multi-session conversations over several weeks or months. in this paper, we propose a novel methodology to collect corpora of follow-up dialogues for the mental health domain (or domains with the similar characteristics). psychotherapists deliver interventions over a long period of time and need to monitor or react to patients' input. in this domain, dialogue follow-ups are a critical resource for psychotherapists to learn about the life events of the narrator as well as his/her corresponding thoughts and emotions in a timely manner. in figure 1 we describe the proposed workflow for the acquisition of personal dialogue data aimed at training dialogue models. we first collect a dataset of personal narratives written by the users who are receiving cognitive behavioral therapy (cbt) to handle their personal distress more effectively foot_0 . in the next step, the narratives are used to generate stimuli for the follow-up conversations with an automatic algorithm. the first part of the stimulus, the common-ground statement, contains the summary of the narrative the user has previously left and the associated emotions and the second part is a follow-up question aimed at reviewing the users life events. in the last step, the stimuli are presented to writers and they are asked to generate a conversation based on the provided stimulus by impersonating themselves as both sides of the conversation, an approach introduced firstly by @xcite , where in our setting the sides are the pha and the patient. the main contributions of this paper can be summarized as follows: knowledge grounded dialogue corpora previously published research have addressed the problem of collecting dialogue data starting from world knowledge facts or predefined persona descriptions. in this regard, zhang et al. (2018) collected a dataset of conversations conditioned on synthetic persona descriptions for each side of the dialogue using amazon mechanical turk (amt) workers. gopalakrishnan et al. (2019) collected a dataset of dialogues grounded in world knowledge by pairing amt workers to have a conversation based on selected reading sets from wikipedia and the washington post over various topics. furthermore, rashkin et al. (2019) have crowdsourced a dataset mental health support dialogue corpora the research in this domain is very recent and resources are scarce. "counseling and psychotherapy transcripts" published by alexander street press foot_1 is a dataset of 4000 therapy session transcriptions on various topics, used as a resource for therapistsin-training. @xcite collected a dataset of 277 motivational interviewing (mi) session videos and obtained the transcriptions for each session either directly from the data source, or by recruiting amt workers. @xcite conducted a data collection process of therapeutic dialogues in wizard of oz manner where the therapists were impersonating a personal healthcare agent. the authors recorded 324 sessions of therapeutic dialogues which were then manually transcribed. furthermore, in the physical health coaching domain, @xcite collected a dataset of conversations where the expert impersonates a pha that engages the users into a healthier life style. for this purpose, a certified health coach interacted with 28 patients using a messaging application. ## dialogue follow-up generation methodology the type of dialogues that we aim at obtaining is different from what has been reported in the literature. while previous works attempted to collect inthe-field therapeutic interactions and convert them into dialogue datasets, we present an elicitation methodology to generate a dataset of follow-up dialogues in the mental health domain, grounded in the personal narratives and with the same conversational structure that the psychotherapists use in order to review the events with the patients in a timely manner. ## evaluation using the introduced elicitation methodology, we collected a corpus of follow-up conversations from the two writer groups foot_4 . we then performed an analysis on the obtained conversations to evaluate the ## conclusions in this work, we address the need for suitable dialogue corpora to train personal healthcare agents in the mental health domain. we present an elicitation methodology for dialogues in the mental health domain grounded in personal recollections. using the proposed methodology, we collected a dataset of follow-up dialogues that psychotherapists would hold with the patients to review the personal events and emotions during a cbt intervention. through an analysis of the collected resource following our proposed methodology, it emerged that the task of validating responses and generating dialogues in the mental healthcare domain can be performed both by using psychotherapists and non-expert dialogue writers. therefore, it suggests the possibility of training a larger number of nonexpert dialogue writers using appropriate guidelines to obtain a valid dataset with less cost while ensuring consistency in the results. furthermore, we investigated the appropriateness of the collected corpus for developing conversational agents in the mental health domain. we reported automatic and human evaluation of a corpusbased response-selection baseline. we found that the test users who interacted with the model over a long-term period (50 days) considered on average 91% of system turns as appropriate and coherent, resulting into 72% of dialogues with acceptable quality. we believe the proposed methodology can be used to tackle the problem of resource scarcity in the mental health domain. in particular, our methodology can be used to obtain corpora of dialogues grounded in personal recollections for developing dialogue models in the mental health domain.