Curating a Large-Scale Motivational Interviewing Dataset Using Peer Support Forums (2022)

Abstract: A significant limitation in developing therapeutic chatbots to support people going through psychological distress is the lack of high-quality, large-scale datasets capturing conversations between clients and trained counselors. As a remedy, researchers have focused their attention on scraping conversational data from peer support platforms such as Reddit. But the extent to which the responses from peers align with responses from trained counselors is understudied. We address this gap by analyzing the differences between responses from counselors and peers by getting trained counselors to annotate ≈17K such responses using Motivational Interviewing Treatment Integrity (MITI) code, a well-established behavioral coding system that differentiates between favorable and unfavorable responses. We developed an annotation pipeline with several stages of quality control. Due to its design, this method was able to achieve 97% of coverage, meaning that out of the 17.3K responses we successfully labeled 16.8K with a moderate agreement. We use this data to conclude the extent to which conversational data from peer support platforms align with real therapeutic conversations and discuss in what ways they can be exploited to train therapeutic chatbots.

## introduction demands of the modern world are increasingly responsible for bringing adverse impacts on our mental health. world health organization estimates psychological distress affects 29% of people in their lifetime @xcite . shortage of mental health workers and the stigma associated with mental health further demotivates people in actively seeking out help. thus, provision of mental health support through the use of ai-driven conversational agents to complement traditional therapy has become an interesting area of research @xcite @xcite . but one challenge associated with developing such agents is the lack of large-scale psychotherapeutic conversations. they are either limited or are not available publicly due to ethical reasons. nowadays, with the expansion of social media, it could be observed that people use social media platforms such as reddit to vent their distress and peers are seen to actively respond to such posts. these conversations are available in abundance and are publicly accessible through web scraping apis. thus, conversations scraped from such platforms are seen as an alternative to overcome the above challenge @xcite . prior work has found that responses from peers contain higher empathic concern for posts for seeking help as many peers share similar distressful experiences @xcite . but the extent to which responses from peers align with responses from trained counselors remain a major limitation. knowing these differences can shed light on to what extent such conversational data could be used in training therapeutic chatbots and what pre-processing or rephrasing steps that one should take if they are being used for such purposes. though studies have been conducted independently to assess the competency of counselors and peers offering support @xcite @xcite @xcite , studies that comparatively analyse the differences between them are limited. one such study was conducted by @xcite , where they show that a classifier can distinguish between responses provided to help-seeking posts regarding mental health by professionals and peers. @xcite conducted a similar analysis between responses collected from psychotherapists and non-expert dialogue writers and noted linguistic variability in the two types of responses. however, all these analyses are limited to the lexical level. to address the above gaps, we comparatively analyze responses from professional counselors and peers by annotating these responses using labels adapted from a well-established behavioral coding scheme named motivational interviewing treatment integrity (miti) code @xcite . the miti code is used in psychological literature to evaluate how well a mental health practitioner responds to those seeking help with their mental health related issues. specific response types from the miti code have shown to increase the likelihood of positive health outcomes @xcite . for this purpose, we make use of post-response pairs scraped from the counselchat website (counselchat.com) that contains high-quality therapist responses to emotional distress related questions from individuals and dialogues curated from several mental-health related subreddits in reddit, in which peers engage in actively to respond to those seeking help. annotating dialogue responses with labels from the miti coding system is known to be very time consuming and expensive as it requires expert annotators trained in the practice of psychology and careful attention to the labelling task @xcite . this human labour is difficult to find. but the availability of crowdsourcing platforms such as upwork foot_0 and fiverr foot_1 have made it more accessible to find human labour that satisfy our requirements. thus, we were able to recruit professionally trained mental health practitioners through upwork to annotate dialogue responses with labels adapted from the miti code. our annotation pipeline consisted of three stages as depicted in figure 1 . two workers were involved in the first stage and high-quality workers who scored high observed agreements with a peer in the first stage acted as judges to resolve conflicting labels in the second and third stages contributing to increased observed agreement and inter-rater reliability. using these annotations, we conducted a comparative analysis between responses from peers and counselors to identify to what extent they align with each other. based on these findings, we recommend ways of boosting peers' responses to match as close as possible to counselors' responses. the recommendations made in this paper can contribute to improve the perceived therapeutic effectiveness of a chatbot trained on data from peer support forums. our contributions are three fold. 1) we develop an mi dataset having client-counselor and peerpeer dialogues, in which ≈17k listeners' utterances are annotated with labels adapted from the miti code. foot_2 2) we discuss the details of the annotation process followed in increasing the agreement between the workers when annotating with miti codes. 3) based on these annotations, we conduct a comparative analysis between counselors' and peers' responses and draw useful conclusions on to what extent responses from peers align with responses from trained counselors and recommend ways of boosting peers' responses such that it can increase the perceived effectiveness of therapeutic chatbots trained on such data. ## related work motivational interviewing treatment integrity (miti) is a behavioral coding system that measures how well a mental health practitioner uses motivational interviewing in therapy @xcite . it exclusively focuses on the verbal behaviour of a counselor and is used to increase clinical skill in the practice of motivational interviewing (mi). this coding system has been used extensively to to improve the understanding of the counseling practice alone @xcite @xcite . @xcite developed an mi dataset consisting of ≈22k counselors' responses during motivational interviewing encounters annotated with 10 behavioral codes from the miti. @xcite conducted a quantitative study on counseling conversations to measure how various linguistic aspects of conversations are correlated with conversation outcomes. however, the datasets used in such analyses are not publicly available due to ethical reasons. thus, it is difficult to use such resources in training therapeutic chatbots even though real counselor responses are the ideal candidates for this purpose. there are a number of research efforts taken to develop therapeutic chatbots @xcite @xcite . among them, recent work focuses on using dialogue data from peer-support forums @xcite . for example, @xcite developed a knowledge-graph containing distress consoling responses for specific types of stressors using peer-support responses from reddit and utilized it in single-turn dialogues. some studies specifically focus on attributes such as perceived empathy and information richness in mental health related discourse in peer support forums that suggests they are good candidates for training such chatbots @xcite @xcite . but these studies lack comparisons with responses generated by professional counselors. in our work, we mainly attempts to address this limitation. ## methodology in this section, we describe the methodology including the labels chosen to annotate the listeners' statements, the datasets used, and different stages of the annotation process. ## results from the remaining 1,234 sentences for which a label was not agreed upon, 841 (68.15%) sentences were annotated by at least one poor worker whose observed agreement with a peer was below average. at the end of stage 3 of the experiment, which was conducted by removing such annotations given by the poor workers, a second judge was able to agree with one of the two remaining labels for 784 sentences, yielding an observed agreement of 93.22%. the updated inter-rater agreement (fleiss' kappa) after the third stage was 0.5453 (moderate agreement), showing a slight increase compared to the score in the previous stage. the lower kappa scores are potentially due to the inherent difficulty of distinguishing some of the mi labels, which we elaborate below. a similar annotation experiment conducted by @xcite report similar kappa scores ranging from 0.31 to 0.64 on different mi labels. the confusion matrices computed at each stage of the experiment are denoted in the appendices. we could observe that the label pair complex reflection -give information had the highest percentage of disagreement between the two workers in both counselchat and red datasets. in addition, the label pairs advise with permission -advise without permission and give information -advise without permission were highly confusing to differentiate in the counselchat dataset. whereas, in red, the label pairs affirm -support and advise without permission -direct were difficult to be differentiated. these observations were quite intuitive since these pairs of labels either contained similar semantic content (e.g. complex reflection -give information, advise with permission -advise without permission, give information -advise without permission, advise without permission -direct) or used similar language constructs (e.f. affirm -support, advise without permission -direct). final aggregated statistics of the three stages of the annotation process is shown in ## analysis of the mi dataset in figure 2 , we show the distribution of labels adapted from the miti code in counselchat and red datasets, separately. questions: there is ≈23% and ≈26% increase in closed and open questions, respectively, in counselor responses from cc compared to peer-support responses from red. questioning plays a central role in therapeutic interactions as it builds up mutual dialogue between client and therapist @xcite . @xcite observed with time-limited counseling, fact finding through closed questions is rated lower in helpfulness. it can result in the speaker saying less and less and the listener feeling pressured to ask more questions to keep the interaction going. however, in both counselchat and red datasets, the number of open questions is nearly half of the number of closed questions. hence, mechanisms should be devised to increase the percentage of open questions to balance the number of closed questions. this combination would be more effective than a disproportionate reliance on closed questions. reflections: the number of reflections is positively associated with the perceived empathy @xcite . it is also a competence indicator in assessing mi competency @xcite . non-surprisingly, both simple and complex reflections are observed to be higher (≈20% and ≈30% increase in simple and complex reflections, respectively) in counselors' responses compared to peers'. thus, it would be beneficial to boost the percentage of reflections among peer support dialogues when using them to train therapeutic agents. scholars emphasize that listeners should formulate more reflections than closed questions @xcite . as we observed, some closed questions such as "are you eating because you are bored?" are identical to reflections, differing only in the voice intonation at the end. they could be easily reformulated into reflections such as "it seems that you are eating because you are bored". ## discussion and conclusion this paper discussed the curation process of a largescale distress consoling dialogue dataset containing utterances from trained counselors and peers. a carefully designed annotation process was followed examples of intra-session disclosure in counselchat: -personally, i can tell you that i would want my clients to tell me about anxiety they feel 100% of the time.