Boosting Distress Support Dialogue Responses with Motivational Interviewing Strategy (2023)

Abstract: AI-driven chatbots have become an emerging solution to address psychological distress. Due to the lack of psychotherapeutic data, researchers use dialogues scraped from online peer support forums to train them. But since the responses in such platforms are not given by professionals, they contain both conforming and non-conforming responses. In this work, we attempt to recognize these conforming and non-conforming response types present in online distress-support dialogues using labels adapted from a well-established behavioral coding scheme named Motivational Interviewing Treatment Integrity (MITI) code and show how some response types could be rephrased into a more MI adherent form that can, in turn, enable chatbot responses to be more compliant with the MI strategy. As a proof of concept, we build several rephrasers by fine-tuning Blender and GPT3 to rephrase MI non-adherent Advise without permission responses into Advise with permission. We show how this can be achieved with the construction of pseudo-parallel corpora avoiding costs for human labor. Through automatic and human evaluation we show that in the presence of less training data, techniques such as prompting and data augmentation can be used to produce substantially good rephrasings that reflect the intended style and preserve the content of the original text.

## introduction demands of the modern world are increasingly responsible for causing severe psychological distress in people. world health organization estimates psychological distress affects 29% of people in their lifetime @xcite . the shortage of mental health workers and the stigma associated with mental health further demotivates people from actively seeking help. with the expansion of the internet, many people are seen resorting to peer support platforms such as reddit and talklife to vent their distress. 1 the anonymity associated with these platforms makes it easier for people to discuss their concerns without being affected by the stigma. distress consolation through aidriven chatbots has also become an emerging solution @xcite @xcite . due to the lack of availability of large-scale psycho-therapeutic conversations, researchers are using data scraped from online peer support forums to train such chatbots @xcite . high levels of perceived empathy and information richness make them good candidates for training @xcite @xcite . but since peers are not professionals, the responses contained in such forums can sometimes be unfavourable to address distress (e.g. confrontations, judgments, orders etc.). so, using this data can have severe risks. one solution for this is identifying favourable and unfavourable response types that appear in distress support dialogues and developing automatic means that can propose omission or rephrasing of such unfavourable response types. figure 1 shows an example. to analyze the types of responses in distress support dialogues, we use labels adapted from a wellestablished behavioral coding system named motivational interviewing treatment integrity (miti) code @xcite . it is used in psychology to evaluate how well a mental health provider responds. specific response types from the miti code have shown to increase the likelihood of positive health outcomes @xcite . it defines favourable response types such as questioning, reflecting, and advising with permission and unfavourable response types such as advising without permission, confronting, and self-disclosing (extra-session). in our previous work, we developed a dataset called the mi dataset, to have a comparative understanding of the differences between online support provided by peers and trained counselors. for this, we hired professional counselors to annotate responses given by peers and counselors with labels derived from the miti code. during analysis, we observed that peers' responses tend to be more supportive, and encouraging than counselors' (as observed by the increased percentage of support and affirm labels). but it was also observed that important therapeutic techniques, such as asking more open questions than closed ones, reflections, giving information, advices with permission, and emphasizing speaker's autonomy were lacking in peers' responses and hence require further boosting. one of the major observations was that among the advises given by the peers, 92.86% of them belonged to the category advise without permission, which is mi non-adherent. this percentage was lower in counselor responses, but still accounted for 77.22% of the advises given by counselors. in this work, we aim to detect such advise without permission responses among distress support dialogues and build a rephraser that can rephrase such responses into advise with permission, which is more mi-adherent. first, we detect such responses through a classifier trained on an augmented version of the mi dataset. next, as we do not have human written responses rephrasing advise without permission responses into advise with permission, we use automatic methods such as template-based replacement and retrieval to construct a pseudo-parallel training corpus containing pairs of advise without permission and advise with permission sentences. since rephrasing is a labor-intensive task compared to labeling and we require professionally trained counselors to do this in the distress consolation setting, using our already labeled dataset to construct a pseudo-parallel corpus saved us both time and cost. we apply the same methods on the augmented version of the mi dataset to form a much larger pseudo-parallel training corpus and use these corpora to fine-tune blenderbot @xcite and gpt3 @xcite . some of the models we fine-tune incorporate different forms of prompting with the aim of obtaining a better outcome with less training examples. we evaluate the rephrasers using automatic and human evaluation. the results mainly show when the training dataset is small, prompting improves the performance of the rephrasers across style transfer and semantic similarity dimensions. they also suggest that when the training dataset is large (in our case through data augmentation), pseudo-parallel data generated through simpler methods such as template replacement produce better results. our contributions are four-fold. 1) we develop an mi classifier that can predict 15 different favourable and unfavourable response types derived from the miti code. 2) we propose a methodology to rephrase responses detected as advise without permission into more mi-adherent advise with permission. we show how this can be done in the absence of human written rephrasings by developing pseudo-parallel corpora using different automatic methods. 3) we evaluate these rephrasers using automatic and human evaluation and show how prompting and data augmentation can improve the performance of the rephrasers when there is less training data. 4) finally, we discuss how this method can be applied to boost chatbot responses, making them more compliant with the mi strategy. our code and the datasets can be found at https://github.com/anuradha199 2/boosting-with-mi-strategy ## related work rephrasing responses recognized as advise without permission into advise with perrmission can be identified as a sub-task falling under the task of text style transfer (tst), in which the goal is to automatically control the style attributes (e.g. sentiment, politeness, humor, etc.) of text while preserving the content @xcite . the field of tst involves traditional linguistic approaches as well as deep learning approaches. traditional approaches to tst rely on term replacement and templates @xcite . with the success of deep learning, various neural methods have been recently proposed for tst. given datasets in which there are direct mappings between the text of the source style and the text of the target style, which are referred to as parallel corpora, standard sequence-to-sequence models are often directly applied for tst @xcite @xcite . but parallel corpora are challenging to find because the development of such data often requires costly human labor. thus, tst on non-parallel corpora has become an emerging area of research @xcite @xcite . parallel and nonparallel datasets have been proposed for common sub-tasks of tst such as sentiment @xcite , topic @xcite , formality (rao and tetreault, 2018), politeness @xcite , and humor @xcite transfer. but to the best of our knowledge, this is the first attempt at introducing a new subtask and releasing an nonparallel corpus for style transfer between mi non-adherent advise without permission and mi adherent advise with permission responses. this task is more challenging than the other sub-tasks because it requires the expertise of professional counselors to generate training data. in this work, we release a nonparallel corpus that can be utilized for this task, which is annotated by professional counselors. we also show how automatic methods could be applied to create pseudo-parallel corpora using this dataset, which can be used to train neural models for this task. ## datasets for this work, we used dialogues curated from two online support platforms. the first one is coun-selchat (counselchat.com), in which verified counselors respond to distress-related posts. the coun-selchat dataset available publicly foot_1 contains 2,129 post-response pairs spanning 31 distress-related topics. we also curated dialogues from a carefully selected set of 8 subreddits: mentalhealthsupport; offmychest; sad; suicidewatch; anxietyhelp; depression; depressed; and depression_help, which are popular among reddit users to vent their distress. this dataset, which we call red (reddit emotional distress), contains 1,275,486 dyadic conversations having on average of 2.66 turns per dialogue. in our previous work, we recruited professional counselors to annotate a subset of 1,000 dialogues each from counselchat and red datasets with labels adapted from the miti code 2.0 @xcite and 4.2.1 @xcite . we call this the mi dataset. we used 15 labels for annotation. they are elaborated in the appendices. out of them, we are interested in the labels advise with permission and advise without permission, which are respectively considered mi-adherent and mi non-adherent response types. the mi dataset contains 16,811 annotated responses, out of which 2.87% (484) and 13.5% (2,285) responses are labeled as advise with permission and advise without permission, respectively. to further augment the mi dataset, we used automatic labeling to expand the 15 labels into unlabeled dialogue responses from counselchat and red datasets. we used two automatic methods for this purpose: 1) n-gram-based matching; and 2) similarity based retrieval. ## mi classifier we developed a classifier to automatically classify responses in distress-support dialogues into one of the 15 labels mentioned above. this is an important step that should be followed before rephrasing, since first it should identify the unfavourable responses types. for this purpose, we developed a classifier that consists of a representation network that uses the bert architecture @xcite , an attention layer that aggregates all hidden states at each time step, a hidden layer, and a softmax layer. we used the bert-base architecture with 12 layers, 768 dimensions, 12 heads, and 110m parameters as the representation network. it was initialized with weights from roberta @xcite . we trained three classifiers. the first one was trained on the smaller human-annotated mi dataset (mi gold) taking 80% of the data for training and leaving 10% each for validation and testing. the other two were trained on the mi augmented (union) and mi augmented (intersection) datasets, leaving out the data used for validation and testing in the first case. in all cases, the optimal model was chosen based on average cross entropy loss calculated between the ground truth and predicted labels in the human-annotated validation set. the classifiers trained on mi gold, mi augmented (intersection), and mi augmented (union) datasets reported accuracies of 68.31%, 67.13%, and 73.44% on the mi gold test set, respectively. the reported accuracies on the mi gold validation set were 67.08%, 64.07%, and 72.67%, respectively for the three classifiers. accordingly, the labels collected through the union of n-gram matching and cosine similarity-based methods improved the accuracy of the classifier by 8.33% and 7.5%, respectively on the validation and test sets compared to the accuracies reported when trained on the gold-labeled mi dataset. ## mi rephraser after identifying the favourable and unfavourable response types, we can choose to omit the unfavourable responses or if possible, rephrase them into a more mi adherent form. a label pair that this rephrasing strategy can be applied directly are advise without permission and advise with permission. through n-gram analysis, we could discover some n-gram patterns that are indicative of the label pair advise without permission (e.g. you should, you need to, you musn't) and advise with permission (e.g. it maybe helpful to, i wonder if you can, you may want to consider). these could be identified as style attributes that vary across the responses identified as advise without permission and advise with permission. thus, given a response identified as advise without permission, the goal of the rephraser would be to rephrase the response to be indicative of advise with permission, without changing the semantic content of the response. as mentioned in section 2, this can be identified as a sub-task under the task of text style transfer (tst). tst is formally defined as, given a target utterance x ′ and the target discourse style attribute a ′ , model p(x ′ |a, x), where x is a given text carrying a source attribute value a. in our case, x corresponds to the response identified as advise without permission, a corresponds to advise without permission, and a ′ corresponds to advise with permission. ## automatic evaluation a successful style-transferred output should be able to demonstrate the correct target style and at the same time preserve the semantic content of the original text @xcite . we refer to the first criterion as style transfer strength and the second as semantic similarity. automatic metrics used to evaluate text generation methods such as the bleu score @xcite , rouge (lin and och, 2004), meteor @xcite , word mover distance (wmd) @xcite , character n-gram f-score (chrf) @xcite , bertscore @xcite and cosine similarity based on sentence embeddings @xcite are used in the literature to evaluate the semantic similarity between the original and the rephrased text. the part-of-speech distance @xcite , a metric specific to tst, is also used to measure semantic similarity. @xcite suggest deleting all attribute-related expressions in the text when applying these metrics to evaluate the output of tst tasks. thus, before evaluation, we removed the style-specific phrases discovered during n-gram analysis from the input and output text. to evaluate the style transfer strength, most works use a style classifier to predict if the output conforms to the target style @xcite @xcite . we used the mi classifier trained on the mi augmented (union) dataset to compute the style transfer strength. it is calculated as the percentage of samples classified as advise with permission out of all test samples. blender-based rephrasers that incorporated generic prompting ranked the best across most metrics over all the other blender-based rephrasers. with the smaller pp training corpus, the gpt-3based rephraser that incorporated generic prompting ranked the best across most metrics. but with the larger ppa training corpus, the gpt-3 based rephraser that was trained on simple templatereplaced pseudo-parallel corpora ranked the best across most automatic metrics. ## human evaluation similar to automatic evaluation, we used two human evaluation criteria to rate the rephrased sentences. the first is how close the rephrased sentence is to advise with permission (style transfer strength). the second is to what extent the rephrased sentence preserves the context/meaning of the original sentence (semantic similarity). we used the upwork crowdsourcing platform ( www.upwork.com ) and recruited four professional counselors to rate the rephrased sentences. given the original advise without permission sentence and a list of rephrased sentences generated by the 10 different rephrasers, we asked two questions from the counselors: 1) is the rephrased sentence indicative of advise with permission?; and 2) does the rephrased sentence preserve the original context? the counselors were asked to answer these questions by indicating a rating on a likert scale ranging from 0 (not at all) to 4 (yes it is). along with the rephrased sentences, we also presented them the corresponding advise with permission sentence obtained from the pseudo-parallel corpora in order to gauge the quality of the corpora used for training. the sentences to be rated were presented to them in a random order to reduce bias. as the combined pp test corpus developed on the mi gold dataset is small (only 34 samples), we used 200 randomly selected samples from the combined ppa test corpus developed on the augmented mi dataset to be rated by the human workers. this was to verify the trend of results reported on the pp test corpus. we bundled 9 randomly selected test cases in one batch and allocated two workers to rate each batch. results were calculated based on the average rating given by the two workers. following @xcite we also calculated the average of style transfer strength and semantic similarity ratings to obtain a single score. we computed the inter-rater agreement based on weighted kappa that uses fleiss-cohen weights @xcite and the scores were 0.5870 (moderate agreement) and 0.6933 (substantial agreement) for style transfer strength and semantic similarity, respectively. the average ratings obtained for style transfer strength and semantic similarity for sentence pairs in the pp test corpus were 3.21 and 3.16, respectively. the sentence pairs in the ppa test corpus scored 3.12 and 2.69 in the above two dimensions, respectively. the average ratings being close to 3 with most of them being above 3 suggests that the training corpora used are of substantial quality. ## discussion in this paper, we presented an example on how distress-consoling responses could be boosted with mi strategy. for this, we first developed a classifier that can identify favourable and unfavourable response types as defined by the miti code. then we narrowed our focus to the mi non-adherent response type advise without permission and developed several rephrasers that can rephrase advise without permission responses into mi adherent response type advise with permission. as curating human written rephrasings was costly, we used templated-based replacement and retrieval methods to create pseudo-parallel corpora from gold-labeled and augmented-labeled mi datasets that contained responses from reddit and counselchat platforms. we used this data to train several blender and gpt3-based rephrasers. we also used generic and n-gram-based prompts to see if prompting can improve the rephrasers' performance. automatic as well as human evaluation results suggested fine-tuning on gpt3 gives better results in rephrasing advise without permission responses into advise with permission. data augmentation techniques we used by expanding the miti labels using n-gram-based matching and similaritybased retrieval improved the performance of the mi classifier as well as the blender and gpt3based rephrasers. the results also suggested when the training datasets are small, the use of generic prompting can enable the rephrasing models to produce better results across style transfer and semantic similarity dimensions. but if you are dealing with large datasets (in our case through data augmentation), pseudo-parallel data generated through simpler methods such as template-based replacement can enable the models to generate substantially good rephrasings closer to the required style and semantically similar to the original sentence. in the future, we hope to develop a chatbot that can respond to psychological distress using the red dataset that contain dialogues curated from several mental health-related subreddits. then we hope to improve the responses generated by this chatbot by applying mi boosting at two different levels: one at the data level; and the other at the model level. at data level boosting, we hope to apply the mi classifier and automatically label the responses in the training data itself. by doing so, we will be able to rephrase the mi non-adherent responses such as advise without permission into more mi-adherent responses and omit the other unfavourable responses from the training data. the mi-boosted training data can then be used to train the chatbot. at model-level boosting, a similar methodology can be applied at the level the chatbot is decoding responses (e.g. beam search). not only generative chatbots but also retrieval-based chatbots could be benefited from this methodology. ## limitations certain parts of our proposed methodology, for example, template-based replacement and n-grambased prompting are applicable only when stylespecific linguistic attributes could be identified between the source and the target text. and due to the cost of human labor and the lack of publicly available client-therapist dialogues, the sample size drawn in the study is small and thus may have an impact on the conclusions drawn. our methods have only been tested for the english language. but we believe similar methods could be applied to other languages given they have unparallel corpora tagged with advise without permission and advise with permission labels. the rephrasing methods described in this paper are tested for short sentences with a maximum sentence length of 98 tokens. thus, the scalability of these methods for long text still remains to be tested. when testing the rephrasers, there are some combinations that could be tried other than the ones already tested. for example, more models can be fine-tuned and tested separately on templatereplaced and retrieval-based pp and ppa corpora but incorporating generic and n-gram prompting. in this work, we first combined these two types of corpora before attempting prompting since we could observe better performance on blender when the corpora were combined. in order to have more data, we combined the advise with permission and advise without permission responses present in counselchat and red datasets. but studies show that there are differences in the language used by counselors and peers @xcite . so, there can be linguistic differences between the same type of response in counselchat and red datasets. future work should attempt to identify these differences and ideally rephrase the responses given by peers to reflect the language of the counselors. ## ethics statement data curation: only publicly available data in reddit and counselchat websites were used in this work. analysis of posts on websites such as reddit is considered "fair play" since individuals are anonymous and users are aware their responses remain archived on the site unless explicitly deleted. it is also stated in reddit's privacy policy that it allows third parties to access public reddit content. 3 also, reddit's data is already widely available in larger dumps such as pushshift @xcite . even though the policies allow it, it should be thoroughly noted that this data contains sensitive information. thus, we adhere to the guidelines suggested by @xcite for working with social media data in health research, and share only anonymized and paraphrased excerpts from the dataset so that it is not possible to recover usernames through a web search with the verbatim post text. in addition, references to usernames as well as urls are removed from dialogue content for de-identification.