Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios (2022)

Abstract: In psychotherapy interactions, the quality of a session is assessed by codifying the communicative behaviors of participants during the conversation through manual observation and annotation. Developing computational approaches for automated behavioral coding can reduce the burden on human coders and facilitate the objective evaluation of the intervention. In the real world, however, implementing such algorithms is associated with data sparsity challenges since privacy concerns lead to limited available in-domain data. In this paper, we leverage a publicly available conversation-based dataset and transfer knowledge to the low-resource behavioral coding task by performing an intermediate language model training via meta-learning. We introduce a task augmentation method to produce a large number of “analogy tasks” — tasks similar to the target one — and demonstrate that the proposed framework predicts target behaviors more accurately than all the other baseline models.

## introduction advances in spoken language processing techniques have improved the quality of life across several domains. one of the striking applications is automated behavioral coding in the fields of healthcare conversations such as psychotherapy. behavioral coding is a procedure during which experts manually identify and annotate the participants' behaviors @xcite . however, this process suffers from a high cost in terms of both time and human resources @xcite . building computational models for automated behavioral coding can significantly reduce the cost in time and provide scalable analytical insights into the interaction. a great amount of such work has been developed, including for addiction counseling @xcite @xcite and couples therapy @xcite @xcite . however, automated coding is associated with data sparsity due to the highly sensitive nature of the data and the costs of human annotation. due to those reasons, both samples and labels of in-domain data are typically limited. this paper aims to train computational models for predicting behavior codes directly from psychotherapy utterances through classification tasks with limited in-domain data. recently, substantial work has shown the success of universal language representation via pretraining context-rich language models on large corpora @xcite . particularly, bert (bidirectional encoder representations from transformers) has achieved stateof-the-art performance in many natural language processing (nlp) tasks and provided strong baselines in low-resource scenarios @xcite . however, these models rely on self-supervised pretraining on a large out-of-domain text corpus. in prior works, the data sparsity issue has also been addressed by introducing an intermediate task pretraining using some other high-resource dataset @xcite @xcite . however, not all the source tasks yield positive gains. sometimes the intermediate task might even lead to degradation due to the negative transfer @xcite @xcite . to improve the chance of finding a good transfer source, we need to collect as many source tasks as possible. another approach is metalearning which aims to find optimal initialization for fine-tuning with limited target data @xcite @xcite . this approach also calls for enough source tasks and is affected by any potential task dissimilarity @xcite . considering that psychotherapy counseling takes place in a conversational setting, here we use a publicly available dataset -switchboard-damsl (swda) corpus @xcite -for the intermediate stages of knowledge transfer. unlike most previous meta-learning frameworks, which require auxiliary tasks from various datasets, our work uses only one dataset and produces the source tasks by a task augmentation procedure. the task augmentation framework evaluates the correlations between the source and target labels. it produces source tasks by choosing subsets of source labels whose classes are in one-to-one correspondence with the target classes. using this strategy, we can generate a large number of source tasks similar to the target task and thus improve the performance of meta-learning. the experimental results show that incorporating our proposed task augmentation strategy into meta-learning enhances the classification accuracy of automated behavioral coding tasks and outperforms all the other baseline approaches. we use data from motivational interviewing (mi) sessions of alcohol and drug abuse problems @xcite for the target task. the corpus consists of 345 transcribed sessions with behavioral codes annotated at the utterance level according to the motivational interviewing skill code (misc) manual @xcite . we split the data into training and testing sets with a roughly 80%:20% ratio across speakers, resulting in 276 training sessions and 67 testing sessions. the statistics of the data are shown in we perform the intermediate task with the swda dataset, which consists of telephone conversations with a dialogue act tag for each utterance. we concatenate the parts of an interrupted utterance together, following @xcite , which results in 196k training utterances and 4k testing utterances. this dataset supports 42 distinct tags, with more details displayed in appendix a. ## experimental results we adopt the mi dataset to perform two tasks: predicting the behavioral codes of the therapist and of the patient. we use swda as the source dataset for intermediate tasks to train the bert model with reptile. we set the number of sessions for both the training and validation sets to 1, 5, and 25 to simulate low-resource situations at different levels. we pick sessions randomly to make pairs of training and validation, and we repeat this 15 times. for each level of data sparsity, we report the averaged prediction results over 15 runs to reduce the effect of data variations. ## conclusion and future work this paper leveraged publicly available datasets to build computational models for predicting behavioral codes in psychotherapy conversations with limited samples. we employed a meta-learning framework with task augmentation based on the idea of analogy tasks to address the data limitation problem. we performed experiments at different sparsity levels and showed improvement over baseline methods. besides, we discussed two task sampling strategies and the effect of a hyper-parameters in our framework. in the future, we plan to leverage contextual utterances into our algorithm and generalize our approach to the natural language understanding task in other fields. a more formal approach to find an optimal match between classes from different domains (i.e., labels of conversational descriptors) is also a topic of our ongoing research.