Evaluating Automatic Speech Recognition Quality and Its Impact on Counselor Utterance Coding (2021)

Abstract: Automatic speech recognition (ASR) is a crucial step in many natural language processing (NLP) applications, as often available data consists mainly of raw speech. Since the result of the ASR step is considered as a meaningful, informative input to later steps in the NLP pipeline, it is important to understand the behavior and failure mode of this step. In this work, we analyze the quality of ASR in the psychotherapy domain, using motivational interviewing conversations between therapists and clients. We conduct domain agnostic and domain-relevant evaluations using standard evaluation metrics and also identify domain-relevant keywords in the ASR output. Moreover, we empirically study the effect of mixing ASR and manual data during the training of a downstream NLP model, and also demonstrate how additional local context can help alleviate the error introduced by noisy ASR transcripts.

## introduction evaluating the quality of psychotherapy is an essential step in assessing the fidelity of treatment and providing feedback to practitioners. in psychotherapy practice, this is usually done through a process called behavioral coding that consists of manually analyzing recordings of therapy conversations and then labeling specific behaviors from participants. recent efforts have addressed the automatic analysis and evaluation of psychotherapy quality, including the study of conversational dynamics between therapists and clients, the analysis of empathy and emotional responses, and the automatic assessment of therapist's skills @xcite @xcite . most of these research studies have been conducted using small collections of manually transcribed counseling conversations due to the need of an accurate representation of what is being said during the conversation. however, the use of manual transcription restricts the inclusion of a larger number of conversations into the analysis as it is a costly and slow process, making it challenging to apply data hungry machine learning approaches. as an alternative, some studies have explored the use of automatic speech recognition (asr) systems that are able to quickly transcribe a large number of conversations @xcite . however, there are several open questions regarding the feasibility of using automatic transcriptions in the evaluation of psychotherapy @xcite . in this work, we study the quality of asr in counseling conversations and its impact on the task of behavioral coding. we use an existing dataset of behavioral counseling conversations consisting of audio recordings and manual transcriptions as well as annotations of ten behaviors related to therapists' counseling skills. we start by generating automatic transcriptions using a commercially available asr system @xcite . using the resulting parallel corpus of manual and asr transcriptions, we conduct an assessment of the asr quality using three main approaches. first, we use automatic evaluation metrics such as word error rate (wer) and semantic distance to conduct domain agnostic evaluations of the asr performance across conversation participants. second, we conduct a domainspecific examination of the asr output by identifying domain-relevant keywords using behavioral codes and keywords identified using the linguistic inquiry and word count (liwc) @xcite . finally, we study the effect of the noisy asr on the downstream behavioral coding task and empirically show that additional local context in the form of neighboring utterances can help alleviate the impact of asr errors. we believe that studying the role of asr systems in the nlp pipeline is an important step to develop and evaluate robust systems for better understanding of counseling dialogues. ## related work as the overall accuracy of asr systems keeps improving, the ability of producing accurate transcriptions of conversational data has enabled the development of nlp applications in health. particularly, in the psychotherapy domain, where a large fraction of therapy sessions are conducted in spoken language, asr can help reduce the burden of manual transcription, potentially allowing for largescale analysis of interactions between counselors and patients. there have been several efforts on applying nlp on conversation analysis and utterance coding tasks in the psychotherapy domain. nlp was used to evaluate counselor behaviors and strategies @xcite @xcite , or to provide feedback by generating appropriate responses to client utterances @xcite . while most of previous work was conducted on manual transcriptions, there are only a few cases where automatically generated transcripts have been used, limiting the use of computational methods in psychiatry @xcite . the main reason behind this is the need for reliable asr systems that are able to produce accurate transcriptions as the error introduced by transcribing words incorrectly can have a great impact on the performance of the overall application. it has been pointed out by previous research that automatic evaluation metrics such as word error rate alone are not a good indicator of accuracy in speech understanding @xcite . our work is similar to @xcite recent work in that we use both agnostic and domain-relevant approaches to assess asr systems in the mental health domain. however, we additionally investigate how the asr error, both domain-agnostic and domain-relevant, propagates through the common nlp pipeline, in training and inference times, and provide an advice for researchers. finally, mani et al. (2020) recently framed postprocessing asr error correction as a machine translation task from noisy transcription to ground truth transcription, and trains a sequence to sequence error correction model. although this approach can provide a modular solution to mitigate asr errors in many speech understanding systems, we note that building such a parallel corpus can be prohibitive for many researchers. the dataset also includes utterance-level annotations for ten behavioral codes from the motivational interviewing treatment integrity (miti) coding scheme, the current gold standard for evaluating mi fidelity. miti is focused on therapist language only and measures how well the therapist adhered to mi strategies by counting behaviors such as asking questions, using reflective language, seeking collaboration and emphasizing autonomy, among others. the dataset annotations were conducted by annotators with previous mi experience and trained on the use of miti system. in addition to the miti coding, our study uses two additional categories for utterances that are not labeled in the original dataset. the first includes therapist's speech that is not labeled under any miti code (nat) and the second includes client's utterances (nac). ## domain-agnostic evaluation we start by conducting a domain-agnostic evaluation of the automatic transcription process that considers that the accuracy of the asr system is equally important for all speech in the conversation. to this end, we focus on two automatic evaluation metrics: word error rate and semantic distance. the first one evaluates transcription error at the wordlevel; the second one aims to evaluate transcription error considering the semantic distance between the asr output and the ground truth i.e., human transcription. ## domain-relevant evaluation although the domain-agnostic evaluation can provide insights into the aggregate performance of an asr system, a domain informed evaluation can help to better understand the quality of derived transcriptions and its potential impact on downstream tasks. in the counseling domain, incorrect transcription of words or phrases related to emotion, mental state, addiction, or medication can cause more harm than the incorrect transcription of other types of words. seeking to evaluate the role of domain on asr quality in our automatically transcribed conversations, we focus on speech that is relevant to counseling quality. to identify such speech, we use the behavioral coding provided in the dataset and also word categories from the linguistic inquiry and word count (liwc) lexicon @xcite . behavioral codes. we measure wer and semantic distance on utterances coded with the ten counselor behaviors included in the dataset and also examined transcription error in uncoded utterances from both, therapists and clients. for wer, we first concatenated all the utterances labeled with a given code in each single conversation, and then averaged the obtained wer across all conversations. semantic distances for each utterance are averaged over all utterances in the dataset. liwc categories. liwc is a psycholinguistic lexicon that maps words and its stems to a set of categories related to psychological processes. there are 69 predefined categories that cover four highlevel topics: psychological processes, personal concerns, linguistic dimensions, and linguistic fillers. for our analysis, we identify and select a subset of categories from psychological processes and personal concerns as they have been found relevant to psychotherapy conversations. for words in the different categories appearing in the ground truth utterances, we evaluated whether the asr system was able to correctly transcribed them. we calculate the true positive, false negative, and false positive rates as well the standard metrics of recall and precision. ## the role of asr on the automatic evaluation of psychotherapy beyond studying the domain-agnostic and domainrelevant error patterns of the automatic transcrip-tion, we also study the relationship between the speech transcription step and the later behavior code classification, where asr transcriptions are fed as input. ## conclusion and lessons learned in this work, we conducted an evaluation of automatic speech recognition in the counseling domain using conversations between counselors and clients. to measure the degree of transcription error introduced by the use of an asr system, we conducted domain-agnostic and domain-relevant evaluations using wer and semantic distance. our analysis showed that while wer and semantic distance are in the 35 to 40% range when conducting a domain agnostic evaluation, the transcription error is slightly lower when considering transcription segments that are relevant to the domain i.e., utterances identified as important in evaluating the quality of counseling. moreover, we examined how the asr step fits in and impacts the larger pipeline of an nlp system for behavioral coding in psychotherapy by comparing how the use of asr data in place of manually transcribed data affects the performance of the downstream nlp system. finally, we empirically showed that augmenting the system input with local context may alleviate the impact of noisy transcription. given the results and analyses of this work, we conclude with the following lessons we learned in this study, on using asr for nlp applications in psychotherapy and counseling: (1) aggregate error measures are not sufficient by themselves, and must be complemented with domain-specific evaluations. (2) asr error rates and performances differ across speaker roles and demographics as well as utterance content/topics. (3) even a relatively small amount of manual transcription effort can help counteract noisy asr and improve performance during the training of nlp models for psychotherapy applications.