Towards Understanding ASR Error Correction for Medical Conversations (2020)

Abstract: Domain Adaptation for Automatic Speech Recognition (ASR) error correction via machine translation is a useful technique for improving out-of-domain outputs of pre-trained ASR systems to obtain optimal results for specific in-domain tasks. We use this technique on our dataset of Doctor-Patient conversations using two off-the-shelf ASR systems: Google ASR (commercial) and the ASPIRE model (open-source). We train a Sequence-to-Sequence Machine Translation model and evaluate it on seven specific UMLS Semantic types, including Pharmacological Substance, Sign or Symptom, and Diagnostic Procedure to name a few. Lastly, we breakdown, analyze and discuss the 7% overall improvement in word error rate in view of each Semantic type.

## introduction off-the-shelf asr systems like google asr are becoming increasingly popular each day due to their ease of use, accessibility, scalability and most importantly, effectiveness. trained on large datasets spanning different domains, these services enable accurate speech-to-text capabilities to companies and academics who might not have the option of training and maintaining a sophisticated state-ofthe-art in-house asr system. however, for all the benefits these cloud-based systems provide, there is an evident need for improving their performance when used on in-domain data such as medical conversations. approaching asr error correction as a machine translation task has proven to be useful for domain adaptation and resulted in improvements in word error rate and bleu score when evaluated on google asr output @xcite . however, it is important to analyze and understand how domain adapted speech may vary from model transcript reference you also have a pacemaker because you had sick sinus syndrome and it's under control google asr you also have a taste maker because you had sick sinus syndrome and it's under control s2s you also have a pacemaker because you had sick sinus syndrome and it's under control reference like a heart disease uh atrial fibrillation google asr like a heart disease asian populations s2s like a heart disease atrial fibrillation asr outputs. we approach this problem by using two different types of metrics -1) overall transcription quality, and 2) domain specific medical information. for the first one, we use standard speech metric like word error rate for two different asr system outputs, namely, google cloud speech api foot_0 (commercial), and aspire model (open-source) @xcite . for the second type of evaluation, we use the umls foot_1 ontology @xcite and analyze the s2s model output for a subset of semantic types in the ontology using a variety of performance metrics to build an understanding of effect of the sequence to sequence transformation. ## related work while the need for asr correction has become more and more prevalent in recent years with the successes of large-scale asr systems, machine translation and domain adaptation for error correction are still relatively unexplored. in this paper, we build upon the work done by mani et al. @xcite . however, d'haro and banchs (d'haro and banchs, 2016) first explored the use of machine translation to improve automatic transcription and they applied it to robot commands dataset and human-human recordings of tourism queries dataset. asr error correction has also been performed based on ontology-based learning in @xcite . they investigate the use of including accent of speaker and environmental conditions on the output of pre-trained asr systems. their proposed approach centers around bioinspired artificial development for asr error correction. @xcite explore the use of noisy-clean phrase context modeling to improve asr errors. they try to correct unrecoverable errors due to system pruning from acoustic, language and pronunciation models to restore longer contexts by modeling asr as a phrase-based noisy transformation channel. domain adaptation with off-the-shelf asr has been tried for pure speech recognition tasks in high and low resource scenarios with various training strategies @xcite @xcite @xcite but the goal of these models was to build better asr systems that are robust to domain change. domain adaptation for asr transcription can help improve the performance of domain-specific downstream tasks such as medication regimen extraction (selvaraj and konam, 2019). ## domain adaptation for error correction using the reference texts and pre-trained asr hypothesis, we have access to parallel data that is in-domain (reference text) and out-of-domain (hypothesis from asr), both of which are transcriptions of the same speech signal. with this parallel data, we now frame the adaptation task as a translation problem. sequence-to-sequence models : sequence-tosequence (s2s) models @xcite have been applied to various sequence learning tasks including speech recognition and machine translation. attention mechanism @xcite is used to align the input with the output sequences in these models. the encoder is a deep stacked long short-term memory network and the decoder is a shallower uni-directional gated recurrent unit acting as a language model for decoding the input sequence into either the transcription (asr) or the translation (mt). attention-based s2s models do not require alignment information between the source and target data, hence useful for monotonic and non-monotonic sequence-mapping tasks. in our work, we are mapping asr output to reference hence it is a monotonic mapping task where we use this model. ## conclusion we present an analysis of how asr error correction using machine translation impacts the different semantic types of the umls ontology for a medical conversation. we run the s2s model on a dataset of doctor-patient conversations as a post-processing step to optimize the google off-theshelf asr system. we use different input representations and compare the performance of our s2s model using wer and bleu scores on google asr and aspire outputs. we deep dive into how our adaptation model affect medical wer for each semantic type, and breakdown the results using precision, recall, f1 and semantic intersection numbers between s2s and google asr. we establish the robustness of s2s model performance for more frequently occurring medical terms. in the future, we want to explore other representations like phonemes which might capture asr errors better, and address the class imabalance problem for rarer medical terms in different semantic types.