Evaluating and Improving Automatic Speech Recognition using Severity (2023)

Abstract: A common metric for evaluating Automatic Speech Recognition (ASR) is Word Error Rate (WER) which solely takes into account discrepancies at the word-level. Although useful, WER is not guaranteed to correlate well with human judgment or performance on downstream tasks that use ASR. Meaningful assessment of ASR mistakes becomes even more important in high-stake scenarios such as health-care. We propose 2 general measures to evaluate the severity of mistakes made by ASR systems, one based on sentiment analysis and another based on text embeddings. We evaluate these measures on simulated patient-doctor conversations using 5 ASR systems. Results show that these measures capture characteristics of ASR errors that WER does not. Furthermore, we train an ASR system incorporating severity and demonstrate the potential for using severity not only in the evaluation, but in the development of ASR. Advantages and limitations of this methodology are analyzed and discussed.

## introduction automatic speech recognition (asr) is the task of processing human speech into text, but no asr is perfect and certain types of errors can cause potential problems. asr has drastically improved over the past decade and has changed the way many people interact with computers in applications such as voice search, dictation, and virtual assistants @xcite . it is common practice to evaluate asr by calculating the word error rate (wer) which can be calculated by counting the number of words that need to be substituted (s), deleted (d), and inserted (i) to go from a ground-truth transcription to the output of an asr. this count is then divided the total number of words in the ground-truth transcription (n) similar to @xcite , often written as (s + i + d)/n . essentially, wer treats each discrepancy between the ground-truth transcription and the asr output equally. however, not all asr errors are equal. previous work studying asr errors has shown that wer is not always well correlated with human judgement or performance in a given downstream task, such as in information retrieval, natural language understanding or named entity recognition @xcite @xcite @xcite . for example, take the sentence "i love you," and suppose an asr system produces "i loathe you." this results in a wer of 0.33. now suppose another asr system predicts "i luv you." this too results in a wer of 0.33 (see example in figure 1 ). although the wers are equal, compared to the ground-truth "i love you," the mistake "luv" is less severe than the mistake "loathe," which gives off the opposite meaning from the ground-truth sentence. being able to understand the severity of asr errors becomes even more critical in high-stake scenarios such as healthcare where asr has been used since the 1970s @xcite . in healthcare research, transcriptions are used in a wide variety of tasks such as in the automated detection of dementia @xcite , in estimating scores of standardized cognitive health screening tests @xcite , and in in the prediction and explanation of diagnosis @xcite . however, these works operate on the basis of having an intelligible and accurate transcript. the purpose of this research is to develop a method for systematically measuring and understanding the quality of asr systems, especially in high-stake settings like healthcare, going beyond wer by looking at the difference in meaning between the ground-truth and asr output. in this work, we propose two methods for the automatic rating of the severity of errors in asr transcriptions by using 1) the difference in sentiment ratings and 2) cosine distances between text embeddings of the output of the asr and the groundtruth human transcription. we compare them with human-labeled severity scores. text embeddings prove to be correlate better with human labels of severity than wer. this work also shows sentiment ratings, text embeddings, and wer capture different aspects of mistakes in transcriptions and shows advantages and limitations of each method. lastly, we demonstrate the potential for severity to be used in the development of asr systems. we conclude by discussing limitations and future areas of research. ## related work to overcome the limitations of wer, other measures for asr evaluation have been proposed such the match error rate (mer), and word information lost (wil) @xcite . in the context of information retrieval, metrics based on named-entity word error rate correlates higher with retrieval performance than wer @xcite . however, these methods are still based on the literal word correctness and do not take into account semantics. @xcite incorporate semantics into a measure by including a weighted distance between word2vec @xcite vectors for misspelled words. they show that their measure correlates better with the perception of people who are deaf or hard of hearing and mention using sentiment analysis in future work. @xcite , similar to this work, take into account semantics by running the ground-truth reference and asr output through roberta and obtain an embedding for each by computing the mean of all the output vectors of the roberta model. the ground-truth and asr output are compared by using the cosine distance between their embeddings. @xcite uses similar methods and look at the poser (the part of speech error rate) and ember (embedding error rate), a wer that is weighted by the semantic similarity of incorrectly transcribed words. our work supports the results of these works in a healthcare context as well as explores the idea of using sentiment analysis. furthermore, our work is the first that we know of to include some approximation of the semantic meaning into the training regime of asr. ## methods when it comes to understanding and automatically rating the seriousness of errors in asr, one needs to have a method for systematically analyzing the difference in meaning between two phrases or sentences. while philosophically what a body of text truly means is a difficult question to answer, we can capture some essence of the meaning of an utterance using sentiment analysis and text embeddings. sentiment analysis is the task of detecting the attitude, emotions, or polarity of a given text. these algorithms usually take in a string as input and output a prediction from -1 to 1 based on how negative or positive the text is. because these algorithms vary, we use 3 different sentiment analyzers from 3 different widely-used nlp libraries nltk, flair, and textblob (tb) @xcite @xcite . this is a naive method of capturing the meaning of text because two texts can have different meanings and both have similar sentiment. although overly simplistic the purpose of using sentiment is to create and test a baseline measure that captures aspects of text besides discrepancies in spelling. another method for numerically capturing the meaning of natural language is to use sentencelevel embeddings (phrases or sentences are converted into n-dimensional vectors). there are a variety of methods for embedding words that range from simple rule-based methods to methods that involve machine learning @xcite @xcite . similarly, methods have been developed for embedding more than just single words @xcite . whether for individual words or phrases, with good embeddings, the more semantically similar words or phrases are, the closer they should be in the n-dimensional vector-space @xcite . we use 4 readily available pre-trained models provided by sentencetransformers 1 to com-pute text embeddings foot_1 : bert-base-nli-mean-tokens (bertnli), all-minilm-l6-v2 (minilm), allmpnet-base-v2 (mpnet), and all-distilroberta-v1 (disrob) @xcite @xcite . these models are selected due to their performance in semantic similarity tasks. these models are able to compute embeddings fast and are not too big, ranging from 80 mb to 420 mb. ## data for the purposes of experimenting in a healthcare scenario, a dataset of simulated patient-physician medical interviews is used @xcite . this dataset contains 272 audio files with transcripts (about 7 to 20 minutes or from 800 to 2200 words). the conversations are categorized into the following cases/subjects: respiratory, musculoskeletal, cardiac, dermatological, and gastrointestinal diseases. the majority of simulations were respiratory cases (78.7%). the files are split into non-silent intervals using librosa foot_2 setting the threshold of silence to 60 decibels. this results in over 39,600 non-silent intervals, which we will call utterances. of these, we take a sample of 110 utterences and run them through the asr systems and get the ground-truth transcription from the corresponding transcription files. these were run through the asr systems to create a list of 550 pairs of strings, one string being the ground-truth and the other coming from an asr. one of the audio files contained no speech and was removed for a final total of 545 pairs. the transcripts were normalized by removing speaker identification notes "p:" and "d:" for patients and doctors (found in the ground-truth), making all letters lowercase, and by removing any special characters and punctuation except for apostrophes (as these could be important in distinguishing words like "its" and "it's". ## exp. 2: severity in asr evaluation this experiment demonstrates the potential of using the severity measures in metrics for the overall evaluation and comparison of asr systems. to test this, we propose three metrics (section 6.1). we use the average wer and each metric to evaluate the performance of the 5 asr systems. these metrics are compared with the average wer and with each other across the asr systems. ## exp. 3: using severity to improve asr up to this point results show that 1) there is reliable consistency among human raters, 2) the cosine distance of text embeddings correlates better with human labels of severity than wer, and 3) using sentiment or text embeddings in a metric for the overall evaluation of asr captures different information than wer. with these results established, the purpose of this experiment is to test if an automatic measure of severity can be used in more than just the evaluation of asr, but in the training regime as well. previous work done in the study of asr errors involves approaches to automatically detect errors using word and text embeddings (and even other features such as acoustic/prosodic features), @xcite @xcite and to automatically repair errors in specific cases (such as in certain homophones in french) @xcite . however, instead of asr error detection or repair which happens post-prediction, our approach is to include severity into the training of an asr system in an attempt to reduce the number of errors (measured by wer) and to reduce the overall severity of the errors produced (measured by the average cosine distance from section 6.3). to do this, we incorporate severity into the loss function during training of an asr. ## conclusion automatic speech recognition (asr) is becoming an increasingly important tool from personal use to the medical field. however, word error rate (wer), a common metric for evaluation, only takes into account word discrepancies (i.e. figure 1 ). in this study we 1) compare wer and measures of severity based on sentiment and text embeddings to human labels of severity in a healthcare setting, 2) use these measures in metrics to evaluate the overall quality of mistakes in transcriptions, and 3) incorporate severity into the training of wer. results show that 1) cosine distance of text embedding correlates better with human ratings than wer, 2) these measures based on sentiment and text embeddings capture different qualities in asr errors and can overcome limitations of wer, and 3) incorporating severity into the training of an asr system increased performance, lowering the overall severity and wer significantly. in future work, we will experiment with different architectures, data, and methods for using semantics in the training of asr systems.