Clinical BERTS core: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings (2023)

Abstract: Automatic Speech Recognition (ASR) in medical contexts has the potential to save time, cut costs, increase report accuracy, and reduce physician burnout. However, the healthcare industry has been slower to adopt this technology, in part due to the importance of avoiding medically-relevant transcription mistakes. In this work, we present the Clinical BERTScore (CBERTScore), an ASR metric that penalizes clinically-relevant mistakes more than others. We collect a benchmark of 18 clinician preferences on 149 realistic medical sentences called the Clinician Transcript Preference benchmark (CTP) and make it publicly available for the community to further develop clinically-aware ASR metrics. To our knowledge, this is the first public dataset of its kind. We demonstrate that our metric more closely aligns with clinician preferences on medical sentences as compared to other metrics (WER, BLUE, METEOR, etc), sometimes by wide margins.

## introduction clinicians in a number of disciplines work in an overburdened healthcare system that leads to difficult working environments and an epidemic of physician burnout @xcite . ai-related technologies have the potential for improving efficiency on repetitive tasks, therefore increasing both patient throughput and decreasing physician burnout. for example, physicians in a number of disciplines spend as much time doing paperwork as with patients @xcite . however, the adoption of speech technology in the medical community has been slow @xcite , and there are a number of speech technologies that could improve efficiency. speech technology can be applied to a number of medical problems including transcribing patientphysician conversations @xcite , help-ing dysarthric patients communicate @xcite , and diagnosing medical conditions from speech @xcite @xcite . in this work, we focus on the task of generating a report after a colonoscopy procedure. one of many reasons for the lower adoption of time-saving speech transcription technologies is that the asr systems often don't perform as well in real-world clinical settings as they do on evaluation benchmarks. the most common metric for measuring asr performance, word error rate (wer), has significant practical drawbacks @xcite @xcite . first, all mistakes are treated equally. in clinical settings, however, medical words are more important (e.g. "had complete resection" → "had complete c-section" is a worse mistake @xmath0, but both have equal wer). second, some mistakes affect the overall intelligibility more than others (e.g. "was no perforation" → "was no puffer age" vs "was not any perforation"). although researchers have proposed alternatives to the wer, no metric combines medical domain knowledge with recent ai advances in language understanding. in this work, we make the following contributions: ## related work there are a number of ways to evaluate transcript quality. the word error rate (wer), is the simplest to compute and most common. it counts the number of insertions, deletions, and substitutions between two text strings, and normalizes by the length of the reference string. the bilingual evaluation understudy (bleu) @xcite measures the amount of n-gram overlap between two text strings (where n is often 4). it captures the intuition that groups of words are important in addition to individual words. meteor @xcite focuses on unigrams, but computes an explicit alignment between two strings and takes both precision and recall into consideration. while these techniques are cheap to compute, they primarily focus on character or string similarity, not semantic similarity. our work most closely follows the bertscore @xcite . this metric computes a neural word embedding for each word in the reference and candidate. embeddings are matched using cosine distance instead of string similarity, and the final score takes precision and recall into account (see fig. 1 ). this method takes semantic similarity into account, but not that some words are more important to preserve in clinical contexts. structured graphs are one way to encode realworld knowledge in a machine-readable format. the knowledge graph (kg) @xcite is a publicly available structure that encodes medical knowledge. previous work has used the medical subset of the kg to learn medical entity extraction @xcite . we primarily follow this approach to determine which words are clinically significant. ## conclusions we present cbertscore, a novel metric that combines medical domain knowledge and recent advances in neural word embeddings. we collect and release a benchmark of clinician rater preferences on transcript errors, demonstrate that cbertscore is more closely aligned with clinician preferences, and release the benchmark for the research community to continue to improve asr in medical contexts.