Robust Prediction of Punctuation and Truecasing for Medical ASR (2020)

Abstract: Automatic speech recognition (ASR) systems in the medical domain that focus on transcribing clinical dictations and doctor-patient conversations often pose many challenges due to the complexity of the domain. ASR output typically undergoes automatic punctuation to enable users to speak naturally, without having to vocalize awkward and explicit punctuation commands, such as “period”, “add comma” or “exclamation point”, while truecasing enhances user readability and improves the performance of downstream NLP tasks. This paper proposes a conditional joint modeling framework for prediction of punctuation and truecasing using pretrained masked language models such as BERT, BioBERT and RoBERTa. We also present techniques for domain and task specific adaptation by fine-tuning masked language models with medical domain data. Finally, we improve the robustness of the model against common errors made in ASR by performing data augmentation. Experiments performed on dictation and conversational style corpora show that our proposed model achieves 5% absolute improvement on ground truth text and 10% improvement on ASR outputs over baseline models under F1 metric.

## introduction medical asr systems automatically transcribe medical speech found in a variety of use cases like physician-dictated notes @xcite , telemedicine and even doctor-patient conversations @xcite , without any human intervention. these systems ease the burden of long hours of administrative work and also promote better engagement with patients. however, the generated asr outputs are typically devoid of punctuation and truecasing thereby making it difficult to comprehend. furthermore, their recovery improves the accuracy of subsequent natural language understanding algorithms @xcite to identify information such as patient diagnosis, treatments, dosages, symptoms and signs. typically, clinicians explicitly dictate the punctuation commands like "period", "add comma" etc., and a postprocessing component takes care of punctuation restoration. this process is usually error-prone as the clinicians may struggle with appropriate punctuation insertion during dictation. moreover, doctor-patient conversations lack explicit vocalization of punctuation marks motivating the need for automatic prediction of punctuation and truecasing. in this work, we aim to solve the problem of automatic punctuation and truecasing restoration to medical asr system text outputs. most recent approaches to punctuation and truecasing restoration problem rely on deep learning @xcite . although it is a well explored problem in the literature, most of these improvements do not directly translate to great real world performance in all settings. for example, unlike general text, it is a harder problem to solve when applied to the medical domain for various reasons and we illustrate each of them: we overcome the data scarcity problem, by using pretrained masked language models like bert @xcite and its successors @xcite which have successfully been shown to produce stateof-the-art results when finetuned for several downstream tasks like question answering and language inference. we approach the prediction task as a sequence labeling problem and jointly learn punctuation and truecasing. we show that finetuning a pretrained model with a very small medical dataset (∼500k words) has ∼5% absolute performance improvement in terms of f1 compared to a model trained from scratch. we further boost the performance by first finetuning the masked language model to the medical speech domain and then to the downstream task. the contributions of this work are: the rest of this paper is organized as follows. section 2 presents related work on punctuation and truecasing restoration. section 3 introduces the model architecture used in this paper and describes various techniques for improving accuracy and robustness. the experimental evaluation and results are discussed in section 4 and finally, section 5 presents the conclusions. ## related work several researchers have proposed a number of methodologies such as the use of probabilistic machine learning models, neural network models, and the acoustic fusion approaches for punctuation prediction. we review related work in these areas below. ## modeling : conditional joint labeling of punctuation + casing we propose a postprocessing framework for conditional and joint learning of punctuation and truecasing prediction. consider an input utterance x 1:@xmath0, x 2 , ..., x t }, of length t and consisting of words x i . the first step in our modeling process involves punctuation prediction as a sequence tagging task. once the model predicts a probability distribution over punctuation, this along with the input utterance is fed in as input for predicting the case of a word x i . we consider the punctuation to be independent of casing and a conditional dependence of the truecase of a word on punctuation given the learned input representations. our plausible reasoning follows from this example sentence -"she took dance classes. she had no natural grace or sense of rhythm.". the word after the period is capitalized, which implies that punctuation information can help in better prediction of casing. a pair of punctuation and truecasing is assigned per word: ## conclusion in this paper, we have presented a framework for conditional joint modeling of punctuation and truecasing in medical transcriptions using pretrained language models such as bert. we also demonstrated the benefit from mlm objective finetuning of the pretrained model with task specific masking. we further improved the robustness of punctuation and truecasing on asr outputs by data augmentation during training. experiments performed on both dictation and conversation corpora show the effectiveness of the proposed approach. future work includes the use of either pretrained acoustic features or pretrained acoustic encoder to perform fusion with pretrained linguistic encoder to further boost the performance of punctuation.