BERT - XML : Large Scale Automated ICD Coding Using BERT Pretraining (2020)

Abstract: ICD coding is the task of classifying and cod-ing all diagnoses, symptoms and proceduresassociated with a patient’s visit. The process isoften manual, extremely time-consuming andexpensive for hospitals as clinical interactionsare usually recorded in free text medical notes. In this paper, we propose a machine learningmodel, BERT-XML, for large scale automatedICD coding of EHR notes, utilizing recentlydeveloped unsupervised pretraining that haveachieved state of the art performance on a va-riety of NLP tasks. We train a BERT modelfrom scratch on EHR notes, learning with vo-cabulary better suited for EHR tasks and thusoutperform off-the-shelf models. We furtheradapt the BERT architecture for ICD codingwith multi-label attention. We demonstratethe effectiveness of BERT-based models on thelarge scale ICD code classification task usingmillions of EHR notes to predict thousands ofunique codes.

## introduction information embedded in electronic health records (ehr) have been a focus of the healthcare community in recent years. research aiming to provide more accurate diagnose, reduce patients' risk, as well as improve clinical operation efficiency have well-exploited structured ehr data, which includes demographics, disease diagnosis, procedures, medications and lab records. however, a number of studies show that information on patient health status primarily resides in the free-text clinical notes, and it is challenging to convert clinical notes fully and accurately to structured data @xcite @xcite . extensive prior efforts have been made on extracting and utilizing information from unstructured ehr data via traditional linguistics based methods in combination with medical metathesaurus and semantic networks @xcite @xcite . with rapid developments in deep learning methods and their applications in natural language processing (nlp), recent studies adopt those models to process ehr notes for supervised tasks such as disease diagnose and/or icd foot_0 coding @xcite @xcite @xcite . yet to the best of our knowledge, applications of recently developed and vastly-successful selfsupervised learning models in this domain have remained limited to very small cohorts @xcite and/or using other sources such as pubmed publication @xcite or animal experiment notes @xcite instead of clinical data sets. in addition, many of these studies use the original bert models as released in @xcite , with a vocabulary derived from a corpus of language not specific to ehr. in this work we propose bert-xml as an effective approach to diagnose patients and extract relevant disease documentation from the free-text clinical notes with little pre-processing. bert (bidirectional encoder representations from transformers) @xcite utilizes unsupervised pretraining procedures to produce meaningful representation of the input sequence, and provides state of the art results across many important nlp tasks. bert-xml combines bert pretraining with multi-label attention @xcite , and outperforms other baselines without self-supervised pretraining by a large margin. ad-ditionally, the attention layer provides a natural mechanism to identify part of the text that impacts final prediction. compare to other works on disease identification, we demonstrate the effectiveness of bert-based models on automated icd-coding on a large cohort of ehr clinical notes, and emphasize the following aspects: 1) large cohort pretraining and ehr specific vocabulary. we train bert model from scratch on over 5 million ehr notes and with a vocabulary specific to ehr, and show that it outperforms off-the-shelf or fine-tuned bert using offthe-shelf vocabulary. 2) minimal pre-processing of input sequence. instead of splitting input text into sentences @xcite @xcite or extracting diagnose related phrases prior to modeling (azam et al., 2019), we directly model input sequence up to 1,024 tokens in both pre-training and prediction tasks to accommodate common ehr note size. this shows superior performance by considering information over longer span of text. 3) large number of classes. we use the 2,292 most frequent icd-10 codes from our modeling cohort as the disease targets, and shows the model is highly predictive of the majority of classes. this extends previous effort on disease diagnose or coding that only predict a small number of classes. 4) novel multi-label embedding initialization. we apply an innovative initialization method as described in section 3.3.2, that greatly improves training stability of the multi-label attention. the paper is organized as follows: we summarize related works in section 2. in section 3 we define the problem and describe the bert-based models and several baseline models. section 4 provides experiment data and model implementation details. we also show the performances of different model and examples of visualization. the last section concludes this work and discusses future research areas. many works further incorporates the attention mechanisms as introduced in @xcite , to better utilize information buried in longer input sequence. in @xcite , the authors introduce a hierarchical attention bidirectional gated recurrent unit(ha-gru) architecture. @xcite use a hierarchical combination of lstms to encode ehr text and then use attention with encodings of the text description of icd codes to make predictions. while these models have impressive results, some fall short in modeling the complexity of ehr data in terms of the number of icd codes predicted. for example, @xcite limit their predictions to the 50 most frequent codes and @xcite predict 32. in addition, these works do not utilize any pretraining and performance can be limited by size of labeled training samples. ## conclusion automatic icd coding from medical notes has high value to clinicians, healthcare providers as well as researchers. not only does auto-coding have high potential in cost-and time-saving, but more accurate and consistent icd coding is necessary to facilitate patient care and improve all downstream healthcare ehr based research. we demonstrate the effectiveness of models leveraging the most recent developments in nlp with bert as well as multi-label attention on icd classification. our model achieves state of the art results using a large set of real world ehr data across many icd classes. in addition we find that domain specific pretrained bert model outperforms bert models trained on general purpose corpora. we note that the off-the-shelf wordpiece tokenizer can naively split domain-specific yet oov words and resulting in a bert model focusing on word completion, while using a specific ehr vocabulary seem to help overcome the problem. lastly, we also observe the benefit of modeling longer sequences. on the other hand, the current work has several limitations. most importantly, while we have found that modeling longer term dependencies improves performance, it comes at a large cost of training time. doubling the input length roughly triples the training and inference time. for many applications this increase in computational demand may offset the gain in model performance. this motivates further exploration on efficient variants of the self-attention modules to accommodate longer input length in similar tasks. additionally, adding xml to the bert architecture generates significant yet rather marginal performance improvement (micro-auc improvement of 0.002 for ehr bert big model with maximum input length of 1024). this also increases the computation complexity