Multiple Sclerosis Severity Classification From Clinical Text (2020)

Abstract: Multiple Sclerosis (MS) is a chronic, inflammatory and degenerative neurological disease, which is monitored by a specialist using the Expanded Disability Status Scale (EDSS) and recorded in unstructured text in the form of a neurology consult note. An EDSS measurement contains an overall ‘EDSS’ score and several functional subscores. Typically, expert knowledge is required to interpret consult notes and generate these scores. Previous approaches used limited context length Word2Vec embeddings and keyword searches to predict scores given a consult note, but often failed when scores were not explicitly stated. In this work, we present MS-BERT, the first publicly available transformer model trained on real clinical data other than MIMIC. Next, we present MSBC, a classifier that applies MS-BERT to generate embeddings and predict EDSS and functional subscores. Lastly, we explore combining MSBC with other models through the use of Snorkel to generate scores for unlabelled consult notes. MSBC achieves state-of-the-art performance on all metrics and prediction tasks and outperforms the models generated from the Snorkel ensemble. We improve Macro-F1 by 0.12 (to 0.88) for predicting EDSS and on average by 0.29 (to 0.63) for predicting functional subscores over previous Word2Vec CNN and rule-based approaches.

## introduction recent advancements of deep learning models with electronic health records (ehr) have shown a great deal of success in many clinical applications @xcite , such as disease detection @xcite , diagnostics @xcite , risk predictions @xcite and patient subtyping @xcite . however, when the data within the ehr is presented in the form of narrative, unstructured clinical notes, extensive work is required by a profes-sional to diagnose and generate labels for a patient @xcite . the development of pre-trained language models, namely bidirectional encoder representations from transformers (bert), have significantly improved natural language processing (nlp) tasks within the general language domain @xcite . however, in specialized domains such as the clinical one, the vocabulary, syntax and semantics differ significantly from general language @xcite and thus pretraining a language model on domain-specific texts is critical to improving performance. this is supported by the observed increase in performance on domain-specific nlp tasks when pretraining a bert model on domainspecific texts @xcite @xcite . take for example bluebert @xcite , which has been further pretrained on over 4 billion words from pubmed abstracts and 500 million words from mimic-iii @xcite and has been shown to outperform bert on multilabel classification from the hallmarks of cancers corpus @xcite . domain-specific language models, such as blue-bert, still face several challenges for clinical nlp tasks. first, clinical texts must be de-identified of sensitive information, with the replacement of key tokens reducing the model's ability to interpret the text @xcite . second, texts from a specific clinical application may contain unique sub-language that the model was not trained on, hindering the model's performance. third, transformer models have a fixed context length of 512 tokens that is significantly shorter than the average length of clinical texts @xcite . as a result of truncating the text to fit the context length, the model is unable to analyze the entire text and may miss important information. these are the challenges of applying existing bert models to specific clinical nlp tasks, which we have addressed through our contributions applied to a multiple sclerosis (ms) dataset. our contributions are as follows: ## methods de-identification of clinical text. the consult notes used in this study contained sensitive information such as patient's name, phone numbers, physician's name and address. we de-identified the data using a curated database of patient and doctor information and regular expression matching. we replaced identifying pieces of information with specific tokens that met the following criteria: (1) the token was within the current bert vocabulary, (2) the token had a similar semantic meaning to the word it replaced, and (3) the token was not found in the original data set. for example, all last names were replaced with "salamanca". in doing so, we aimed to limit the loss of contextual information that results from de-identification. we also overcame challenges with sub-optimal placeholder replacements often present in clinical datasets, like mimic-iii @xcite . as an example, mimic-iii may replace a patient's last name with "[**last name placeholder**]", which is tokenized by bert into at least 7 tokens (one for each square bracket, one for each star and at least one for the place holder within the brackets). a list of our de-identification replacements can be found within the appendices (contribution [3]). ## experiments multiple sclerosis (ms) is one of the most common non-traumatic disabling neurological condition among young adults worldwide @xcite . onset of ms typically occurs between the ages of 20 to 40 years, with women more often affected than men @xcite . ms is a disease that impacts the central nervous system (cns) @xcite , leading to the degradation of myelin sheathing and axons within the nervous system. this degradation is highly varied and unpredictable in both location and intensity within the body. resulting symptoms include but are not limited to: visual impairment, loss of balance, numbness, bladder dysfunction and fatigue @xcite . ms is typically monitored by the expanded disability status scale (edss) @xcite . edss is used to evaluate the degree of cns impairment on a scale from 0 to 10. edss also includes eight functional subscores @xcite such as an ambulation score and a visual score. a full list of functional subscores is found within edss and functional subscores are discussed in a patient's consult note, dictated by a physician and manually transcribed. edss is determined by a combination of functional subscores and is typically stated within consult notes. however, functional subscores are not typically stated within a consult note and need to be derived from contextual information about the patient's health. traditionally, both edss and functional subscores are manually derived by an expert within the field and logged into the patient's health record. minute differences in patient descriptions can correspond to different edss and functional subscore values. through consultation with ms healthcare professionals we expect the qualitative descriptions of ms symptoms contained within the clinical notes to remain uniform across healthcare systems. ## concluding thoughts in this work we present methods to overcome the challenges that arise when applying a modern transformer model on a specific clinical nlp task, specifically ms severity prediction. we did this through: (1) de-identifying clinical texts in a we believe that the ms-bert language model and its improved ability to understand ms consult notes will aid clinicians in the diagnosis and treatment of ms. furthermore, we believe that being trained on more clinical text, ms-bert has the potential to improve other nlp tasks within the clinical domain.