Clinical XLN et: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation (2020)

Abstract: Clinical notes contain rich information, which is relatively unexploited in predictive modeling compared to structured data. In this work, we developed a new clinical text representation Clinical XLNet that leverages the temporal information of the sequence of the notes. We evaluated our models on prolonged mechanical ventilation prediction problem and our experiments demonstrated that Clinical XLNet outperforms the best baselines consistently. The models and scripts are made publicly available.

## introduction unstructured clinical notes within electronic health records (ehr) contain valuable information to support clinical decisions @xcite . however, most prognostic models used in medical practice currently rely on scoring systems that only incorporate structured data @xcite @xcite . a major challenge to utilize unstructured clinical data is in representing notes in ways that allow effective mining of clinically meaningful knowledge. there are many recent advances in the standard natural language processing domain, such as bert @xcite , xlnet @xcite . however, clinical notes are far different from the general domain text @xcite . for example, clinical notes contain jargon and abbreviations, different grammar and syntax. it is notoriously difficult to obtain an effective note representation. recently, clinicalbert, which adapts the bert model from the standard nlp domain to model clinical notes @xcite achieved superior performance in clinical text prediction. however, previous works still have the following limitations: in the standard nlp domain, bert ignores the discrepancy of masked positions between the pretraining and finetuning stage. an autoregressive pretraining method named xl-net was recently developed and empirically outperforms bert on many nlp tasks @xcite . in this paper, we present clinical xlnet, which processes a patient's notes and predicts clinical outcomes. in particular, this model mitigates the aforementioned limitations via the following technical contributions: we apply the permutation language modeling method proposed in xlnet on a corpus of clinical notes to generate better clinical embeddings, as demonstrated in section. 4. we examined clinical xlnet's performance on a new but important clinical nlp task: predicting prolonged mechanical ventilation (pmv). mechanical ventilation is to use an artificial breathing machine to assist or replace patient's breath. pmv stands for longer than normal period of mechanical ventilation. pmv consumes substantial healthcare resources, results in financial and emotional burdens for patients and their families, and is associated with high one-year mortality around 50-60% @xcite @xcite . it is projected that over 600, 000 patients in the united states will require pmv by 2020 @xcite . patients with pmv can have a surgical procedure called tracheostomy to establish better airway access for long-term mechanical ventilation @xcite . tracheostomy is associated with improved patient comfort, decreased duration of icu and hospital stay, and reduced mortality @xcite . however, the decision to place a tracheostomy is challenging for two main reasons: (1) tracheostomy may not be necessary if a patient's condition improves without requiring pmv, and (2) tracheostomy may not be helpful if the patient is at high risk of short-term mortality. thus, an early and correct decision of tracheostomy is critical and depends on the likelihood of pmv and short-term mortality. accurately predicting these factors using patients' clinical notes could support clinical decision making. we compared clinical xlnet with several stateof-the-art baselines including bert @xcite , xlnet @xcite , and clinical-bert @xcite on both pmv and mortality predictions. we performed meticulous cohort curation in mimic-iii dataset @xcite to set up an actionable prediction task ac-cording to the real clinical setting. clinical notes used in this prediction task were strictly within the 48-hour time window starting from the initial mechanical ventilation event. experimental results showed that clinical xlnet outperformed the best baselines consistently (section. 4). ## data we use the multiparameter intelligent monitoring in intensive care iii (mimic-iii) dataset @xcite hosted on physionet @xcite for our model development and experiment. it consists of 61,532 icu stays out of 58,976 hospital admissions from 46,476 patients in the intensive care unit of the beth israel deaconess medical center (bidmc) between 2001 and 2012 and it has 2,083,180 clinical note events. cohort selection. comprehensive inclusion and exclusion criteria were applied to the mimic-iii dataset to generate our patient cohort who were above 18 years old, and were on mechanical ventilation for at least 2 days with more than 6 hours each day. we excluded patients who were organ donors or transferred patients from other hospitals. as certain diseases always lead to pmv, to alleviate confounding, we further removed patients with neuromuscular disease, head and neck cancer, and extensive burns @xcite . for each hospital admission, we used the first icu stay. for clinical notes, we included nursing and respiratory notes within 48 hours from the start of the first ventilation event. the reason for only selecting nursing-related notes was to expand the cohort as mimic-iii is missing physician notes from 2001 to 2008. additional criteria that are applied in our data curation process are in fig. 2 in the appendix. in the end, we obtained a cohort of 7,287 unique patients and their corresponding 73,224 clinical notes. table. 1 shows the cohort demographics. cohort labels. our cohort was labeled with pmv and 90-day mortality as a binary outcome. pmv was defined as being on mechanical ventilation for more than 7 days with at least 6 hours each day (boles jm, 2007). short-term mortality was defined as death occurring within 90 days of the first icu admission. we use 90 days in contrast to 30 days because recent studies showed that 30 days mortality may underestimate the evaluation @xcite . ## methods this section presents our clinical xlnet framework (fig. 1 ). clinical xlnet is an extension of xlnet @xcite on the clinical text domain. it first generates a deep latent representation for clinical notes and then by applying a bidirectional lstm (bi-lstm) layer, it also leverages the sequential order of notes. problem settings. our target task aims to leverage a patient's clinical notes to predict variables such as pmv and mortality. we denote a patient as p, and each patient p is associated with an ordered sequence of notes {n 1 , • • • , n i }, where i is the total number of notes. to predict mortality l m and pmv l p , we aim to learn two mappings m : pretraining clinical xlnet. the xlnet is pretrained on common language corpora such as book-corpus, wikipedia, common crawl and etc. however, these corpora are different from clinical notes which are filled with jargon, abbreviations and difficult syntax and grammar. hence, to learn an effective representation of clinical notes, we further pre-trained the xlnet using nursing, nursing/others, and respiratory therapy notes available in the mimic-iii dataset. the clinical notes used in pre-training were not in the holdout test set to avoid biased results. xlnet is a stack of transformer-xl encoder @xcite . for pre-training, it uses permutation language modeling (plm) to tackle the challenge of @xcite token information gap between pre-training and finetuning in bert @xcite . for each sequence, it is appended with a [cls] classification token at the beginning of the sequence for downstream task usage. for a more detailed description, we refer the readers to the original paper @xcite . a patient is associated with many notes, and each notes length varies. since xlnet can only take 512 maximum tokens, we follow @xcite to first concatenate all the notes and partition them into snippets n i . then, we use the last encoder layer hidden representation e i of the @xcite token to represent the note snippet. as we train with a supervised signal in the downstream task, [cls] token would gather useful information in the entire note sequence due to the transformer-xl's self-attention mechanism. now, given a temporally ordered sequence of note snippets associated with a patient, we obtain a temporally-ordered sequence of notes representations finetuning clinical xlnet. to leverage the temporal information among the note snippets, we feed {e 1 , • • • , e i } into a sequential modeling layer. specifically, we use bi-lstm model @xcite . we use bidirectional model because not only the latter notes depend on the previous notes as patients develop their symptoms in a temporal order but also the latter notes may contain useful clinical knowledge to help enrich the representation of previous notes. the output of the bi-lstm layer h n is then fed into a predictor neural network, which at last, generates a probability p that measures the likelihood of downstream target variable, pmv, and mortality. the network is then tuned using binary classification loss. pre-finetuning. as each patient is associated with many notes snippets and each snippet corresponds to a large model, it is computationally infeasible to train the model end-to-end. in order to alleviate the memory cost of end-to-end training for every clinical note, we propose to approximate the task-specific note representation through an additional pre-finetuning stage. we assume that any part of the notes associated with a patient is correlated to the label. thus, during pre-finetuning, we use one piece of note as input, and further train the pre-trained clinical xlnet through the downstream task label signal from the corresponding patient. the pre-finetuned network can then generate the ordered representation sequence is then fed into a bidirectional lstm layer, which then outputs a fixed size latent vector h n , representing the entire sequence. h n is finally fed into a predictor neural network to generate a probability p measuring the likelihood of the target variable. a task specific note representation. then, during the finetuning stage, we freeze the note representation module, and use a static fixed note representation from the pre-finetuned clinical xlnet to feed into the bi-lstm layer. ## experiments to evaluate our model, we examined the prediction performance under a realistic setup. we used the first 48 hours of clinical notes starting from the initial mechanical ventilation event to predict two variables: mechanical ventilation longer than 7 days and 90-days mortality. foot_0 hyperparameters. for the data split, we first obtain a 10% holdout test set. then we generate different 8:1 train:validation splits using different random seeds for model performance robustness examination. for the pre-training, we further pretrain the xlnet embedding for another 200k steps using 16 batch size. for the pre-finetuning, we use 32 batch size with learning rate 1e-5 for four epochs with early stopping on the area under the receiver operating characteristic curve (auroc) score of validation. for finetuning, we used a two layers bi-lstm module with batch size 128 and learning rate 1e-4. the pre-training and pre-finetuning probaselines. we conducted a thorough set of experiments with several popular baselines: note that for bert, xlnet, and clinicalbert, we all attach a bidirectional-lstm layer on top of them to leverage the sequential dimension of notes. and for clinicalbert, we pre-train using the same corpus as the clinical xlnet. these steps ensure a fair comparison between note representation power. note that it is computationally infeasible to test the ablation of pre-finetuning. ## discussion in this work, we propose a method for predicting prognosis based on only contextual information available from clinician notes. the proposed method is based on the recent advancements in the field of nlp. therefore we compare it with other recently proposed methods in nlp for a fair comparison. we perform one ablation study to show the relevance of the proposed sequential modeling of the notes embedding in the time domain and we compare against several state of the art baselines which have been used extensively in the natural language domain as well as in the clinical context. clinical relevance. our work provides timely aid in clinical decision making. for a patient under the icu observation, the clinical team could start the evaluation to consider a tracheotomy procedure as soon as 48 hours after mechanical ventilation. the time period of 48 hours was chosen in consultation with a team of clinicians. furthermore, a predictive analytics on the prolonged mechanical ventilation for seven days or more is important for clinicians in deciding the tracheotomy decision. besides, the doctors could reduce the risk of a burdensome procedure and treatment by considering the patient's 90-days mortality prediction. this approach assist patients and their families by providing more time to process and make a major decision. model efficiency. the proposed method uses xlnet @xcite which uses trans-formerxl @xcite as the base architecture to extract embedding from the notes. since every set of notes require their individual embeddings, we run the base architecture for multiple runs where the number of runs is proportional to the number of notes. therefore, obtaining the embedding for the whole sequence of notes is computationally expensive both during the training as well as inference. however, there is a recent line of work @xcite which can allow executing the transformer based models at a much lesser computational cost. limitations and future work. our proposed method only mines task relevant information from clinical notes from nurses and respiratory therapists. however, one can utilize other sources of data as well such as structured notes. while structured data are commonly used in prognostic models, our preliminary study showed that they did not improve the performance by any significant factor. one future direction could be to explore a novel architecture design that could utilize both sources of information to improve the performance further. another future direction would be to explore ways of combining multiple sources of clinical notes such as physician notes, admission notes, and dsischarge notes.