How to leverage the multimodal EHR data for better medical prediction? (2021)

Abstract: Healthcare is becoming a more and more important research topic recently. With the growing data in the healthcare domain, it offers a great opportunity for deep learning to improve the quality of service and reduce costs. However, the complexity of electronic health records (EHR) data is a challenge for the application of deep learning. Specifically, the data produced in the hospital admissions are monitored by the EHR system, which includes structured data like daily body temperature and unstructured data like free text and laboratory measurements. Although there are some preprocessing frameworks proposed for specific EHR data, the clinical notes that contain significant clinical value are beyond the realm of their consideration. Besides, whether these different data from various views are all beneficial to the medical tasks and how to best utilize these data remain unclear. Therefore, in this paper, we first extract the accompanying clinical notes from EHR and propose a method to integrate these data, we also comprehensively study the different models and the data leverage methods for better medical task prediction performance. The results on two prediction tasks show that our fused model with different data outperforms the state-of-the-art method without clinical notes, which illustrates the importance of our fusion method and the clinical note features.

## introduction under the serious struggle of the covid-19, the healthcare research domain has attracted more and more attention nowadays. with the improvement of information technology, many hospitals have begun to use ehr (electronic health record) systems to monitor all the data produced during the entire hospital admission. the large amount of data generated in this process offers an opportunity for * corresponding author. deep learning technology to improve healthcare, such as diagnoses prediction @xcite , medication recommendation @xcite , mortality prediction @xcite , and readmission prediction @xcite . however, comparing to common academic datasets, such as imagenet @xcite and wmt @xcite , real-world ehr data is longitudinal, heterogeneous, and multimodal, which proposes big challenges to leverage the information included in it. to further understand the complexity of realworld ehr data, we depict the data in figure 1 . the data can be split into three modalities: (1) time-invariant data, such as age, sex of a patient. usually, it will not change during the hospital admission. (2) time-series data, such as vital signs and laboratory measurements. these data share the same property that they change over time and the distribution is nonuniform in time. for instance, vital signs like heart rate and blood pressure are recorded continuously for hours or days, while laboratory measurements such as blood test is a discrete event happened in some time during the admission. (3) clinical notes, are unstructured free text and normally sparser than the time-series data. more importantly, these notes are full of abbreviations, jargon, and unusual grammatical structures, which are hard for non-professionals to read and understand. given the uneven distribution of time-series data, researchers are mostly focused on this data. for example, both @xcite and @xcite extract the raw time-series data into usable hourly features, performing essential operations such as unit conversion, missingness reduction, and outlier handling. however, the clinical notes that contain significant clinical value @xcite are beyond the realm of these preprocessing pipelines' consideration. due to the importance of clinical notes, it is necessary to combine them with other data sources for the integrity of clinical features. since the common pre-trained language models such as bert @xcite do not consider the specific complexity of clinical notes, we apply the clinicalbert @xcite that are pre-trained on clinical notes for handling the notes data in this paper. though the above modalities have strong potential in deep learning, modeling the joint representation is nontrivial, as naively adding more features could result in worse performances @xcite . in this work, with the inspiration from multimodal adaptation gate (mag) @xcite , we use attention gate for the fusion of the aforementioned three modalities. the core idea behind is to adjust the representation of one modality with a displacement vector derived from the other modalities. for simplicity, we refer to the modality being adjusted as the main modality and the other modalities as auxiliary modalities. the question is, which modality should be the main modality? the answer may be clear in multimodal sentiment analysis since language/text always contains the richest sentiment information. however, for medical prediction tasks, such as acute respiratory failure (arf), we doubt that text/notes should be the main modality since it happens beyond the physician's expectation. therefore, we conduct experiments on two tasks, diagnoses prediction and acute respiratory failure (arf) prediction, and we give comprehensive explorations of the fusion strategy. the results show that our model with clinical notes outperforms the models without them, which illustrates the importance of the clinical notes and the effectiveness of our fusion method. the contributions of this paper can be summarized as follows: we conduct our work based on the data extracted from mimic-iii (johnson et al., 2016), a realworld ehr database comprising information relating to patients admitted to intensive care unit (icu). recently, there has been surge of methods which apply deep learning on these tabular domain data. yoon et al. (2020) propose a self-and semisupervised learning frameworks for value imputation and data augmentation in tabular domain data. wang et al. (2021) propose an improved deep & cross network (dcn) to learn explicit feature interactions. given the complexity of ehr data, wang et al. (2020) propose a pipeline to transform the raw mimic-iii data into usable hourly time-series data. to break through the limitation of a specific dataset, tang et al. (2020) propose a systematic preprocessing technique named fiddle for ehr data. we use the data extracted by fiddle in this paper. for single modality tasks, medical codes are commonly extracted and input to rnn for diagnoses prediction @xcite or medication prediction @xcite . recently, since the thrive of multimodal machine learning, researchers have begun to leverage the multimodal nature of ehr data to improve prediction performance @xcite . @xcite improve diagnoses prediction by combining medical codes and clinical notes through a multimodal attentional neural network. @xcite propose a recurrent attentive model to fuse continuous patient monitoring data, such as electrocardiogram (ecg), and discrete clinical events for predicting length of icu stay. comparing to the handcrafted models, @xcite propose a neural architecture search (nas) method to simultaneously search across multimodal fusion strategies and modalityspecific architectures for diagnoses prediction. our model builds upon clinicalbert @xcite , which has the same architecture with bert @xcite . similarly, clin-icalbert is pre-trained on the clinical notes of mimic-iii @xcite with two unsupervised tasks, masked language modeling (mlm) and next sentence prediction (nsp). the method of integrating multimodal information into large pre-trained transformers like bert has also been explored in @xcite . only that (rahman et al., 2020) apply attention gate at word-level to combine a lexical input vector with its visual and audio accompaniments for sentiment prediction. differently, we first pass the features to subnetworks and then fuse the outputs using the attention gate for diagnoses and arf prediction. ## our method our method consists of three logical parts: encoding, fusion, and prediction. in this section, we will describe these parts in detail. for clarity, we first define the data notations used in our method here. the data are split into two categories according to the method used to extract it. the data extracted from preprocessing pipeline framework such as @xcite is split into time-invariant data and time-series data. given the batch size by b, the time-invariant data can be represented as i @xmath0, where d 1 is the dimension of timeinvariant feature. similarly, the time-series data is denoted as i @xmath1, where l represents the length of the icu stay counted by hours and d 2 is the dimension of the time-series data. for clinical notes, wordpiece @xcite is applied to tokenize and transform them into token ids. given the length of the notes by d 3 , the token ids are represented as i @xmath2. ## experiments in this section, we first introduce the mimic-iii dataset. then we describe the tasks and the metrics used to evaluate the performance. after that, we introduce the baseline models adopted in this paper and the detailed experimental design of this work. ## results and discussion in this section, we first evaluate the performance of all the fused models on the arf and diagnoses tasks, and discuss the effect of the main modality on each of them. then we perform an ablation study on the best model for understanding the influence of individual modules in our method. finally, we perform experiments by using other fusion strategies to study the effect. ## conclusion in this paper, we propose to integrate the data extracted from the preprocessing pipeline and the accompanying clinical notes for better medical prediction. enlightened by the mag method, we propose a fusion method for our tasks and explore four different encoders to study the effect. besides, to understand the importance of each modality, we switch the main modality in the fusion models and find that time-series data and clinical notes are the main modalities of the arf task and diagnoses task respectively. finally, we investigate other fusion strategies and the results show that our fusion method achieves state-of-the-art performance. in the future, we will investigate the tabular data related methods stated in section 2 to see if they can improve the performance.