Hierarchical Pretraining on Multimodal Electronic Health Records (2023)

Abstract: Pretraining has proven to be a powerful technique in natural language processing (NLP), exhibiting remarkable success in various NLP downstream tasks. However, in the medical domain, existing pretrained models on electronic health records (EHR) fail to capture the hierarchical nature of EHR data, limiting their generalization capability across diverse downstream tasks using a single pretrained model. To tackle this challenge, this paper introduces a novel, general, and unified pretraining framework called MedHMP, specifically designed for hierarchically multimodal EHR data. The effectiveness of the proposed MedHMP is demonstrated through experimental results on eight downstream tasks spanning three levels. Comparisons against eighteen baselines further highlight the efficacy of our approach.

## introduction pretraining is a widely adopted technique in natural language processing (nlp). it entails training a model on a large dataset using unsupervised learning before fine-tuning it on a specific downstream task using a smaller labeled dataset. pretrained models like bert @xcite and gpt @xcite have demonstrated remarkable success across a range of nlp tasks, contributing to significant advancements in various nlp benchmarks. in the medical domain, with the increasing availability of electronic health records (ehr), researchers have attempted to pre-train domainspecific models to improve the performance of various predictive tasks further @xcite . for instance, clinicalbert @xcite and clinicalt5 @xcite are pretrained on clinical notes, and med2vec @xcite and mime @xcite on medical codes. these models pretrained on a single type of data are too specific, significantly limiting their transferability. although some pretraining models @xcite @xcite are proposed to use multimodal ehr data foot_0 , they ignore the heterogeneous and hierarchical characteristics of such data. the ehr data, as depicted in figure 1 , exhibit a hierarchical structure. at the patient level, the ehr systems record demographic information and capture multiple admissions/visits in a timeordered manner. each admission represents a specific hospitalization period and contains multiple stay records, international classification of diseases (icd) codes for billing, drug codes, and a corresponding clinical note. each stay record includes hourly clinical monitoring readings like heart rate, arterial blood pressure, and respiratory rate. in addition to the intricate hierarchy of ehr data, the prediction tasks vary across levels. as we move from the top to the bottom levels, the prediction tasks become more time-sensitive. patientlevel data are usually used to predict the risk of a patient suffering from potential diseases after six months or one year, i.e., the health risk prediction task. admission-level data are employed for relatively shorter-term predictions, such as readmission within 30 days. stay-level data are typically utilized for hourly predictions, such as forecasting acute respiratory failure (arf) within a few hours. designing an ideal "one-in-all" medical pretraining model that can effectively incorporate multimodal, heterogeneous, and hierarchical ehr data as inputs, while performing self-supervised learning across different levels, is a complex undertaking. this complexity arises due to the varying data types encountered at different levels. at the stay level, the data primarily consist of time-ordered numerical clinical variables. however, at the admission level, the data not only encompass sequential numerical features from stays but also include sets of discrete icd and drug codes, as well as unstructured clinical notes. as a result, it becomes challenging to devise appropriate pretraining tasks capable of effectively extracting knowledge from the intricate ehr data. in this paper, we present a novel hierarchical multimodal pretraining framework (called medhmp) to tackle the aforementioned challenges in the medical domain. medhmp simultaneously incorporates five modalities as inputs, including patient demographics, temporal clinical features for stays, icd codes, drug codes, and clinical notes. to effectively pretrain medhmp, we adopt a "bottom-to-up" approach and introduce level-specific self-supervised learning tasks. at the stay level, we propose reconstructing the numerical time-ordered clinical features. we devise two pretraining strategies for the admission level. the first focuses on modeling intra-modality relations by predicting a set of masked icd and drug codes. the second involves modeling inter-modality relations through modality-level contrastive learning. to train the complete medhmp model, we utilize a two-stage training strategy from stay to admission levels 3 . we utilize two publicly available medical datasets for pretraining the proposed medhmp and evaluate its performance on three levels of downstream tasks. these tasks include arf, shock and mortality predictions at the stay level, readmis-3 it is important to note that we have not incorporated patient-level pertaining in medhmp. this decision is based on the understanding that the relations among admissions in ehr data are not as strong as consecutive words in texts. arbitrary modeling of such relations may impede the learning of stay and admission levels. ## methodology as highlighted in section 1, ehr data exhibit considerable complexity and heterogeneity. to tackle this issue, we introduce medhmp as a solution that leverages pretraining strategies across multiple modalities and different levels within the ehr hierarchy to achieve unification. in the following sections, we present the design details of the proposed medhmp. ## experiments in this section, we first introduce the data for pretraining and downstream tasks and then exhibit experimental results (mean values of five runs). ## related work predictive modeling using ehr data has attracted significant attention in recent years @xcite @xcite . to enhance predictive performance, pretraining techniques have been explored. in this section, we provide a concise overview of studies conducted on pretraining with both single-modal and multimodal ehr data. (llms) @xcite in nlp, researchers have endeavored to train medical-specific language models using clinical notes @xcite @xcite and pubmed data @xcite @xcite @xcite . however, these models primarily rely on mask language modeling techniques for pretraining, thereby overlooking the distinctive characteristics of medical data. ## conclusion in this paper, we present a novel pretraining model called medhmp designed to address the hierarchical nature of multimodal electronic health record (ehr) data. our approach involves pretraining medhmp at two levels: the stay level and the admission level. at the stay level, medhmp uses a reconstruction loss applied to the clinical features as the objective. at the admission level, we propose two losses. the first loss aims to model intra-modality relations by predicting masked medical codes. the second loss focuses on capturing inter-modality relations through modality-level contrastive learning. through extensive multimodal evaluation on diverse downstream tasks at different levels, we demonstrate the significant effectiveness of medhmp. furthermore, experimental results on unimodal evaluation highlight its applicability in low-resource clinical settings and its ability to accelerate convergence.