Revisiting Clinical Outcome Prediction for MIMIC - IV (2024)

Abstract: Clinical Decision Support Systems assist medical professionals in providing optimal care for patients.A prominent data source used for creating tasks for such systems is the Medical Information Mart for Intensive Care (MIMIC).MIMIC contains electronic health records (EHR) gathered in a tertiary hospital in the United States.The majority of past work is based on the third version of MIMIC, although the fourth is the most recent version.This new version, not only introduces more data into MIMIC, but also increases the variety of patients.While MIMIC-III is limited to intensive care units, MIMIC-IV also offers EHRs from the emergency department.In this work, we investigate how to adapt previous work to update clinical outcome prediction for MIMIC-IV.We revisit several established tasks, including prediction of diagnoses, procedures, length-of-stay, and also introduce a novel task: patient routing prediction.Furthermore, we quantitatively and qualitatively evaluate all tasks on several bio-medical transformer encoder models.Finally, we provide narratives for future research directions in the clinical outcome prediction domain. We make our source code publicly available to reproduce our experiments, data, and tasks.

## introduction estimating the future clinical state of a patient upon admission to a medical care facility is a task of critical importance. clinicians must be able to promptly gauge not only the main affliction of patients, but also all the resources needed to streamline their care. a clinical decision support system (cdss) aids clinicians in a multifaceted way; for instance, they can interact with a clinician in a conversational manner or they can assist in the diagnosis process by offering discrete suggestions. generative medical assistants, like amie @xcite , enable clinicians to derive diagnostics and treatments by engaging in a conversation with the language model. one way of communicating these findings is to use the international classification of diseases (icd) taxonomy which is also used by medical practitioners to document the admission of a patient, their stay, and release from a medical care facility. while conversational cdss can provide reasonable answers and may identify important treatment strategies, their suggestions veer substantially from expert suggestions @xcite . furthermore, validating these suggestions is difficult, given the arbitrarily large output space of decoder-based transformer architectures such as amie. however, it is essential for clinicians to validate the predictions of such systems in order to safeguard the wellbeing of their patients. given the discrete space of the icd taxonomy and the necessity of validation, we argue that classification with encoder models is relevant for the clinical outcome prediction domain. ## cop-iv tasks we revisit the task creation process of @xcite and update it for the mimic-iv data. ## experiments we fine-tune all models in all outcome prediction tasks on both mimic-iv splits, except for the los and pr tasks. these tasks are exclusive to the icu and ed split as mentioned in section 3. we report performance in auroc-macro as well as in pr-auc. in contrast to van aken et al. (2021), we include pr-auc as an additional metric. while the auroc provides insight into performance for the majority of the patients, the pr-auc provides a more balanced view, since it emphasizes the performance of labels that are less frequent in the data. for comparability, we evaluate all cop-iv tasks with the encoder models used in @xcite , namely biobert, core, clinical-bert, and dischargebert. additionally, we extend this evaluation to pubmedbert. we conduct a hyper-parameter-optimization (hpo) on pub-medbert for all tasks for the learning rate and warmup steps using ray @xcite and @xcite . we use the resulting hyperparameters in all experiments. we use early stopping on auroc with a patience of 5 epochs as in van @xcite . we keep a consistent batch size of 50 for all tasks and models. for every experiment, we use a single a100 40gb gpu. ## results we present all experimental results in overall performance. pubmedbert outperforms all models across all tasks. biobert is the second best performing model, followed by core. clincalbert and dischargebert are the worst performing models. ## conclusion in this work, we introduce cop-iv, a clinical outcome prediction set of tasks based on mimic-iv, which updates cop-iii. in addition, we introduce the novel task of patient routing at admission time to clinical outcome prediction. we evaluate qualitatively this task for various patient demographics, as well as hospital care units. we explain in detail our preprocessing approach to reproduce the cop-iv tasks. furthermore, we present a comprehensive evaluation of several bio-medical encoder models and discuss their weaknesses, as well as challenges such as the pronounced class imbalance. moreover, we give relevant insights into data distribution shifts between cop-iii and cop-iv. lastly, we propose future research directions for clinical outcome prediction. we release our source code to reproduce the data for our benchmark, experiments, and results.