Improving information fusion on multimodal clinical data in classification settings (2022)

Abstract: Clinical data often exists in different forms across the lifetime of a patient’s interaction with the healthcare system - structured, unstructured or semi-structured data in the form of laboratory readings, clinical notes, diagnostic codes, imaging and audio data of various kinds, and other observational data. Formulating a representation model that aggregates information from these heterogeneous sources may allow us to jointly model on data with more predictive signal than noise and help inform our model with useful constraints learned from better data. Multimodal fusion approaches help produce representations combined from heterogeneous modalities, which can be used for clinical prediction tasks. Representations produced through different fusion techniques require different training strategies. We investigate the advantage of adding narrative clinical text to structured modalities to classification tasks in the clinical domain. We show that while there is a competitive advantage in combined representations of clinical data, the approach can be helped by training guidance customized to each modality. We show empirical results across binary/multiclass settings, single/multitask settings and unified/multimodal learning rate settings for early and late information fusion of clinical data.

## introduction a variety of clinical use cases emerge where it is not sufficient to use a single data modality as input to a learning or decision making system @xcite . a single data modality is often known to be insufficient for a clinical purpose. for instance, diagnoses that require imaging data as well as lab values or outcomes that depend on values routinely recorded in the narrative text but not elsewhere. an additional modality can be used to characterize additional features. for instance, information in narrative text that conflicts with or adds specificity to diagnosis or procedure codes or imaging data that can indicate severity of a condition not recorded in structured form or qualitatively mentioned in narrative text. sometimes, a modality with highly predictive or informative features is particularly expensive or invasive and an alternative source is present that may have features unintelligible or hard to parse for humans. also, most clinical machine learning systems focus on one clinical prediction task at a time @xcite . however, in real-world systems more than one such task are often performed simultaneously and are interrelated @xcite . there is a need to investigate task-specific unified representations of multimodal clinical data in both single-task and multi-task settings to improve decisions in the clinical workflow by demonstrating an increase in predictive power, robustness, and confidence over any single mode of data @xcite . besides creating and combining efficient representations of data from more than one modality, we also need to study the factors that affect the design and evaluation of these multimodal representations. ## multimodal representations there are various ways modalities of clinical data can be combined. multimodal deep learning models integrate information at various possible steps. this can occur in the following ways -• by finding a common representation for input data for a specific task before modeling. e.g., extracting clinical mentions from narrative text and concatenating it with independent diagnostic signals to form a model input. as detailed in @xcite , multimodal models can be categorized by the fusion techniques based on which they learn the join representations of underlying data. the most common approaches are called early fusion (chen and jin, 2015) where individual modality features are combined right after feature extraction and late fusion (atrey et al., 2010) which combines outputs from unimodal predictors jointly. early fusion is expected to capture some of the feature-level interactions of each modality and often is easier to model and train. on the other hand, late fusion allows for more flexibility and is expected to model individual modalities better and also handles scenarios where one or more modalities are missing. however, it cannot be expected to capture low level interaction between the modalities. while training late fusion models, the simplest choice is to use the same learning rate across all modalities. but it is both intuitive as well as demonstrable through layer analysis (yao and mihalcea, 2022) that learning rates for different modalities can differ a lot and must be handled separately to optimize learning from heterogeneous sources. 3 methodology 3.1 data source we use a publicly available clinical data set -medical information mart for intensive care (mimic) johnson et al. (2016) -containing data across various modalities for patients admitted and readmitted to the intensive care unit (icu). mimic-iii is a large, freely-available database comprising deidentified health-related data associated with over forty thousand patients who stayed in critical care units of the beth israel deaconess medical center between 2001 and 2012. the database includes information such as demographics, vital sign measurements made at the bedside ( 1 data point per hour), laboratory test results, procedures, medications, caregiver notes, imaging reports, and mortality, including post-hospital discharge. it contains highly granular data, including vital signs, laboratory results, and medications. ## results since the mortality prediction is a binary classification problem, we report the auc-pr numbers, which is a standard evaluation metric. because diseases can co-occur and a majority of patients often have more than one diagnosis, phenotyping is a multilabel classification problem, which requires the performance to be reported by averaging across labels or examples. these labels have varying base rates. in imbalanced tasks, @xcite show that if the predictive features for rare labels are lost, which is possible due to feature selection, macro f1 is an unsuitable metric. we report the macro auc-roc, which is the unweighted mean of auc-roc for each label. we also add a weighted average auc-roc metric accounting for base rate of the diseases in we also report error bounds for the experiments by choosing observations at the 2.5th percentile and the 97.5th percentiles and reporting the median. this was computed by drawing 5000 samples with replacement 100 times from the test set. ## related work we refer to @xcite that uses a single modality of time series in a multi-task setting using lstms and channel-wise lstms. similarly, @xcite presents a unimodal model with clinical notes only for individual task settings but they also report additional results in a multimodal setting using both time series and text data without using multitask learning. we reuse some of the multitasking configuration for the mimic dataset described in @xcite . there are also available comparisons against baseline logistic regression and random forest models in @xcite . all of these use only a unified learning rate across modalities. a number of works note the need of exploiting modality-specific features such as @xcite for combining text with other modalities such as image and audio. in the late-fusion setting, a closely related work is @xcite that investigates modality-specific learning rates. they do not investigate a multitask setting and also study modalities structurally different from ours. another closely related work is @xcite that take up the issue of potential overfitting to certain modalities. their approach is via early stopping and is still closer to our unified learning rate set up. it is also worth noting that the typical modalities in clinical data are very domain-specific and even well-studied modalities such as text in generaldomain nlp often behave differently in the clinical domain @xcite . ## limitations and future work this work investigates one way to adapt learning rates to modalities. there can be more adaptive strategies that take a priori clinical knowledge about a modality into account, which is a possible topic of future work. the late fusion methods discussed are also occasionally unstable during training. it is also conceivable that clinical text with different linguistic structure (e.g. short, more standardised radiology reports vs longer, less structured progress reports) behave differently when combined with other modalities. further investigation is required to mitigate these issues. the current work aims to show the effect of adding modalities and adapting parameters specific to useful modal