Paper: Multimodal Patient Representation Learning with Missing Modalities and Labels (ICLR-2024)

Reviewer comment (Experiments):
More baselines should have been added for a more comprehensive comparison, e.g., MMIN (Zhao et al., Missing modality imagination network for emotion recognition with uncertain missing modalities.) and Robust Multimodal Transformer (Ma et al., Are multimodal transformers robust to missing modality?).

Author response:
**A1.** In response to the reviewer's suggestion, we have conducted additional experiments on new datasets and baselines.

> New datasets.

We have conducted additional experiments on datasets from the MULTIBENCH [1] benchmark encompassing various areas such as Robotics, Finance, HCI, and Multimedia. Due to the time constraint, we selected one dataset from each area and evaluated our proposed MUSE and the two best-performing baselines (GRAPE and M3Care). To simulate the missing modalities and labels scenario, we randomly masked out labels with a probability of 0.5 and modalities with a probability of 0.1. In summary, our results indicate that MUSE performs exceptionally well on a variety of tasks and domains, demonstrating its generalizability.

*Table: Additional experiment on the MULTIBENCH benchmark. MUSE achieves the best performance on 3/4 tasks, and MUSE+ further outperforms all baselines by incorporating samples with missing labels into the training process.* 

| Dataset | MUJOCO PUSH (Robotics) | STOCKS-F&B (Finance) | ENRICO (HCI) | MM-IMDB (Multimedia)  |
|----------|----------|----------|--------|------------|
| **Metrics** | **MSE $\downarrow$**  | **MSE $\downarrow$**    | **Acc $\uparrow$**  | **Micro F1 $\uparrow$**  |
| GRAPE  | 8.0035  | 2.1165  | 0.3698 | 0.3851   |
| M3Care  | 8.8192  | **2.0734**  | 0.3815 | 0.3882   |
| MUSE (Ours)  | **7.5314**  | 2.0852  | **0.3887** | **0.3915**   |
| MUSE+ (Ours)  | **5.6311**  | **2.0631**  | **0.4152** | **0.4232**   |

> New baselines.

We have extended our comparison by including three additional baselines: MedFuse [2], MMIN [3], and MMCL [4]. Notably, MMCL also incorporates samples with missing labels during training, making it a fair comparison with our enhanced method, MUSE+. The table below summarizes the performance comparison, with MUSE and MUSE+ consistently outperforming all baselines.

*Table: Comparison between MUSE and three additional baselines in terms of AUC-ROC scores. MUSE and MUSE+ still outperform all baselines.*

|      | MIMIC-IV Mortality   $\uparrow$   | MIMIC-IV Readmission  $\uparrow$ | eICU Mortality   $\uparrow$   | eICU  Readmission  $\uparrow$ | ADNI  $\uparrow$  |
|------------|--------------------|--------------|-------------------|--------------|--------|
| MedFuse  | 0.8733       | 0.6876    | 0.8813      | 0.7514    | 0.8916 |
| MMIN    | 0.8712       | 0.6900     | 0.8756      | 0.7508    | 0.8844 |
| MUSE  (Ours)  | **0.9004**       | **0.7152**    | **0.9017**      | **0.7709**    | **0.9158** |
| MMCL    | 0.9134       | 0.7102    | 0.9192      | 0.7855    | 0.9218 |
| MUSE+  (Ours)  | **0.9201**       | **0.7351**    | **0.9332**      | **0.8003**    | **0.9309** |

Reviewer comment (Experiments):
If we regard labels as a subset of features, the whole architecture can be perceived as a pretraining framework combining two contrastive learning loss terms, along with a classification loss serving downstream tasks. I wonder if the authors have conducted experiments to pre-train MUSE with two contrastive learning terms and then fine-tune it using the classification loss term. The two-stage pattern might help the model to take a further step towards better performance by separating pretraining and fine-tuning.

Author response:
**A2.** We appreciate the thoughtful suggestion from the reviewer. Initially, we employed a two-stage training approach and subsequently merged them into an end-to-end training pipeline to improve efficiency. Empirically, we observed that both training schemes yield comparable performance, but the end-to-end approach significantly reduces training time. Regarding the idea of treating labels as a subset of features, we find it intriguing and believe it presents an interesting avenue for future research. We will include this idea in the future work section of the updated manuscript.

Reviewer comment (Experiments):
A weakness of the paper is that some of the architecture, experimental design, and parameter choices are not explained and not justified. For example, there are many ways to compute similarity (such as the normalized dot product used in the well-established InfoNCE loss), so a discussion on the choice of cosine as the similarity function is needed.

Author response:
**A1.** We would like to note that we did not extensively tune the hyperparameters for the experiments. Specifically, we only tune the learning rate for both MUSE and the baseline methods while keeping the other hyperparameters aligned. We leave extensive hyperparameters tuning for future works. As for design choice, we largely adhered to the settings presented in some of the classic papers, including the use of cosine similarity [1], a 15% edge drop rate [2], and the implementation of Siamese GraphSage [3]. It is important to note that we try to align the experimental settings between MUSE and other baselines as much as possible to ensure a fair comparison.

Reviewer comment (Experiments):
While it is helpful for reproducibility to list the hyperparameters used in the model, this alone is not enough as their choice is not justified. Searching through the main paper as well as the appendix, there is no mention of hyperparameter tuning. Hyperparameter tuning is important both for the baselines and for the proposed MUSE to establish the fairness of comparison, as the wrong choice of parameter could lead to a baseline performing worse than it could. This is especially true as the baseline architectures are vastly different (ranging from auto-encoders to Transformers).

Author response:
**A1.** We would like to note that we did not extensively tune the hyperparameters for the experiments. Specifically, we only tune the learning rate for both MUSE and the baseline methods while keeping the other hyperparameters aligned. We leave extensive hyperparameters tuning for future works. As for design choice, we largely adhered to the settings presented in some of the classic papers, including the use of cosine similarity [1], a 15% edge drop rate [2], and the implementation of Siamese GraphSage [3]. It is important to note that we try to align the experimental settings between MUSE and other baselines as much as possible to ensure a fair comparison.

Reviewer comment (Experiments):
Other unexplained choices include using 15% as the edge dropout rate, the use of Siamese GNN over other GNN methods, doing the missing ratio experiment only on the mortality prediction task for MIMIC, and choosing 30% randomly masked modalities in the analysis of the learned representations.

Author response:
**A1.** We would like to note that we did not extensively tune the hyperparameters for the experiments. Specifically, we only tune the learning rate for both MUSE and the baseline methods while keeping the other hyperparameters aligned. We leave extensive hyperparameters tuning for future works. As for design choice, we largely adhered to the settings presented in some of the classic papers, including the use of cosine similarity [1], a 15% edge drop rate [2], and the implementation of Siamese GraphSage [3]. It is important to note that we try to align the experimental settings between MUSE and other baselines as much as possible to ensure a fair comparison.

Reviewer comment (Experiments):
The paper mentions the use of bootstrapping 1000 times, however, it is unclear if the reported performance is an average over the random weight initializations.

Author response:
**A3.** We followed the evaluation protocol established in M3Care, which entails evaluating a trained model 1000 times by bootstrapping (i.e., sampling with replacement) from the test set.

Reviewer comment (Evaluation):
While the dataset statistics are mentioned in the Appendix, it is important to point out the class label imbalance present in the datasets in the main paper. This contextualizes the performance of the models; for example, MUSE+ performs with ~92% AUC-ROC on MIMIC mortality prediction, but only 9% of the data has a positive mortality prediction. Meaning, a model that does not truly learn the task, can simply guess the majority label and get 91% accuracy on the task. This may not be the case here, but this information is important to better understand performance.

Author response:
**A4.** We understand the reviewer's concern regarding label imbalance. It is important to note the difference between accuracy and AUC-ROC scores. In datasets with only 1% positive labels, a classifier predicting all zeros would achieve a 99% accuracy score but only a 0.5 AUC-ROC score. Notably, AUC-ROC and PR-AUC scores are commonly used in mortality and readmission tasks on the MIMIC and eICU datasets [8, 9, 10].

Reviewer comment (Writing):
In the Missing Modalities and Labels paragraph, it is unclear what the text means by “fail to fully use inter-patient relationships?”

Author response:
**A5.** We appreciate the reviewer's feedback and recognize the need for clarification. Modality imputation approaches usually try to generate the missing modalities based on the available modalities from the same patient. However, we believe that the missing modalities could also be inferred from similar patients, while modality imputation approaches do not explicitly model the relationship between different patients.

Reviewer comment (Presentation):
It is stated in the Training and Inference Section that for MUSE, the authors only feed the original graph through a GNN, but this contradicts the model diagram, which shows both augmented and original graphs going through a GNN. Can this be explained or clarified in the text?

Author response:
**A7.** We apologize for the confusion. The model diagram illustrates the training process, where both the augmented and original graphs pass through the GNN. During inference, the augmentation branch is omitted. We will revise our manuscript to make this distinction explicit.

Reviewer comment (Reproducibility):
The implementation details are unclear. “We train all models for 100 epochs on the training set, and select the best model by monitoring the performance on the validation set.” What is meant by “all models”? Does this include the baselines or only differently initialized MUSE models? Are the authors using the validation performance to determine all hyperparameters, and what specific metric is used? Is there early stopping for epochs, or are models trained for the full 100 epochs?

Author response:
**A8.** To clarify, we trained both MUSE and the baseline methods for 100 epochs. We saved model weights at the end of each epoch and selected the best-performing model by monitoring its performance on the validation set. The specific metric monitored varies depending on the task: we used the PR-AUC score for mortality and readmission tasks and the one-vs-one macro AUC-ROC score for the Alzheimer's disease prediction task.

Reviewer comment (Reproducibility):
The paper cites the Transformer paper (Vaswani et al.) as the backbone encoder for several tabular modalities. The original Transformer paper cited is meant for text and not necessarily tabular data, so were there further adjustments made for this encoder?

Author response:
**A9.** We use the Transformer encoder to encode the medical codes (e.g., diagnosis, procedures, and medications). However, they are sequence data instead of tabular. Each patient visit usually has a sequence of medical codes recorded at different timestamps. We will make this clear in the updated manuscript.

Reviewer comment (Evaluation):
While ablation studies were provided that showed an array of dropout, did authors find a rate of missingness that ultimately resulted in the model ignoring certain features? (e.g. really rare lab examinations)

Author response:
**A2.** Yes. In fact, we observed that in the MIMIC-IV dataset, naive multimodal models (such as direct concatenation) frequently relied heavily on the clinical text while largely disregarding other modalities. We have referred to this issue as "Modality Collapse" in the introduction section of our manuscript. This observation was a critical motivator for developing the mutual-consistent graph contrastive learning framework. This framework is designed to facilitate the learning of features that are both modality-agnostic and decisive for accurate predictions.

Reviewer comment (Evaluation):
For the experiment analyzing the learned representation (Section 4.5) are you using the average cosine similarity across all patients? Could you also provide average Euclidean distance between representations since GRAPE minimizes MSE loss for continuous values?

Author response:
**A3.** In our experiment analyzing the learned representations (Section 4.5), we calculate the average cosine similarity across all patients. We appreciate the suggestion to include the average Euclidean distance between representations. Here is the table with the requested information:

*Table: Euclidean distance of the representations from the same patients with different modalities.*

| Method      | Euclidean Distance  $\downarrow$ |
|-------------------|--------------------|
| CM-AE       | 0.7352       |
| SMIL       | 0.6337       |
| MT       | 0.5839       |
| GRAPE       | 0.3001       |
| HGMF       | 0.4426       |
| M3Care      | 0.3268       |
| MUSE (Ours)    | **0.2437**       |
| MUSE+  (Ours)   | **0.2232**       |

Reviewer comment (Experiments):
The datasets studied in the paper are fairly limited from a modality perspective in the classical sense (e.g. MIMIC-IV has time series and text, and eICU has only time series). Though the authors split the time-series into multiple modalities (e.g. labs, vitals, diagnoses), it is unclear what a modality has to encompass in order for the method to work well. In the extreme case, could each individual measurement have been its own modality?

Author response:
**Q1. What should a modality encompass for multimodal models to work well?**

**A1.** Generally speaking, we believe that this is still an open research question. I.e., when and why does multimodal outperform unimodal jointly? However, our paper primarily focuses on addressing the challenge of missing modalities and labels rather than delving into the theoretical aspects of modality selection. We designed our experiments based on real-world clinical practices, where various modalities play a role in clinical decision-making. We also empirically verified that multimodal models outperform unimodal methods in our setting during the pilot study. Understanding when and why multimodal models will work is out of the scope of this paper. Instead, we focus on the setting where multimodal models can outperform unimodal models, but the missing modalities and labels limit their improvements. We believe this question should not undermine the practical significance of the problem or the novelty of the proposed method.

Reviewer comment (Experiments):
The method proposed in the paper tackles a healthcare problem, but could definitely be applied to datasets beyond healthcare. The authors should consider benchmarking their method on datasets from [1], potentially by masking out labels and modalities.

Author response:
**Q2. Additional experiments on datasets beyond healthcare.**

**A2.** In response to the reviewer's suggestion, we have conducted additional experiments on datasets from the MULTIBENCH [1] benchmark encompassing various areas such as Robotics, Finance, HCI, and Multimedia. Due to the time constraint, we selected one dataset from each area and evaluated our proposed MUSE and the two best-performing baselines (GRAPE and M3Care). To simulate the missing modalities and labels scenario, we randomly masked out labels with a probability of 0.5 and modalities with a probability of 0.1.

*Table: Additional experiment on the MULTIBENCH benchmark. MUSE achieves the best performance on 3/4 tasks, and MUSE+ further outperforms all baselines by incorporating samples with missing labels into the training process.* 

| Dataset | MUJOCO PUSH (Robotics) | STOCKS-F&B (Finance) | ENRICO (HCI) | MM-IMDB (Multimedia)  |
|----------|----------|----------|--------|------------|
| **Metrics** | **MSE $\downarrow$**  | **MSE $\downarrow$**    | **Acc $\uparrow$**  | **Micro F1 $\uparrow$**  |
| GRAPE  | 8.0035  | 2.1165  | 0.3698 | 0.3851   |
| M3Care  | 8.8192  | **2.0734**  | 0.3815 | 0.3882   |
| MUSE (Ours)  | **7.5314**  | 2.0852  | **0.3887** | **0.3915**   |
| MUSE+ (Ours)  | **5.6311**  | **2.0631**  | **0.4152** | **0.4232**   |

In summary, our results indicate that MUSE performs exceptionally well on a variety of tasks and domains, demonstrating its generalizability.

Reviewer comment (Reproducibility):
It was not clear to me reading through the paper exactly how the method handles completely new patients at test-time. Are they added as new patient nodes and edges in the bipartite graph, and the GNN is run in inference with all of the pre-training patient nodes still in the graph?

Author response:
**Q3. How does the method handle entirely new patients during testing?**

**A3.** In both the training and inference phases, we create patient graphs independently within each batch. We will clarify this point in the revised manuscript.