Paper: Towards Dynamic EHR Phenotyping: A Generative Clustering Model (ICLR-2024)

Reviewer comment (Novelty):
The distinction of this work from AC-TPC and T-Phenotype is not clearly articulated in Section 2. These methods are all considered "dynamic temporal phenotyping" approaches, capable of incorporating any outcome of interest at each time step, implying that these method can also consider future (e.g., step-ahead) observations and/or the clinical outcome available at the end of the sequence as their “outcomes” at each time step.

Author response:
**[Distinction with AC-TPC and T-Phenotypes]**:     Apologies for the lack of clarity in clarifying how DeepGC differs from AC-TPC [1] and T-Phenotypes [2]. 

AC-TPC, a deep learning approach for identifying phenotypes across two dimensions, achieves this through an iterative process involving cluster assignment and refinement. These subprocesses utilize an outcome prediction task, enabling weight backpropagation through the entire network. Unlike DeepGC, AC-TPC is limited in its capability to a) handle data generation, b) model both observation and outcome data, and c) was specifically designed for a binary outcome task, making it less adept at handling multiclass outcome, obtaining particularly poor performance when classes are highly imbalanced [3]. Additionally, AC-TPC's dynamic phenotyping relies on adverse event occurrences during the observation temporal period (I.e, it updates its phenotypes when new data are being observed). The dynamic modeling approach employed by DeepGC effectively captures the evolving phenotype assignments of a patient over time, a feature that AC-TPC struggles to achieve. T-Phenotype, builds upon AC-TPC with Laplace Encoders, seeks to enhance the trade-off between strong performance on the outcome and observation aspects. However, it shares largely similar limitations to those with AC-TPC.

Reviewer comment (Evaluation):
The impact of utilizing both future observations and clinical outcomes is not convincingly motivated nor adequately supported by the experimental results.

Author response:
**[Usefulness of leveraging Future Data]**:  We disagree with the reviewer with respect to the usefulness of future observations and clinical outcomes. The premise of our work is that there are naturally occurring phenotypes in medical cohorts based on various aspects, such as observation and outcome. Incorporating outcomes, therefore, can be invaluable in improving subgroup identification and phenotyping. This can be seen regarding clustering performance. Except for TSKM, we outperform all other benchmarks with respect to clustering observation data, including purely unsupervised models. 

In the medical setting, having a generative model would serve the benefit of allowing us to generate trajectories of physiological variables that change over time to demonstrate and explain the behavior of our model prediction. For example, if an individual is predicted to have a cardiac arrest in the next two days, we would like to show the clinician the deterioration of his/her EHR information that changes over time. Having the ability to generate such trajectories would be helpful in model interpretation and explanation. 

We have empirically demonstrated that our model is able to utilize the generated information to improve outcome prediction (see Table 1 where DeepGC outperformed CAMELOT in almost all metrics on all datasets). Furthermore, it is very important to have the ability to provide information about potential patient outcomes as it is the ultimate function that clinicians look for in a decision-support tool. We will motivate the rational of future observation and outcome prediction in our revised manuscript.

Reviewer comment (Evaluation):
The description of performance metrics should assist readers in comparing clustering performance across different methods. Additionally, it is unclear how similarity (distance) is measured to compute the Silhouette Index, Davies-Bouldin Index, and Variance Ratio Criterion.

Author response:
**[Clustering Metrics]**: The evaluation metrics were calculated via the scikit-learn [4] package in Python 3. To account for the temporal 3D dimensionality of the input data, we represented each unique feature-time pair of values as a unique feature when evaluating clustering performance.

Reviewer comment (Experiments):
Experimental results on benchmarks and baselines appear incomplete. There is a notable absence of results on AC-TPC (as mentioned in Section 4.2) and T-Phenotype, both of which can incorporate either future observations or final outcomes of interest with a simple modification. Furthermore, results on the Normalized Mutual Information (NMI) are lacking. The discriminative power of the proposed method should be compared with the same network architecture without incorporating clustering, rather than relying solely on simple ML baselines, which may not be suitable for handling time-series data.

Author response:
**[Benchmarking]**: We did not include experiments for AC-TPC due to the work in [5] which showed CAMELOT outperformed AC-TPC in a slightly different task. We will include the experiments with AC-TPC and T-Phenotype in the final edited submission. We note that the top performance obtained by these two benchmarks was lower to DeepGC on both datasets at least 5% (average AUROC), 4% (average F1), 2% (average Recall), 2% (average Precision).

We also agree with the reviewer that it would be important to compare our baselines with similar models that do not leverage clustering, to ensure that its advantages are properly taken care of, and to that end we illustrate in the manuscript the performance obtained from using the VRNN model.

Reviewer comment (Evaluation):
Regarding Weakness #3: What similarity (distance) metric is used to compute SIL, DBI, and VRI? Are the ground truth future observations taken into account?

Author response:
**[Similarity Distance for Metric Computation]**: The SIL, VRI, and DBI metrics are all computed in the input space with the assumption of an Euclidean distance. Furthermore, we did not include “future observations” when computing the clustering performance. Both choices were completed in order to ensure we could compare with all benchmarks in a fair manner.

Reviewer comment (Evaluation):
The discriminative power is relatively small, and it is not clearly stated how well the evaluated methods and the proposed method handle imbalanced labels. The authors should also provide performance at each class level and AUPRC, which would be more appropriate for assessing methods under imbalance.

Author response:
**[Imbalanced Data]**: Due to the imbalanced nature of the dataset, we were mainly focused on optimising for the average F1 score on the outcome aspect. The average F1 score is, by definition, an average over all outcomes, while also being more sensitive about the data imbalance for each outcome. We considered using AUPRC, but decided against usage of this metric for 2 reasons: a) unlike binary AUROC, its values are not between [0.5, 1], with binary AUPRC being bounded by the number of outcome examples across the whole cohort, for each outcome, and b) unlike AUROC, AUPRC does not have a natural associated probabilistic interpretation.

Reviewer comment (Experiments):
How was the number of clusters (K) determined? This is crucial for gauging the discriminative power of the discovered clusters.

Author response:
**[Selection of K]**: The number of clusters was determined via Occam's Razor [6] approach (same as CAMELOT), as described in the last paragraph of Section 4. We increased the number of clusters until we no longer observed a statistically significant increase in the average F1 performance of DeepGC.

Reviewer comment (Presentation):
The figure is slightly pixilated, please use svg or pdf format. Additionally I found it confusing. The green solid line has no arrow, whereas the other lines do. Furthermore, the generation of x' is not depicted.

Author response:
**[Typos and Editing]**: Figures will be made clearer on an eventual submission, as well as better care with boldening vector objects, and editing the document for errors in writing.

Reviewer comment (Writing):
Paper writing needs some edits. E.g. spaces after commas, X and $X$ both being used to refer to datasets, etc.

Author response:
**[Typos and Editing]**: Figures will be made clearer on an eventual submission, as well as better care with boldening vector objects, and editing the document for errors in writing.

Reviewer comment (Experiments):
No ablations performed

Author response:
**[Ablation]**: We believe that any type of ablation studies on DeepGC would simmer down to benchmarking our model against benchmarks already considered. Specifically, the benchmark VRNN and its variant VRNN-GMM, both subjects of comparison, can be seen as simplified versions of DeepGC. To elaborate further, VRNN-GMM is 2-stage process (i.e., the latent space is fixed and learnt from VRNN separately, then clustering is achieved via GMM), where ours is end-to-end joint learning with the combination of outcome modelling and phenotyping. Additionally, pure VRNN lacks the phenotyping component. Our argument posits that it is the synergistic combination of outcome prediction, phenotyping, and sequential modelling that enhances the models' ability to learn improved phenotypes, sensitive to both outcome and observation aspects.

Reviewer comment (Experiments):
Is it possible to to run results from general tabular data generators, like TabDDPM (https://arxiv.org/abs/2209.15421) or PAR Synthsizer (https://docs.sdv.dev/sdv/sequential-data/modeling/parsynthesizer)?

Author response:
**[Generative Benchmarks]**: We thank the reviewer for the suggestion to use TabDDPM and PAR Synthesizer as comparisons to evaluate the quality of our generative component. We will use these comparisons for further analysis, however, we would like to stress that generation is not our main goal. DeepGC was designed to tackle the task of phenotyping across two different aspects. The addition of a generative component results from the fact that observations closer in time to an event were more separable with respect to outcome than earlier observations. Combined with the fact that observations are seen only earlier to a certain window of time from the outcome time, we developed a generative approach to generate forward looking data to allow phenotypes to better capture the relevant outcome particularities. Furthermore, the generative feature provides explainability to clinicians for the model decision making as well as improving the outcome prediction

Reviewer comment (Evaluation):
Why does CAMELOT display a similar performance to DeepGC on one dataset and not the others?

Author response:
**[CAMELOT Performance]**: It is not immediately clear why Camelot performs similarly to DeepGC on the HAVEN dataset, but not so on the MIMIC data. It could potentially be due to the ratio between different outcome prevalences in the data. As an example, the ratio of the number of patients with a given outcome between the most frequent and second most frequent outcomes is around 12% on HAVEN, but closer to 21% on MIMIC. Consequently, it could be the case that DeepGC has more modelling capabilities than CAMELOT, and henceforth is better equipped to navigate this variability in outcome distributions.

Reviewer comment (Novelty):
The paper's technical novelty appears somewhat limited. In particular, it fails to adequately address the unique challenges presented by modeling EHR data compared to other sequential data. It also does not sufficiently explain the novelty in terms of model design compared to existing sequential generative models. It's important to note that in the machine learning context, several VAE-based architectures have been proposed to handle various types of sequential data, such as images, audio, and videos [1, 2].

Author response:
**[Technical Novelty]**: Addressing the technical novelty and challenges related to EHR data, we do not claim to have solved EHR modelling, but we disagree with the reviewer regarding the usefulness of DeepGC given the results obtained. DeepGC obtains close to top performance across all evaluation metrics (evaluated at both the observation and outcome aspects) and all datasets. We note that both models reference by the reviewer are extremely similar in methodology to VRNN [1] (in fact, one can argue that they are a particular type of VRNN-type network), but, more fundamentally, they lack the ability to phenotype or dynamically identify clusters (i.e., clustering changes over time). Recall our main task is that of developing methods capable of dynamically identifying relevant phenotypes/clusters across a variety of aspects (in our case a temporal, and an outcome aspect).

Reviewer comment (Theory):
The model's design lacks clear explanations or motivation. For instance, it is unclear why VAE was chosen over other generative models for modeling the EHR data distribution. Furthermore, the rationale behind modeling the cluster probability distribution using a Dirichlet distribution remains unexplained.

Author response:
**[Model Design]**:  The rationale for adopting a generative approach in our study is grounded in empirical observations. It became apparent through experimentation that existing phenotypes encountered challenges in learning meaningful and expressive patterns across both modeling aspects, particularly in a dynamic approach. We observed that the separability of outcome sub-cohorts is bigger at time indices closer to the actual event occurrence. Consequently, we decided to leverage a component capable of generating future observation data to reduce the conflation between modeling both aspects. 

In the medical setting, having a generative model would serve the benefit of allowing us to generate trajectories of physiological variables that change over time to demonstrate and explain the behavior of our model prediction. For example, if an individual is predicted to have cardiac arrest in the next two days, we would like to show the clinician the deterioration of their EHR information as it changes over time. Having the ability to generate such trajectories would be helpful in model interpretation and explanation. Furthermore, we have empirically demonstrated that our model is able to utilise the generated information to improve outcome prediction (see Table 1 where DeepGC outperformed CAMELOT across both tasks on almost all metrics) 

Crucially, our modeling assumptions hinge on the utilization of the Dirichlet distribution, playing a pivotal role in our model's performance. To achieve dynamically changing phenotype assignments, accurately modeling the temporal evolution of cluster assignment probabilities is essential. We opted for the Dirichlet distribution due to its parametrization over the space of probability vectors of dimension K, and its useful mathematical properties, such as a closed formula for the Kullback-Leibler divergence and an approximate reparameterization scheme that enables efficient backpropagation through the distribution parameters.

Reviewer comment (Experiments):
Connected to the previous concerns, it's uncertain whether the improvement in prediction performance stems from modeling patient subgrouping or is merely a consequence of the neural network architecture's high capacity. In other words, it is unclear whether patient subgrouping is essential, and the paper does not explore the consequences of removing patient subgrouping from DeepGC. To clarify, why did the authors not use a VAE architecture to model the temporal data (i.e., $P(X,Y)$) directly, instead of modeling $P(X,Y,\pi)$?

Author response:
**[Performance Improvement]**: We believe that the observed improvement in performance is not attributable to an increase in modeling capacity. Where applicable, all Deep Learning (DL) models underwent evaluation with an identical number of layers and nodes, where applicable. In fact, a closer look into our modeling assumptions shows the modeling capacity of DeepGC is slightly inferior to that of VRNN [1] due to the difference in latent variables. Notably, the benchmark VRNN, along with its variant VRNN-GMM model, can be viewed as ablated versions of DeepGC. Both approaches do not model outcome, however, they can model the multidimensional time-series data directly via latent variables. Although VRNN-GMM also implements a clustering component, this is a 2-stage process (i.e., the latent space is fixed and learned from VRNN separately, then clustering is achieved via GMM), where ours is end-to-end joint learning with the combination of outcome modeling and phenotyping that contributes significantly to the observed performance improvements.

Reviewer comment (Evaluation):
The concerns raised are somewhat substantiated by the results presented in Table 1. The model with strong patient subgrouping performance (TSKM) does not perform well on downstream prediction tasks. Additionally, DeepGC's performance falls within one standard deviation of the second-best baseline models. Consequently, the paper would benefit from statistical testing to establish the significance of the proposed method.

Author response:
**[Comparison with Benchmarks]**:  Recall that the main goal of our paper is to develop a model capable of identifying phenotypes across a variety of distinct aspects. The comparative performance of TSKM against DeepGC across both the outcome prediction aspect and observation aspect (clustering input time-series) highlights the difficulty in modeling the 2 aspects at the same time. For example, TSKM is the only model that outperforms DeepGC over the observation aspect, and yet it performs so much more poorly on the outcome prediction aspect. The cluster evaluation metrics such as Silhouette Score, Davies-Bouldin Index and Variance Ratio Criterion are naturally biased to favor TSKM as the metrics prefer convex clusters and the direct clustering on the input observation space. 

DeepGC outperforms all other benchmarks on the observation aspect in a statistically significant fashion using a Friedman’s hypothesis test. Furthermore, in the critical evaluation metric F1, which considers the severe outcome imbalance in the data, DeepGC outperforms all benchmarks on the MIMIC dataset, while showing top performance on the HAVEN data (albeit not statistically significant improvement over CAMELOT).

Reviewer comment (Reproducibility):
Several important technical details are omitted, such as the evaluation of patient subgrouping in an unsupervised setting. Furthermore, the paper does not explain how clustering metrics like SIL, DBI, and VRI were calculated. Data statistics and processing steps are also missing, including information about which clinical features were employed and how missing values were handled.

Author response:
**[Clustering Metrics]**: The evaluation metrics were calculated via the scikit-learn [2] package in Python 3. To account for the temporal 3D dimensionality of the input data, we represented each unique feature-time pair of values as a unique feature when evaluating clustering performance.  

Data handling followed previous work in [3], as was mentioned in the last paragraph of Section 4.1 “Both datasets underwent processing based on clinical knowledge and empirical validation as in Aguiar et al. (2022)”, and we will clarify in a revised version we refer to pre-processing steps, feature selection and dealing with missing values. Further descriptions of the data will be added to the appendix.

Reviewer comment (Reproducibility):
The paper lacks the provision of code and supplementary documentation, which would enhance clarity and reproducibility.

Author response:
**[Code and other documentation]**:  All code and supplementary material will be shared after revision. This is to ensure that we have the finalised the code for all analyses after revision of the manuscript.

Reviewer comment (Experiments):
It's not clear to me what outcome prediction task was chosen by the authors. Also, one randomly chosen outcome prediction task seems like a somewhat incomplete evaluation.

Author response:
**[Outcome Prediction]**: The outcome prediction task in our analysis stems directly from the clinical applications. For example, within the HAVEN ecosystem, admissions to the wards were categorized based on the occurrence of clinically significant events such as cardiac arrest (Cardiac), admission to an Intensive Care Unit (ICU), death (Death), or successful hospital discharge (Discharge). We represented outcomes in a 4-class multiclass setting, emulating clinical practice and focusing on key clinical variables of interest to practitioners. Notably, our models excluded data within 48 hours of an event/discharge to prevent learning immediate deteriorations and aimed to provide clinicians with insights into potential future events.

Similarly, for the MIMIC dataset, we defined outcomes based on patient location within 48 hours post-discharge from the Emergency Department (E.D.): death (Death), successful discharge (Discharge), ICU escalation (ICU), or whether the patient remained within a ward under observation (Ward). This approach, mirroring the 4-class representation above, captures clinically relevant measures of patient severity. Our rationale for these outcome definitions is grounded in their alignment with clinical practice, ensuring the applicability and robustness of our models in real-world settings.

Reviewer comment (Experiments):
I am curious why the authors chose SVM, xgboost as standard outcome prediction methods instead of say more standard time series approaches like RNN and LSTM. It would be nice to see these benchmarks.

Author response:
**[Benchmarking]**: We benchmarked Support Vector Machines (SVMs), XGBoost as standard outcome prediction models due to their relevance and usage in healthcare, as well as relatively easy training and hyper-parameter training procedure. Given we are proposing a Deep Learning (DL) methodology, it is unclear what would be the best LSTM or RNN-type architecture to validate our models against. Not only this, but previous approaches have been validated against types of LSTM/RNN baselines. However, we will include this in the final manuscript for completeness.

We selected Support Vector Machines (SVMs) and XGBoost as standard outcome prediction models due to their established relevance and widespread usage in healthcare. Furthermore, these models are also straightforward to train and tune, making them suitable benchmarks for our objective. Regarding comparisons with other Deep Learning (DL) models, we note this is a difficult problem as there are a huge number of architectures to select and there are yet no clearly defined benchmarks in our clinical setting. Nevertheless, previous studies have employed LSTM/RNN baselines, we acknowledge the importance of including such benchmarks for a more comprehensive evaluation and will incorporate this comparison in the final manuscript. We note, however, that we have already included some DL methods as part of our benchmarks, e.g. CAMELOT, and VRNN.