Paper: HawkesVAE: Sequential Patient Event Synthesis for Clinical Trials (ICLR-2024)

Reviewer comment (Novelty):
The primary weakness is limited novelty. This is not the first paper to propose combining VAEs with Hawkes, e.g.  
- Sun et al. (2021) "Towards a predictive analysis on disease progression: a variational Hawkes process model" IEEE Journal of Biomedical and Health Informatics  
- Lin et al. (2021) "Disentangled deep multivariate hawkes process for learning event sequences" ICDM  
- Pan et al. (2020) "A variational point process model for social event sequences" AAAI  
Hence the primary point of novelty appears to be the scheme for sampling next events and the fact that using the logits is better than greedy sampling, which is not surprising.

Author response:
Thank you so much for pointing us to these papers; we have added them to our literature review. We believe that although our method applies VAE and Hawkes Processes together, it has merit as it outperforms synthetic tabular generation models as well as LSTMs and PAR. We also believe that the detailed level of control (allowing the user to specify event types and times) is highly valuable in settings such as clinical trial patient generation [2,3], and are not explicitly supported by other models. Finally, we have performed a literature review on the papers cited, and found that none of them, including Transformer Hawkes Models and Cox-Hawkes models, test on whole-sequence time and event generation like in our evaluations. We believe that a source of our novelty also lies in our application, where sequential event data generation is an underexplored, yet exciting field. We have not experimented with Cox-Hawkes, but this looks like an exciting future direction.

Reviewer comment (Evaluation):
Even with the limited baselines, the empirical performance is not compelling (this might be less of a concern if the contribution was more novel). While the proposed method generally has the best performance, the confidence intervals often overlap with DDPM (both on downstream performance and classification of real vs synthetic instances) — this is somewhat confusing in the presentation of the results as "best performance" is bolded for multiple methods, but competing methods with overlapping confidence intervals are not bolded.

Author response:
A quick comment regarding the performance of HawkesVAE. You are absolutely correct on the bolding of the ROC metric, and we have corrected this, as it now indicates if the original data mean ROC is within 1 standard deviation. Post initial submission, we actually realized that our downstream model was underfitting, as we discovered that our sanity check results (from training the model on original data) did not match previous existing work [1]. We have now reran the results with a more robust downstream classifier, and seen a boost in all performance for all models in general. HawkesVAE (Multivariate) outperforms the and the next best model (in 4/7 datasets and is within 1 standard deviation with the rest of the datasets). We have updated Tables 2 and 5 accordingly.

Reviewer comment (Theory):
While the method is well described intuitively, some specifics are not given in clear formal terms, e.g. greedy vs. logit sampling could be provided formally/mathematically, the equations for generating sequences in the encoder/decoder and combining them in the encoder should be provided.

Author response:
We agree with the reviewer, and have taken steps to clarify the variables in Section 4.1. Specifically, we clarify that the encoder model $E_(\mathcal{H}\_i) \rightarrow \hat{\mu}, \hat{\sigma}$ takes in the original event types and times, and predicts the mean and standard deviation to sample $z \sim Normal(\hat{\mu}, \hat{\sigma})$. This is used for $q_\phi(\cdot | S_z)$ in the loss function and follows the previous Transformer Hawkes Process implementation of a transformer encoder, with alternating sine and cosine waves to denote temporal information. 

However, the decoder model is more complicated. Hawkes Process is usually evaluated via one prediction step at a time (i.e., the input is the ground truth time-step and event type, and the task is to predict the next time step and event type). For our purposes, we need to adapt this into an autoregressive decoding scheme similar to the language modeling task.

At training time, the input to the decoder $D(z, \mathcal{H}\_i) \rightarrow (\hat{t}\_{i+1}, \hat{k}_{i+1}, \lambda)$ is hidden vector $z$ and a sequence of ground truth event types and times. It is tasked with predicting the next event type $\hat{k}$, next event time $\hat{t}$, and the $\lambda$s necessary to compute $P\_\theta(\cdot|S_z)$. 

At inference time, the input to decoder is only $z$, and we auto-regressively decode the predicted event types and times. To predict next time and event tuple $(\hat{t}\_i, \hat{k}\_i)$, the input is the previously predicted times and events $\{(\hat{t}\_1, \hat{k}\_1), \dots, (\hat{t}\_{i-1}, \hat{k}_{i-1}))\}$). (each predicted time and event is repeatedly appended to the input). 

We have clarified this in section 4.1 as well as Appendix A.3.

Reviewer comment (Novelty):
What do the authors consider to be the primary point of novelty over existing approaches which combine VAEs with Hawkes processes or related approaches like Transformer Hawkes?

Author response:
Thank you so much for pointing us to these papers; we have added them to our literature review. We believe that although our method applies VAE and Hawkes Processes together, it has merit as it outperforms synthetic tabular generation models as well as LSTMs and PAR. We also believe that the detailed level of control (allowing the user to specify event types and times) is highly valuable in settings such as clinical trial patient generation [2,3], and are not explicitly supported by other models. Finally, we have performed a literature review on the papers cited, and found that none of them, including Transformer Hawkes Models and Cox-Hawkes models, test on whole-sequence time and event generation like in our evaluations. We believe that a source of our novelty also lies in our application, where sequential event data generation is an underexplored, yet exciting field. We have not experimented with Cox-Hawkes, but this looks like an exciting future direction.

Reviewer comment (Experiments):
Have the authors compared their approach to Transformer Hawkes Process or Cox-Hawkes?

Author response:
Thank you so much for pointing us to these papers; we have added them to our literature review. We believe that although our method applies VAE and Hawkes Processes together, it has merit as it outperforms synthetic tabular generation models as well as LSTMs and PAR. We also believe that the detailed level of control (allowing the user to specify event types and times) is highly valuable in settings such as clinical trial patient generation [2,3], and are not explicitly supported by other models. Finally, we have performed a literature review on the papers cited, and found that none of them, including Transformer Hawkes Models and Cox-Hawkes models, test on whole-sequence time and event generation like in our evaluations. We believe that a source of our novelty also lies in our application, where sequential event data generation is an underexplored, yet exciting field. We have not experimented with Cox-Hawkes, but this looks like an exciting future direction.

Reviewer comment (Reproducibility):
Does the proposed model require the same sequence length for all patients?

Author response:
Our model supports patients of any sequence length. We have clarified this point in the manuscript (Section 5 Datasets)

Reviewer comment (Novelty):
The paper seems like a straightforward application of NHP and VAEs to the time-series data.

Author response:
We believe that although our method applies two methodologies together, it has merit as it outperforms synthetic tabular generation models as well as LSTMs and PAR. We also believe that the detailed level of control (allowing the user to specify event types and times) is highly valuable in settings such as clinical trial patient generation, and are not explicitly supported by other models.

Reviewer comment (Writing):
The writing could be improved to focus on the paper's key contributions. For instance, given that a previous derivation of the NHP in Section 3.2 has been proposed, unless there are new insights, the NHP could be summarized.

Author response:
This is a good point, and we have shortened the relevant work to a single page to reflect this change. We have moved the information regarding traditional Hawkes Processes and VAEs to the appendix as well.

Reviewer comment (Reproducibility):
The description for HawkesVAE is lacking in clarity regarding the following: 1) How is \( p_\theta(|zS_z) \) parametrized? 2) How is \( q_{\phi}(z|S_z) \) parametrized? 3) How is \( z \) used in the log-likelihood \( p_{\theta}(S_z|z) \)?

Author response:
We agree with the reviewer, and have taken steps to clarify the variables in Section 4.1. Specifically, we clarify that the encoder model $E_(\mathcal{H}\_i) \rightarrow \hat{\mu}, \hat{\sigma}$ takes in the original event types and times, and predicts the mean and standard deviation to sample $z \sim Normal(\hat{\mu}, \hat{\sigma})$. This is used for $q_\phi(\cdot | S_z)$ in the loss function and follows the previous Transformer Hawkes Process implementation of a transformer encoder, with alternating sine and cosine waves to denote temporal information. 

However, the decoder model is more complicated. Hawkes Process is usually evaluated via one prediction step at a time (i.e., the input is the ground truth time-step and event type, and the task is to predict the next time step and event type). For our purposes, we need to adapt this into an autoregressive decoding scheme similar to the language modeling task.

At \textit{training time}, the input to the decoder $D(z, \mathcal{H}\_i) \rightarrow (\hat{t}\_{i+1}, \hat{k}_{i+1}, \lambda)$ is hidden vector $z$ and a sequence of ground truth event types and times. It is tasked with predicting the next event type $\hat{k}$, next event time $\hat{t}$, and the $\lambda$s necessary to compute $P\_\theta(\cdot|S_z)$. 
At \textit{inference time}, the input to decoder is only $z$, and we auto-regressively decode the predicted event types and times. To predict next time and event tuple $(\hat{t}\_i, \hat{k}\_i)$, the input is the previously predicted times and events $\{(\hat{t}\_1, \hat{k}\_1), \dots, (\hat{t}\_{i-1}, \hat{k}_{i-1}))\}$). (each predicted time and event is repeatedly appended to the input).

Reviewer comment (Experiments):
Figure 1: It seems the model outputs event times and event lengths. Does the model predict event types? Why does HawkesVAE require event lengths and event types at inference time? Are these provided to baselines as well?

Author response:
The model autoregressively predicts event types and times, and event lengths and event type information are optional inputs. The model without event type information is denoted as HawkesVAE (Multivariate), and the model with event type information is denoted as HawkesVAE (Events Known). The comparisons are indeed fair, as baseline models vs HawkesVAE (Multivariate) is given the same amount of information. We have also taken steps to clarify Figures 1 and 3.

Reviewer comment (Presentation):
Tables 2, 3, 5: HawkesVAE results are bolded even in instances where baselines are better than HawkesVAE, which is misleading.

Author response:
A quick comment regarding the performance of HawkesVAE. You are absolutely correct on the bolding of the ROC metric, and we have corrected this, as it now indicates if the original data mean ROC is within 1 standard deviation. Post initial submission, we actually realized that our downstream model was underfitting, as we discovered that our sanity check results (from training the model on original data) did not match previous existing work [1]. We have now reran the results with a more robust downstream classifier, and seen a boost in all performance for all models in general. HawkesVAE (Multivariate) outperforms the and the next best model (in 4/7 datasets and is within 1 standard deviation with the rest of the datasets). We have updated Tables 2 and 5 accordingly.

Reviewer comment (Experiments):
Given that HawkesVAE requires access to real-world data to learn a generative model, the benefits of using the synthetic data over real-world data are not motivated.

Author response:
We understand the reviewer’s concern regarding this. In fact, synthetic trial generation is a recently proposed field of research, and has already become an industry product. Relying on real world data to train such models is indeed an issue inherent to all synthetic data generative models, but we believe that the use cases (data insights without revealing patient personal information) are highly practical for real world applications.

Reviewer comment (Reproducibility):
How are the encoder and decoder functions specified?

Author response:
We agree with the reviewer, and have taken steps to clarify the variables in Section 4.1. Specifically, we clarify that the encoder model $E_(\mathcal{H}\_i) \rightarrow \hat{\mu}, \hat{\sigma}$ takes in the original event types and times, and predicts the mean and standard deviation to sample $z \sim Normal(\hat{\mu}, \hat{\sigma})$. This is used for $q_\phi(\cdot | S_z)$ in the loss function and follows the previous Transformer Hawkes Process implementation of a transformer encoder, with alternating sine and cosine waves to denote temporal information. 

However, the decoder model is more complicated. Hawkes Process is usually evaluated via one prediction step at a time (i.e., the input is the ground truth time-step and event type, and the task is to predict the next time step and event type). For our purposes, we need to adapt this into an autoregressive decoding scheme similar to the language modeling task.

At \textit{training time}, the input to the decoder $D(z, \mathcal{H}\_i) \rightarrow (\hat{t}\_{i+1}, \hat{k}_{i+1}, \lambda)$ is hidden vector $z$ and a sequence of ground truth event types and times. It is tasked with predicting the next event type $\hat{k}$, next event time $\hat{t}$, and the $\lambda$s necessary to compute $P\_\theta(\cdot|S_z)$. 
At \textit{inference time}, the input to decoder is only $z$, and we auto-regressively decode the predicted event types and times. To predict next time and event tuple $(\hat{t}\_i, \hat{k}\_i)$, the input is the previously predicted times and events $\{(\hat{t}\_1, \hat{k}\_1), \dots, (\hat{t}\_{i-1}, \hat{k}_{i-1}))\}$). (each predicted time and event is repeatedly appended to the input).

Reviewer comment (Experiments):
It’s not clear to me what types of events are available in those datasets, how general adverse events, serious adverse events, and death are handled and once generated would that generated event (by serious/severity) lead to more/less frequent or cancellation of afterward event generation.

Author response:
Each dataset contains events and the times at which they occur, e.g. medications, procedures, as well as some adverse events like vomiting, etc. We use these datasets to predict if the subject experiences the death event, which is an external label.

Reviewer comment (Writing):
P3, What are the other events mentioned in “death event reduces the probability of most other events”?

Author response:
We have changed this to be more representative of our datasets and evaluation task, as death is not actually an event that is in the training set. Rather, it is an external label of patient status, and the events consist of events and the times at which they occur, e.g. medications, procedures, as well as some adverse events like vomiting, etc. We will use these datasets to predict if the subject experiences the death event, which is an external label. We have also changed the example to “e.g., medication reduces the probability of adverse events”.

Reviewer comment (Presentation):
P4, Figure 1. I didn’t quite follow how the [0,1,1,0,0] (5 events) event indicator was paired with your two event time vectors [1,3,5,6] and [1,2,3] (total 7 time points), and how matched with the event length [4,3] (two durations for the two “1” right).

Author response:
We have clarified the figure so that the event lengths are labeled. Originally, I meant to demonstrate that within 5 events, events 2 and 3 occur in the subject with times [1,3,5,6] (length 4) and [1,2,3] (length 3). I realize that this is confusing and have clarified the figure with labels of a more intuitively understandable event sequence (“Vomiting” and “Fever”). Please see Figures 1 and 3.

Reviewer comment (Experiments):
P6 Citation for CTGAN (why is it not included as a baseline?).

Author response:
- Why is CTGAN not included as a baseline? 

We believed that since TabDDPM [1] showed showed significant superior performance vs CTABGAN [2] and CTABGAN+ [3] (both of these models outperform CTGAN), we assumed that it would be a stronger baseline of general synthetic tabular generation compared to CTGAN, and left out CTGAN in favor of more recent work.

Reviewer comment (Presentation):
Table 2: Overlap between HawkesVAE results and models trained on original data (should also be bolded).

Author response:
A quick comment regarding the performance of HawkesVAE. You are absolutely correct on the bolding of the ROC metric, and we have corrected this, as it now indicates if the original data mean ROC is within 1 standard deviation. Post initial submission, we actually realized that our downstream model was underfitting, as we discovered that our sanity check results (from training the model on original data) did not match previous existing work [4]. We have now reran the results with a more robust downstream classifier, and seen a boost in all performance for all models in general. HawkesVAE (Multivariate) outperforms the next best model (in 4/7 datasets and is within 1 standard deviation with the rest of the datasets). We have updated Tables 2 and 5 accordingly.