Paper: Graph Transformers on EHRs: Better Representation Improves Downstream Performance (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is a combination of building blocks originated from several previous papers, e.g. graph transformer, BEHRT. It is a good application paper that utilizes all these aspects, but the methodological impact is limited.

Author response:
**Good application paper, limited methodological impact**
While our study, at its core, is application-driven (i.e., EHR analysis), we believe it presents several fundamental algorithmic contributions. A key contribution is presenting an effective method to extract visit-level embeddings for the transformer-based methods, instead of the common code-level embeddings. Making such pipeline work required many engineering and design considerations, especially using a graph-based approach. We also show that our pipeline helps with the quadratic complexity of transformer-based methods. Moreover, we present two new pretraining methods, that are specifically tailored to our design. We have made adjustments to the Intro to more clearly present our technical contributions.

Reviewer comment (Evaluation):
In the ablation study section, the author claims that "both pre-training strategies seem to have a great impact on the overall results, as showcased by the second, fourth, and fifth rows.". However, in fact, Table 3 shows the performance differences between experiments with and without NAM, MNP, and GT are marginal. The confidence interval of the last row and the first row (the simplest version) even overlap. The simple linear model seems to already perform very well compared to GT. A better way of justifying the impact of pretraining can be directly using linear probing on the frozen pre-trained representation.

Author response:
**Ablation study clarification**
We agree that the linear model already performs well in comparison to the other baselines, however, we believe that the impact of pre-training should be compared using the same architecture baseline. That is, one should compare the results of the linear model with pre-training (2nd row) and without pre-training (1st row), and compare the GT model with pre-training (4th/5th row) and without (3rd row) pre-training. By doing so, it can be seen that when using GT, 8 of the 12 reported metrics using the pre-training outperform the GT baseline (these differences are also statistically significant using a t-test). Additionally, we believe that the true impact of the pre-training process is better highlighted by the final version of the model, which uses the entire two-step pre-training process. Nevertheless, we agree that the impact of pre-training on the linear model, while showing some improvements, remains marginal and we have made this aspect clearer in the manuscript.

Reviewer comment (Experiments):
The power of pretraining seems to be limited by the size of the dataset (around 20K patients). Under this condition, pretraining a large model may not significantly outperform a simple linear model. It will be more exciting to apply the proposed method to a larger dataset.

Author response:
**Use a larger dataset**
We have now run all experiments on a large cohort of the All of Us dataset (a very large dataset of ~1 million individuals). The Experiments section is updated accordingly.

Reviewer comment (Evaluation):
How is the F1 score calculated? Which threshold is picked? Some experiments with similar AUPRC result in substantially different F1 scores.

Author response:
**F1 score clarification**
For all experiments run, the F1 scores were calculated using the “torcheval” library, using the standard formula and a threshold of 0.5. The disparity in the F1 scores can probably be explained by a poorer calibration of the models where the threshold of 0.5 is suboptimal. We added a short note to the Experiments section about this.

Reviewer comment (Presentation):
The method description can be improved upon. Consider adding an illustration of the training strategy and describing the methods using the illustrations.

Author response:
**Improve Method, new illustration**
We replaced Figure 1 (architecture visualization) with a new improved one. We have also updated the Method section to improve the descriptions. Moreover, we added a new section (Appendix B) containing the new Figure 4, to improve the illustration of the training strategy.