Paper: GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge Graphs (ICLR-2024)

Reviewer comment (Theory):
The knowledge graph building is somewhat unclear. A knowledge graph usually contains edges with multiple types. However, the edge type is not reflected in KG extraction, edge clustering, and the graph neural network.

Author response:
**W2: The knowledge graph building is somewhat unclear. A knowledge graph usually contains edges with multiple types. However, the edge type is not reflected in KG extraction, edge clustering, and the graph neural network.**

Thank you for your observation regarding the construction of our knowledge graph (KG). We included our KG construction details in ***Appendix D***. Our approach to KG assembly through LLM prompting leads to a variable number of edge types, which may differ from traditional KGs that typically feature a predefined edge taxonomy. The statistics regarding the number of edge types after clustering are revealed in ***Table 7 in Appendix D.3***.

Reviewer comment (Experiments):
One suggestion is to compare more clustering methods.

Author response:
**W3: One suggestion is to compare more clustering methods.**

Thank you for the valuable suggestion to explore a broader range of clustering methods. We agree that a comprehensive comparison, including hyperparameter tuning, is crucial for a thorough evaluation of clustering strategies. Due to the extensive nature of these experiments, we will diligently work on this analysis and aim to present the results in our final revision. 

**(References are attached in Part 2)**

Reviewer comment (Reproducibility):
In node/edge clustering, a node can be a phrase. What is the word embedding of such nodes?

Author response:
**Q1: In node/edge clustering, a node can be a phrase. What is the word embedding of such nodes?**

As described by Appendix C.1, we retrieve embeddings using the $\texttt{text-embedding-ada-002}$ provided by the OpenAI service [2], which generates a single vector embedding for input text regardless of token count. Consequently, even when a node or an edge represents a phrase, the resultant embedding is a singular vector with a dimensionality of 1536. This dimension is consistent with that of embeddings for single-word nodes/edges.

Reviewer comment (Theory):
Above Equation (2), the authors said that reducing the size of node/edge embedding can handle the sparsity problem. What is the sparsity problem here, and why can the node/edge embedding have such a sparsity problem, given that they are dense vectors? If there indeed is a sparsity problem, can we decrease the initial embedding size?

Author response:
**Q2: Above Equation (2), the authors said that reducing the size of node/edge embedding can handle the sparsity problem. What is the sparsity problem here, and why can the node/edge embedding have such a sparsity problem, given that they are dense vectors? If there indeed is a sparsity problem, can we decrease the initial embedding size?**

We apologize for any confusion caused by the mention of a 'sparsity problem.' This term was included by mistake and has been corrected in our revised manuscript. The reduction in node/edge embedding size is intended to enhance the model's efficiency. We do not encounter a sparsity issue as the embeddings employed are dense. Regarding the initial size of the embeddings, we use the dimensions provided by the LLM, which are fixed at 1536.

Reviewer comment (Evaluation):
The performance of mortality prediction is questionable. While previous literature has reported AUPRC values exceeding 20 on the MIMIC dataset, the results presented in Table 2 fall notably short of this benchmark [1]. What leads to such a gap?

Author response:
Thank you for highlighting the performance discrepancy in mortality prediction as compared to previous literature. The observed gap can be attributed to several key differences:

1. **Task Definition:** Our study defines mortality prediction as forecasting the outcome of the next visit based on previous visits, aligning with studies [3, 4]. This contrasts with [1], which considers mortality within specific time windows and employs a different feature selection approach.
2. **Input Features:** Unlike [1], which selects features from multiple tables (*chartevents, labevents, outputevents, icustays, diagnoses_icd*) in the MIMIC-III datasets, our study uses features from the (*diagnoses_icd, procedures_icd, prescriptions*) tables, consistent with the setting in [3].
3. **Medical Coding:** We use the CCS and ATC (level-3) coding systems, as mapped from ICD-9 and NDC respectively, a well-recognized setting in literature [3, 6, 7]. This differs from the ICD-9 coding system used in [1].

These factors collectively contribute to the differences in performance results.

Reviewer comment (Writing):
The paper delves into various engineering details, such as Knowledge Graph (KG) construction and attention initialization, without offering sufficient clarity. The notation in sec 3.3 are too complicated to follow. Many of them are randomly presented without definition.

Author response:
Thank you for pointing out the need for greater clarity in our explanation of the Knowledge Graph (KG) construction and attention initialization, as well as the complexity of notations in Section 3.3. 

**Engineering Details:** To enhance clarity, we have comprehensively updated ***Appendix C***. This appendix now contains detailed explanations of each step mentioned in Section 3, ensuring a clearer understanding of the engineering aspects of our work.

**Notations Definition:** We understand that the notations may have seemed complex. To address this, we have included a comprehensive notation-description table in ***Appendix H - Table 11*** in the updated version of our paper. This table is designed to provide clear definitions and explanations for the notations used, aiding in easier comprehension of the paper.

We hope these updates make the technical aspects of our paper more accessible and easier to follow.

Reviewer comment (Experiments):
The paper only compares with the baseline methods without leveraging external information / knowledge. However, it will be more important to compare with knowledge graph generated by other methods [2].

Author response:
Thank you for suggesting a comparison with the method described in [2], which utilizes the Google Health Knowledge Graph (GHKG). We acknowledge the value of such a comparison; however, there are practical constraints to consider:

1. **Availability of GHKG:** The GHKG, a key component of the method in [2], is not publicly available. Access to this resource is crucial for replicating or comparing with their approach. Unfortunately, obtaining permission from Google for access to the GHKG is not feasible within the limited time frame of this discussion window.

2. **Comparison with Other Graph-Based Methods:** While we were unable to compare with the GHKG-based method, we have conducted comparisons with other graph-based methods, GRAM [8] and GAMENet [9]. These comparisons are detailed in Table 2 and Figure 2 of our paper, providing insights into the performance of our approach relative to other graph-based methods that do not rely on proprietary datasets like the GHKG.

We believe these comparisons, although not with the specific method you suggested, still offer valuable insights into the efficacy of our approach in the context of available graph-based methods in the field.

Reviewer comment (Writing):
What is κ-hop subgraph in Section 1?

Author response:
The $\kappa$-hop subgraph concept refers to a graph sampling technique where we start with a specific entity, represented by the concept $e$, and then include all connected entities within $\kappa$ steps (or hops) from $e$. This process effectively captures the local neighborhood around the concept $e$ up to a distance of $\kappa$ edges.

For example, in a 2-hop subgraph, we would include not only the entities directly connected to $e$ but also those that are two connections away. This method allows us to construct a localized view of the knowledge graph, focusing on the immediate and near-immediate relational context of a given entity.

The $\kappa$-hop subgraph sampling is further detailed in ***Appendix D.2***.

Reviewer comment (Experiments):
How is distance threshold δ selected for node and edge clustering?

Author response:
We detailed the tuning of threshold $\delta$ in ***Appendix G.1*** where we experimented with thresholds ranging from 0.05 to 0.20 to determine the most effective value. The performance comparison in ***Table 9*** and the examples shown in **Figure 9** both indicate that a threshold of 0.15 yields the best balance.

Reviewer comment (Writing):
What is AGGREGATE in equation 1?

Author response:
The term "AGGREGATE" in Equation 1 refers to a method used in Graph Neural Networks (GNNs) to combine, or 'aggregate', information from neighboring nodes/edges of a graph. In simple terms, for each node (a point in the graph), we look at its neighbors (other connected points) and gather their information to update the node's own information.

In our work, this process of aggregation not only involves bringing together information from neighboring nodes (including the node itself) but also incorporates the type and significance of their connections. This is detailed in Equation 4, where we show how both node information and connection (edge) information are combined to enhance the understanding of each node within the graph.

Reviewer comment (Reproducibility):
The complexity of the proposed framework presents challenges for implementation. The paper would benefit from a more comprehensive explanation of the implementation specifics of the framework.

Author response:
Thank you for your feedback regarding the complexity of our framework and the need for clearer implementation details. We agree that a more detailed explanation would enhance the understanding of our framework.

1. In our original submission, we provided a summary of the implementation details in the README.md file within the supplementary material's code folder. This was intended to offer quick access to the necessary information for replicating our framework.

2. Following your suggestion, we have revised and expanded the ***Appendix C - Implementation Details*** section. This update aims to provide a more thorough and organized explanation of the implementation process, ensuring that readers can easily understand and apply our framework. We have also created a comprehensive notation table in ***Appendix H*** to enhance the clarity of our paper.

We believe these enhancements will greatly assist in comprehending and utilizing our proposed framework effectively.

Reviewer comment (Evaluation):
While the authors discuss the interpretability of GraphCare in Section 4.3, the depth of analysis is confined to observations on entity connections and their impact on model performance. It would be beneficial if the framework could uncover deeper insights in line with existing literature, or identify longer-distance relationships that might inspire new scientific discoveries.

Author response:
Thank you for your valuable feedback regarding the depth of our interpretability study in GraphCare. We agree that exploring deeper insights and longer-distance relationships could significantly enhance its application in real-world scenarios and foster scientific discoveries.

To address your point, GraphCare could indeed be adapted to investigate critical factors in various scenarios. For instance, in identifying key causes of a disease, we can focus on the disease as the main prediction target, systematically examining and pruning long-distance relationships to pinpoint the most crucial ones. Similarly, we could study how these relationships might correlate with a patient's age or other demographic details. The potential applications are practically endless.

It's essential to recognize that conducting such in-depth analysis, especially in a clinical setting, demands comprehensive validation. This might involve integrating sophisticated fact-checking methods with advanced information retrieval techniques. While this level of detailed scrutiny is crucial, it extends beyond the primary scope of our current study. 

Our main contribution lies in developing GraphCare as a versatile framework. This framework is instrumental in constructing personalized KGs and adeptly managing diverse prediction tasks. Nonetheless, we acknowledge the significance of your suggestion. It aligns well with our objectives, and we are eager to explore it as a central aspect of our future research.

Reviewer comment (Experiments):
In relation to the chosen experimental tasks, it would be helpful if the authors could elaborate on their selection criteria. Is there a particular preference for the information that the constructed graph helps in elucidating?

Author response:
Thank you for highlighting the need for further explanation on our selection of experimental tasks in relation to the constructed graph in GraphCare. Our decision was strategically aligned with showcasing the graph's ability to extract and utilize complex patient data in various healthcare scenarios:

1. ***Mortality Prediction:*** This task was chosen to demonstrate the graph's capability in processing critical, binary outcomes. The graph's structure and embeddings are particularly adept at highlighting key indicators relevant to mortality, which is a crucial, high-stakes decision-making scenario in clinical settings.

2. ***Readmission Prediction:*** This task leverages the graph's ability to capture short-term patient trajectories, utilizing the temporal data embedded in the graph. It showcases the framework's proficiency in interpreting dynamic patient data for predicting readmissions, a key aspect of hospital resource management and patient care.

3. ***Length-Of-Stay Prediction:*** The task of predicting ICU stay lengths is aligned with the graph's ability to categorize and analyze time-sensitive data. It exemplifies the graph's utility in operational healthcare planning, where understanding and predicting patient flow and resource needs are essential.

4. ***Drug Recommendation:*** This complex task was specifically chosen to highlight the graph's strength in multi-dimensional analysis. It demonstrates how the graph can integrate and interpret a multitude of patient-specific data points, essential in the multi-faceted decision process of medication prescription. This task underscores the graph's capacity to handle multiple predictions simultaneously, a scenario that mirrors the complexities of real-world clinical decision-making.

Each of these tasks was selected not only for their relevance to healthcare but also for their alignment with the strengths of our graph in processing diverse, intricate, and multi-dimensional patient data. This careful alignment ensures that our experimental tasks comprehensively evaluate and showcase the capabilities of GraphCare in interpreting and utilizing the rich information provided by the constructed graph.

Reviewer comment (Experiments):
The authors are encouraged to provide additional results on the quality assurance of their constructed personalized knowledge graphs. The significance of this is underscored by the direct impact these graphs have on the results.

Author response:
Thank you for your question about the quality assurance of our constructed personalized knowledge graphs (KGs) and your concern regarding potential biases or misleading information. 

1. **Quality Assurance:** In Figure 3, we have shown the effect of different KG sizes on performance, which indirectly reflects the quality of the KGs in terms of their completeness. We believe completeness is a critical measure of quality. To ensure this, as detailed in Appendix D.1, we employed a meticulous process of human-annotated triple selection, highlighted in blue, to eliminate any inaccurate or potentially biased information from the KGs.

2. **Addressing Biases and Misleading Information:** To further address your concern about biases and misleading information, we conducted additional tests. These involved reintroducing the previously removed 'negative' information (inaccurate, biased, and misleading) into the KGs. The results, tested on the MIMIC-III dataset using GPT-UMLS-KG, are as follows (with the bracketed results representing the cleaned KG):

|                            | AUPRC       | AUROC       |
| -------------------------- | ----------- | ----------- |
| **Mortality Prediction**   | 16.2 (16.7) | 69.7 (70.3) |
| **Readmission Prediction** | 73.1 (73.4) | 69.2 (69.7) |

|                               | AUROC       | Kappa       | accuracy    | F1-Score    |
| ----------------------------- | ----------- | ----------- | ----------- | ----------- |
| **Length-of-Stay Prediction** | 80.9 (81.4) | 29.0 (29.5) | 42.9 (43.2) | 37.0 (37.5) |

|                         | AUPRC       | AUROC       | F1-Score    | Jaccard     |
| ----------------------- | ----------- | ----------- | ----------- | ----------- |
| **Drug Recommendation** | 79.5 (80.2) | 95.2 (95.5) | 66.0 (66.8) | 49.1 (49.7) |

  These results demonstrate that while the inclusion of negative information in the KG can slightly reduce performance, GraphCare still consistently outperforms all baselines. This indicates the framework's resilience to noisy graphs and its ability to handle some level of imperfect data.

We hope these explanations and results adequately address your concerns about KG quality and the steps we've taken to mitigate potential biases and inaccuracies in our framework.