Paper: Bi-level Contrastive Learning for Knowledge Enhanced Molecule Representations (ICLR-2024)

Reviewer comment (Novelty):
The preliminary steps for pretraining the original molecule graph representation have previously been explored in the paper titled "Does GNN Pretraining Help Molecular Representation?" (Sun et al, 2022).

Author response:
We acknowledge that molecule graph pre-training strategies have been well-studied, as in the paper by Sun et al. [1]. However, the focus of our work is to investigate how biomedical knowledge graphs can enhance molecule representations, rather than proposing new strategies for molecule graph pre-training.

Reviewer comment (Novelty):
The pretraining strategies for bio Knowledge graph representation were also discussed in "Pre-training Graph Neural Networks for Molecular Representations: Retrospect and Prospect" (Xia et al, 2022).

Author response:
The paper by Xia et al. [2] does not discuss pre-training strategies for bio-knowledge graphs. Instead, it mentions only the **Chemical Element KG**, as proposed by KCL [3], which is significantly different from our constructed biomedical knowledge graph, MolKG:

   - **Chemical Element KG** is sourced from the Periodic Table of Elements and focuses on the attributes of chemical elements.
   - **MolKG** is a molecule-centric biomedical KG, integrating molecule-tailored biomedical information from PubChemRDF (the largest KG for chemistry) and PrimeKG (a precision medicine-oriented KG).

Reviewer comment (Novelty):
The construction of a Biomedical Knowledge Graph was previously detailed in the paper "A unified drug–target interaction prediction framework based on knowledge graph and recommendation system" (Ye et al, 2021).

Author response:
Ye et al.'s work [4] does not construct its own biomedical knowledge graph but uses existing datasets (Luo's dataset, Hetionet, Yamanishi_08's dataset, and BioKG). None of these datasets contains the abundant molecules found in our downstream datasets. In contrast, our MolKG construction can scale to any molecules contained by PubChem or PrimeKG, with their relevant KG triples extracted. We detailed our construction steps in Appendix B, along with the processing scripts.

Reviewer comment (Novelty):
The concept of contrastive learning between the molecular graph and an augmented knowledge graph for molecules was previously addressed in "Molecular Contrastive Learning with Chemical Element Knowledge Graph" (Fang et al, 2022).

Author response:
We would like to clarify some key differences in our approach compared to Fang et al, 2022 [3].

   In both papers by KCL [3] and KANO [5], contrastive learning is conducted between **(1) the molecule graph** and **(2) the augmented molecule graph**, where (2) is obtained by linking element-wise KG triples from the Chemical Element KG directly to each element of the molecule.

   In our work, differently, we conduct contrastive learning between **(1) the molecule graph**, and **(2) the central molecule node in (1)'s corresponding KG sub-graph**. Unlike KCL or KANO, we do not augment molecule graphs explicitly but treat the molecule as a node in the KG sub-graph. We also designed a self-supervised pre-training strategy (Eq. 3) for this KG sub-graph, the effectiveness of which is demonstrated in Appendix G.1.

   Please also feel free to read Appendix F for more details regarding our comparison to similar works.

Reviewer comment (Novelty):
This work is a combination of the previously mentioned research, with some customization.

Author response:
As explained in points 1 to 4, our work is not a simple combination of the mentioned research but introduces major innovations:

   1. A tailored molecule-centric KG providing comprehensive biomedical knowledge.
   2. A novel integration of molecular graphs with biomedical domain knowledge, treating molecules as nodes in KG sub-graphs.

   KANO [5] is an improved version of KCL [3] in both simplicity and performance stability. Thus, our comparison with KANO, as shown in Tables 2 and 3, underscores our unique contribution.

Reviewer comment (Novelty):
The performance of GODE appears to be close to that of "Molecular Contrastive Learning with Chemical Element Knowledge Graph" (Fang et al, 2022) and "Knowledge graph-enhanced molecular contrastive learning with functional prompt" (Fang et al, 2023).

Author response:
We believe that **being comparable to (and consistently outperforming) KANO means our method is good**, for the following reasons:

- **Distinct Information Utilization**: We use a completely different information source (molecule-centric biomedical KG), unlike KANO's Chemical Element KG, to enhance molecule representations and improve molecular property prediction.

- **Complementary to KANO's Contribution**: The molecule graph can be augmented either explicitly through KANO's element-wise linking or implicitly through our Gode's graph-as-a-node approach, or by exploring the combination of both.

- **Scalability and Performance Potential**: Figure 3 in our paper suggests Gode's performance may increase with a larger MolKG. Currently, we have only extracted triples for 65,454 molecules from PubChemRDF, which contains over 100 million molecules, indicating significant potential for further performance improvements.

- **Performance Improvement of Past SOTA Models:** We list the performance improvement over the previous SOTA reported by the past papers (including ours) on their common classification tasks:

  |                 |           BBBP           |         SIDER         |       ClinTox       |           BACE           |          Tox21          |      | Avergae |
  | :-------------- | :----------------------: | :-------------------: | :-----------------: | :----------------------: | :---------------------: | :--: | :-----: |
  | **MolCLR**  [6] |   -19.2% (over N-GRAM)   |    -0.1% (over RF)    | +3.0% (over D-MPNN) |     +2.7% (over RF)      |    -2.4% (over SVM)     |      |  -3.3%  |
  | **GEM** [7]     | +2.0% (over PretrainGNN) |  +0.6% (over N-GRAM)  | -0.6% (over D-MPNN) | +1.3% (over PretrainGNN) | 0.0% (over PretrainGNN) |      |  +0.7%  |
  | **KANO** [5]    |    +3.6% (over CMPNN)    | +2.0% (over GraphMVP) | +4.2% (over DMPNN)  |     +5.9% (over GEM)     |    +3.6% (over MPNN)    |      |  +3.9%  |
  | **Gode** (ours) |    +0.9% (over KANO)     |   +5.3% (over KANO)   |  +0.5% (over KANO)  |    +1.5% (over KANO)     |   +2.1% (over DMPNN)    |      |  +2.1%  |

Reviewer comment (Evaluation):
The author states the embedding produced by GODE is robust, but I do not see the corresponding discussion about robustness learning in the later sections. Could authors elaborate more?

Author response:
**Q1 (Robustness of Gode):** We believe that the leading performance of Gode across 10/11 datasets (which include 676 tasks in total) can indicate the robustness of the learned molecule representation.

Reviewer comment (Writing):
Explain more about what the positive/negative classes are in contrast learning.

Author response:
**Q2 (Positive/Negative Samples in Contrastive Learning):** 

Each positive sample in $\mathcal{D}^{+} =$ {$(m_i, G\_{\mathrm{sub}(m\_i, \kappa)}), y\_i = 1$}$\_{N\_p}$  is composed by **a molecule graph $m\_i$** and **its corresponding $\kappa$-hop molecule-centric KG sub-graph $G\_{\mathrm{sub}(m\_i, \kappa)}$** extracted from MolKG. 

Each negative sample in $\mathcal{D}^{-} =$ {$(m\_i, G\_{\mathrm{sub}(m\_j, \kappa)})\_{j \neq i}, y\_i = 0$}$\_{N-N\_p}$ is composed by **a molecule graph $m\_i$** and **a non-corresponding $\kappa$-hop molecule-centric KG sub-graph $G_{\mathrm{sub}(m_j, \kappa)})_{j \neq i}$** sampled from MolKG. To make the task more challenging, we further divide $\mathcal{D}^{-}$ into $\mathcal{D}^{-}\_{rand}$, and $\mathcal{D}^{-}\_{nbr}$, which are (1) randomly sampled from all negative molecule-centric KG sub-graphs, and (2) sampled from the sub-graphs of the neighbor molecule nodes connected to the positive molecule node ($m\_i$), respectively. 

We also improved the corresponding content in the revised paper for clarity.

Reviewer comment (Experiments):
In page 4 "Embedding Initialization", four methods were cited. Which method was used in the experiments? If all of them were used, did the authors observe any differences in performance?

Author response:
**Q3 (Embedding Initialization):** We use TransE [8] for the KG embedding (KGE) initialization, as we specified in ***Section 4.1 - Implementation*** and ***Appendix D.1***. For KGE methods, we tried TransE, RotatE [9], DistMult [10], and TuckER [11], among which TransE performed the best regarding the final performance on the property prediction tasks.

Reviewer comment (Experiments):
My primary concern stems from the experimental setup's consistency. The baseline methods used for comparison seem to follow different settings, as the authors mention in the appendix that they have reproduced these baselines using their official implementations. Some of these baselines follow the settings from [1], while others have their own network structures. To validate the effectiveness of their representation learning, experiments should ideally be conducted under the same settings, including the same backbone model, node/edge feature selection, and other relevant parameters. Different representation learning methods can yield different results when applied to different settings. For instance, the outcome of using Grover on GIN might differ from using it on GTransformer[2]. If each method is used with different settings, it becomes challenging to assess the effectiveness of the proposed method. The final results may be influenced by the quality of the chosen backbone models, making it difficult to draw meaningful comparisons and conclusions about the proposed method's performance. Therefore, it is essential to standardize the experimental settings to ensure a fair and consistent evaluation of all methods.

Author response:
**W1: Experiments should ideally be conducted under the same settings.**

Thank you for highlighting the importance of uniform experimental settings across different models. In our study, we paid careful attention to this aspect to ensure a fair and meaningful comparison of our GODE method with the baseline models.

To address your concern regarding the standardization of experimental settings, we acknowledge that our original description in Appendix D may have caused some confusion. To clarify, this meant that while we utilized the original model architectures as provided by the authors, we also meticulously standardized all other experimental settings for a fair comparison. Our goal was to ensure that each method was evaluated on a consistent and comparable basis. 

Taking your feedback into consideration, we have further refined our experimental design for an even more equitable comparison. Specifically, we have now reproduced $\text{MolCLR}\_{\text{GTrans}}$ using the GTransformer backbone and $\text{KANO}\_\text{CMPNN}$ with its default CMPNN backbone as specified in the KANO paper. It's crucial to note that the results for GROVER and KANO presented in our initial submission were both based on the GTransformer backbone.

In our revised submission, we have added a new section, ***Appendix D.2 Baseline Models***, which provides detailed insights into the experimental settings used for each baseline model. This addition aims to offer a transparent and comprehensive view on how we ensured uniformity in our comparative analysis.

The results reflecting these changes also have been updated in ***Table 2*** and ***Table 3*** of our revised paper (in blue for easy identification). We believe these enhancements and clarifications in our methodology significantly strengthen the validity of our findings and address your concerns regarding the experimental setup's consistency.

Reviewer comment (Reproducibility):
Furthermore, the experimental results for GROVER appear to exhibit significant deviations from what is reported in the original paper. This raises concerns about whether the authors may have overlooked or omitted some critical details during the reproduction process. It is imperative for the authors to thoroughly investigate and address any potential discrepancies in the results, ensuring that the experiments align closely with the findings presented in the GROVER paper.

Author response:
**W2&Q2:  The experimental results for GROVER appear to exhibit significant deviations from what is reported in the original paper.**

We appreciate your concern regarding the observed deviations in GROVER's performance in our study compared to the results reported in its original paper [1]. We would like to emphasize that such discrepancies are not unique to our research. Similar deviations have been reported in other significant studies, including MGSSL [2], GEM [3], Mole-bert [4], and KANO [5], all of which were published in highly respected venues.

Our experimental settings strictly adhered to the guidelines and configurations outlined in the original GROVER paper.

Reviewer comment (Reproducibility):
Why is the GROVER results in the paper significantly lower than the reported results in the original paper?

Author response:
## **Regarding GROVER's performance**

Thank you for your valuable feedback regarding GROVER's performance. In response, we have incorporated the fine-tuned models as suggested, and our findings are presented in the tables below. We also include the results of other studies' reproduction of GROVER, highlighting the issue of reproducibility.

**Performance on Classification Tasks:**

|                                                           | BBBP           | SIDER          | ClinTox         | BACE           | Tox21          | ToxCast        |
| --------------------------------------------------------- | -------------- | -------------- | --------------- | -------------- | -------------- | -------------- |
| Fine-tuned $\text{GROVER}_{\text{Large}}$ provided by [1] | 93.9 $\pm$ 1.9 | 65.8 $\pm$ 2.3 | 94.4 $\pm$ 2.1  | 89.4 $\pm$ 2.9 | 83.1 $\pm$ 2.5 | 73.7 $\pm$ 1.0 |
| GROVER reproduced by KANO [2]                             | 86.8 $\pm$ 2.2 | 61.2 $\pm$ 2.5 | 70.3 $\pm$ 13.7 | 82.4 $\pm$ 3.6 | 80.3 $\pm$ 2.0 | 56.8 $\pm$ 3.4 |
| GROVER reproduced by GEM [3]                              | 69.5 $\pm$ 0.1 | 65.4 $\pm$ 0.1 | 76.2 $\pm$ 3.7  | 81.0 $\pm$ 1.4 | 73.5 $\pm$ 0.1 | 65.3 $\pm$ 0.5 |
| GROVER reproduced by Mole-BERT [4]                        | 66.9 $\pm$ 2.1 | 61.0 $\pm$ 1.0 | 77.7 $\pm$ 2.7  | 73.0 $\pm$ 3.3 | 73.6 $\pm$ 0.7 | 62.3 $\pm$ 0.6 |
| GROVER reproduced by us                                   | 86.2 $\pm$ 3.9 | 57.6 $\pm$ 1.6 | 74.7 $\pm$ 4.4  | 82.5 $\pm$ 4.4 | 76.9 $\pm$ 2.3 | 66.7 $\pm$ 2.6 |

**Performance on Regression Tasks:**

|                                                           | FreeSolv          | ESOL              | Lipophilicity     | QM7            | QM8               |
| --------------------------------------------------------- | ----------------- | ----------------- | ----------------- | -------------- | ----------------- |
| Fine-tuned $\text{GROVER}_{\text{Large}}$ provided by [1] | 1.544 $\pm$ 0.397 | 0.831 $\pm$ 0.120 | 0.560 $\pm$ 0.036 | 72.7 $\pm$ 3.8 | 0.013 $\pm$ 0.002 |
| GROVER reproduced by KANO [2]                             | 1.947 $\pm$ 0.615 | 1.423 $\pm$ 0.288 | 0.823 $\pm$ 0.010 | 91.3 $\pm$ 1.9 | 0.018 $\pm$ 0.001 |
| GROVER reproduced by GEM [3]                              | 2.272 $\pm$ 0.051 | 0.895 $\pm$ 0.017 | 0.823 $\pm$ 0.010 | 92.0 $\pm$ 0.9 | 0.022 $\pm$ 0.000 |
| GROVER reproduced by Mole-BERT [4]                        | -                 | -                 | -                 | -              | -                 |
| GROVER reproduced by us                                   | 2.445 $\pm$ 0.761 | 1.028 $\pm$ 0.145 | 0.890 $\pm$ 0.050 | 95.3 $\pm$ 5.6 | 0.020 $\pm$ 0.003 |

Reviewer comment (Experiments):
Number of seeds (3) in experiments seems pretty low

Author response:
Thanks for pointing out that the number of seeds (3) is low in experiments. While we recognize the value in your suggestion that using a greater number of seeds could yield more robust and convincing results, this experimental setup was chosen to maintain consistency with previous works [1, 2, 3], as suggested by MoleculeNet's paper [4]. We value your suggestion and will explore the use of additional seeds in our future research.

Reviewer comment (Evaluation):
Amount of relevant information in a knowledge graph likely varies significantly for a particular molecule. Eg aspirin is very well studied compared to other molecules. Any thoughts on when the knowledge graph information is useful or even harmful for a particular molecule property prediction task?

Author response:
This is an insightful question. In our work using GINE for K-GNN, both node (entity) and edge (relation) information in the knowledge graph can influence molecular representation. We recognize the importance of identifying which information in the KG is beneficial or detrimental and plan to conduct a systematic study on this in the future. To this end, we conducted a preliminary experiment to understand the impact of different relations on molecular property prediction tasks:

| Knowledge Graph                                              | BBBP                    | BACE                    |
| ------------------------------------------------------------ | ----------------------- | ----------------------- |
| MolKG                                                        | 94.5                    | 91.8                    |
| w/o relation "indication"                                    | 93.8 ($\downarrow$ 0.7) | 91.6 ($\downarrow$ 0.2) |
| w/o relations "xlogp3" and "xlogp3-aa"                       | 93.5 ($\downarrow$ 1.0) | 91.1 ($\downarrow$ 0.7) |
| w/o relations "tautomer_count" and "covalent_unit_count"     | 94.3 ($\downarrow$ 0.2) | 90.9 ($\downarrow$ 0.9) |
| w/o relations "neighbor_2d", "neighbor_3d", and "has_same_connectiviy" | 94.9 ($\uparrow$ 0.4)   | 91.3 ($\downarrow$ 0.5) |

For the BBBP task, removing "xlogp3" and "xlogp3-aa" led to a significant decrease in performance, suggesting their vital role in the model. In contrast, omitting "neighbor_2d", "neighbor_3d", and "has_same_connectivity" resulted in a slight improvement, indicating that these relations may not be crucial or could be harmful.

In the BACE task, the impact of removing "tautomer_count" and "covalent_unit_count" was the most pronounced (a decrease of 0.9%), highlighting their importance in this specific prediction task. The minimal impact of removing "indication" (a decrease of 0.2%) suggests that this relation might not be as critical for BACE as for BBBP.

These results provide initial insights into the task-specific importance of different relations in the knowledge graph. We plan to extend this preliminary analysis into a more comprehensive and systematic study in the future, which will allow us to more accurately determine the role and impact of various knowledge graph relations (or entities) on different molecule property prediction tasks.

Reviewer comment (Presentation):
In Table 2 and Table 3: would be clearer if there is some visual grouping of the methods that leverage molecule information only vs methods that additionally leverage knowledge graph information

Author response:
Thank you for your suggestion. We have split the non-KG-based and KG-based methods in ***Table 2*** and ***Table 3*** in our revised paper.

Reviewer comment (Experiments):
Why wasn’t the Ye et al, 2021 model benchmarked?

Author response:
Thanks for this important note! We have benchmarked it (named ***KGE_NFM***) in ***Table 2*** and ***Table 3*** in our revised paper.