Paper: Protein-ligand binding representation learning from fine-grained interactions (ICLR-2024)

Reviewer comment (Experiments):
I hold a significant concern regarding the potential data leakage in the evaluation process. It appears that the representations learned from BioLiP, which contains a subset of PDBind used for validating the results, may introduce a form of leakage, thus giving pretrained methods an advantage over other baseline techniques. To address this, it is advisable to consider re-executing the experiments with the exclusion of all overlapping data between BioLiP and the downstream tasks, including overlaps with datasets for binding affinity prediction (LBA 30%, LBA 60%, LEP), virtual screening (DUDe, AD), and docking (CASF-2016).

Author response:
Thank you for your valuable suggestions. We acknowledge the need to conduct additional experiments to address the missing aspects you highlighted. We intend to incorporate this supplementary content in the revised version of our paper to provide a more comprehensive and complementary analysis. Interestingly, the questions you raised align with inquiries posed by other reviewers, further emphasizing the importance of addressing these points in the revised manuscript.

>  Could you please kindly consider removing the potential leakage as advisable in the weakness discussion and release new results without leakage?(also asked by reviewer PEGf)

Indeed, the Biolip data exhibits overlap with downstream tasks such as LBA and CASF. However, **the impact of this overlap is dependent on the objectives of the downstream tasks**. In the case of the docking task, which aims to predict complex structures and is closely related to the pretraining target, it is imperative to eliminate the overlap to ensure a fair comparison. Conversely, for tasks involving binding affinity or virtual screening, where the goal is to predict binding affinity values or select positive protein-ligand pairs from a vast pool of negatives, the absence of such information as supervised signals means that the overlap may not significantly influence performance.
To validate our hypothesis, we eliminated the overlap between the pretraining data and the two test datasets, which include LBA (30%) and the docking task. Subsequently, we conducted re-pretraining on both versions of the pretraining data and reassessed performance on the corresponding downstream tasks. The results are presented in the tables below.

To validate our hypothesis, we eliminated the overlap between the pretraining data and the two test datasets, which include LBA (30%) and the docking task. Subsequently, we conducted re-pretraining on both versions of the pretraining data and reassessed performance on the corresponding downstream tasks. The results are presented in the tables below.

| Docking                  | 1.0 Å | 1.5 Å | 2.0 Å | 3.0 Å | 5.0 Å |
| ------------------------ | ----- | ----- | ----- | ----- | ----- |
| BindNet                  | 46.68 | 72.98 | 80.35 | 90.87 | 95.79 |
| BindNet (remove overlap,the result reported in paper) | 45.26 | 69.82 | 80.35 | 89.12 | 94.38 |


| Binding affinity prediction                  | RMSE  | Pearson | Spearman |
| ------------------------ | ----- | ------- | -------- |
| BindNet                  | 1.340 | 0.632   | 0.620    |
| BindNet (remove overlap) | 1.396 | 0.625   | 0.616    |

As depicted in the table above, the influence of overlap on the docking task performance is nonnegligible, leading us to report the results in the second row (remove overlap version) in our paper. Conversely, for the binding affinity task, the disparity between the non-overlap and overlap is relatively marginal, and this may be caused by our insufficient pre-training(only 6 epochs due to time limitation vs the orignial 30 epochs) . Consequently, we retained the overlap to maintain alignment with other studies [1], [2] that also utilized Biolip as a pretraining dataset.

[1]CoSP: Co-supervised pretraining of pocket and ligand

[2]General-purpose Pre-trained Model Towards Cross-domain Molecule Learning

Reviewer comment (Experiments):
I recommend including state-of-the-art protein representations like ESM-2 and ESM-1b (accessible at https://github.com/facebookresearch/esm), molecular representations such as MolFormer (available at https://github.com/IBM/molformer), and simple yet effective drug representations like Morgan fingerprints as baseline representations for a more comprehensive comparison.

Author response:
> Could you please kindly include the comparison to state-of-the-art protein representations like ESM-2 and ESM-1b and molecular representations such as MolFormer and Morgan fingerprints as baseline representations for a more comprehensive comparison?
(asked by all reviewers)

Thank you for your valuable suggestion. This particular question has been raised by all three reviewers. To answer this question, we have made significant modifications, specifically by replacing both the molecular encoder and pocket encoder. This change aims to demonstrate the adaptability of our proposed framework to a variety of pre-existing encoders.

On the molecular side, we have substituted the **molecular encoder** from UniMol to the pretrained encoder provided by **Frad**[1], which is a molecular pretraining model with a focus on quantum-related tasks. Frad's backbone is built on torchmd-net, which is a gnn and does not output pair representations like 'unimol.' Integrating Frad into our framework is seamless as we can simply use the concatenation of corresponding node-level embeddings as pairwise representations. We conducted pretraining of the interaction module based on Frad for 8 epochs, considering time limitations. We reevaluated the performance of LBA, and additionally, we present results for the setting where the interaction module is unpretrained, as shown in the table below.

On the protein side, we replaced the **pocket encoder** with the **ESM2**[2] encoder and altered the protein input for pocket structure to the protein sequence. Pretraining was also conducted based on the ESM2 encoder. Similarly, we include results for the setting where the interaction module is unpretrained based on the ESM2 encoder, as detailed in the table below.
| idx | Methods                                | LBA(30%) RMSE | LBA(30%) Pearson | LBA(30%) Spearman | LBA(60%) RMSE | LBA(60%) Pearson | LBA(60%) Spearman |
| --- | -------------------------------------- | ------------- | ----------------- | ------------------ | ------------- | ----------------- | ------------------ |
|     | **Replace molecular encoder to Frad**  |               |                   |                    |               |                   |                    |
| 0   | BindNet(Frad encoder) wo pretrain      | 1.417         | 0.562             | 0.543              | 1.62          | 0.582              | 0.571              |
| 1   | BindNet(Frad encoder) with pretrain    | 1.386         | 0.612              | 0.597              | 1.341         | 0.761              | 0.762               |
| 2   | BindNet(UniMol) wo pretrain            | 1.434         | 0.565             | 0.541               | 1.357         | 0.753             | 0.753               |
| 3   | BindNet(UniMol encoder) with pretrain  | 1.341          | 0.632             | 0.622               | 1.232          | 0.793             | 0.788              |
|     | **Replace pocket encoder**             |               |                   |                    |               |                   |                    |
| 4   | BindNet(esm2 encoder) wo pretrain      | 1.561         | 0.444             | 0.445              |               |                   |                    |
| 5   | BindNet(esm2 encoder) with pretrain    | 1.492         | 0.522             | 0.550               |               |                   |                    |

Upon examining the table above, it is evident that replacing the molecular or pocket encoder with Frad or ESM consistently results in superior performance for the pretrained version of Bindnet compared to the unpretrained version (idx0 vs idx1, idx4 vs idx5). Notably, Bindnet based on Frad continues to achieve SOTA results among other baselines, highlighting the flexibility of our framework.


It is essential to draw attention to the ESM2 encoder, which has nearly 3 billion parameters. Due to limitations in computational resources and time constraints, we were compelled to randomly select residues when the sequence number exceeded 500, and the model was trained for only 4 epochs.  Such constraints may have harmful effects on its overall performance. These constraints may have deleterious effects on its overall performance. We firmly believe that alleviating these constraints could significantly enhance its performance.

[1] Fractional Denoising for 3D Molecular Pre-training. ICML 2023

[2] Language models of protein sequences at the scale of evolution enable accurate structure prediction.

Reviewer comment (Reproducibility):
Another significant concern is the reproducibility of the work. The absence of available code, missing details regarding hyperparameter settings, and the lack of transparency in releasing experimental settings, including dataset splits and configurations for downstream tasks, hinder the reproducibility of the research. Addressing these issues would greatly enhance the credibility and transparency of the work.

Author response:
> Could you please kindly work-out on the reproducibility as advisable the weakness discussion?

We have reorganized our code and uploaded it to an anonymous Git repository. Additionally, we have provided a comprehensive README file that includes both the pretraining and fine-tuning  and data processing scripts, complete with the hyperparameters utilized during training. You can access the code repository through this link:https://anonymous.4open.science/r/BindNet-0060/README.md . Furthermore, we have included a pretrained model, available at the provided link(https://drive.google.com/file/d/1k40YI_Et472S3lG1CMs5-of4-Gf_op8_/view?usp=sharing), we also upload the processed data(https://drive.google.com/drive/folders/1NfXpsbbsR7FNIEoU88EA04Jmgnio0q_O?usp=sharing) . If you have any further questions regarding reproducibility or if there are specific details you would like us to expand upon, please feel free to ask. We welcome any inquiries and are eager to provide additional information as needed.

Reviewer comment (Experiments):
The authors claim that it is a "self-supervised" method, but actually the pocket-ligand complex data are still required for pre-training. So, the data volume is limited by this requirement. For example, only 458k BioLip data is used for pre-training, and thus earlier stage pretraining like UniMol is required.

Author response:
The Biolip data is also taken as pre-training dataset for other self-supervised methods[1] [2]. While it is recognized that the volume of complex data is relatively limited compared to single molecule or protein data, we notice that a recent method called ProFSA[3] introduces a novel approach to generate simulated complex data utilizing the existing protein structures, which contains approximately **5 million** data items. To validate the adaptability of our method to such "pseudo" complex data, we conducted pretraining and assessed its performance on the LBA 30% task. The results are presented in the table below.

| Methods                                      | RMSE  | Pearson | Spearman |
| -------------------------------------------- | ----- | ------- | -------- |
| GeoSSL                                       | 1.451 | 0.577   | 0.572    |
| BindNet (pretrained on ProFSA data, 4 epochs) | **1.337** | 0.614   | 0.591    |
| BindNet (pretrained on biolip, 30 epochs)     | 1.34  | **0.632**   | **0.62**     |

As depicted in the table above, BindNet pretrained on the ProFSA data still achieves state-of-the-art (SOTA) results when compared to other baseline models. It is worth noting that the pretraining process was limited to just 4 epochs due to time constraints during the rebuttal phase and the big scale of pretraining data. Despite this limitation, the model has the potential to further improve in performance with more training iterations.

[1] CoSP: Co-supervised pretraining of pocket and ligand

[2] General-purpose Pre-trained Model Towards Cross-domain Molecule Learning

[3] Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment

Reviewer comment (Experiments):
Another concern is about label leakage. Is there any overlap between the BioLip and downstream datasets (LBA, DUD-E, PDBBind, etc.)?

Author response:
Indeed, the Biolip data exhibits overlap with downstream tasks such as LBA and CASF. However, **the impact of this overlap is dependent on the objectives of the downstream tasks**. In the case of the docking task, which aims to predict complex structures and is closely related to the pretraining target, it is imperative to eliminate the overlap to ensure a fair comparison. Conversely, for tasks involving binding affinity or virtual screening, where the goal is to predict binding affinity values or select positive protein-ligand pairs from a vast pool of negatives, the absence of such information as supervised signals means that the overlap may not significantly influence performance.
To validate our hypothesis, we eliminated the overlap between the pretraining data and the two test datasets, which include LBA (30%) and the docking task. Subsequently, we conducted re-pretraining on both versions of the pretraining data and reassessed performance on the corresponding downstream tasks. The results are presented in the tables below.

To validate our hypothesis, we eliminated the overlap between the pretraining data and the two test datasets, which include LBA (30%) and the docking task. Subsequently, we conducted re-pretraining on both versions of the pretraining data and reassessed performance on the corresponding downstream tasks. The results are presented in the tables below.

| Docking                  | 1.0 Å | 1.5 Å | 2.0 Å | 3.0 Å | 5.0 Å |
| ------------------------ | ----- | ----- | ----- | ----- | ----- |
| BindNet                  | 46.68 | 72.98 | 80.35 | 90.87 | 95.79 |
| BindNet (remove overlap,the result reported in paper) | 45.26 | 69.82 | 80.35 | 89.12 | 94.38 |


| Binding affinity prediction                  | RMSE  | Pearson | Spearman |
| ------------------------ | ----- | ------- | -------- |
| BindNet                  | 1.340 | 0.632   | 0.620    |
| BindNet (remove overlap) | 1.396 | 0.625   | 0.616    |

As depicted in the table above, the influence of overlap on the docking task performance is nonnegligible, leading us to report the results in the second row (remove overlap version) in our paper. Conversely, for the binding affinity task, the disparity between the non-overlap and overlap is relatively marginal, and this may be caused by our insufficient pre-training(only 6 epochs due to time limitation vs the orignial 30 epochs) . Consequently, we retained the overlap to maintain alignment with other studies [1], [2] that also utilized Biolip as a pretraining dataset.

[1]CoSP: Co-supervised pretraining of pocket and ligand

[2]General-purpose Pre-trained Model Towards Cross-domain Molecule Learning

Reviewer comment (Experiments):
The authors claim that "a variety of pre-existing encoders for pockets and ligands can be utilized.", but only Uni-Mol used in experiments.

Author response:
Thank you for your valuable suggestion. This particular question has been raised by all three reviewers. To answer this question, we have made significant modifications, specifically by replacing both the molecular encoder and pocket encoder. This change aims to demonstrate the adaptability of our proposed framework to a variety of pre-existing encoders.

On the molecular side, we have substituted the **molecular encoder** from UniMol to the pretrained encoder provided by **Frad**[1], which is a molecular pretraining model with a focus on quantum-related tasks. Frad's backbone is built on torchmd-net, which is a gnn and does not output pair representations like 'unimol.' Integrating Frad into our framework is seamless as we can simply use the concatenation of corresponding node-level embeddings as pairwise representations. We conducted pretraining of the interaction module based on Frad for 8 epochs, considering time limitations. We reevaluated the performance of LBA, and additionally, we present results for the setting where the interaction module is unpretrained, as shown in the table below.

On the protein side, we replaced the **pocket encoder** with the **ESM2**[2] encoder and altered the protein input for pocket structure to the protein sequence. Pretraining was also conducted based on the ESM2 encoder. Similarly, we include results for the setting where the interaction module is unpretrained based on the ESM2 encoder, as detailed in the table below.
| idx | Methods                                | LBA(30%) RMSE | LBA(30%) Pearson | LBA(30%) Spearman | LBA(60%) RMSE | LBA(60%) Pearson | LBA(60%) Spearman |
| --- | -------------------------------------- | ------------- | ----------------- | ------------------ | ------------- | ----------------- | ------------------ |
|     | **Replace molecular encoder to Frad**  |               |                   |                    |               |                   |                    |
| 0   | BindNet(Frad encoder) wo pretrain      | 1.417         | 0.562             | 0.543              | 1.62          | 0.582              | 0.571              |
| 1   | BindNet(Frad encoder) with pretrain    | 1.386         | 0.612              | 0.597              | 1.341         | 0.761              | 0.762               |
| 2   | BindNet(UniMol) wo pretrain            | 1.434         | 0.565             | 0.541               | 1.357         | 0.753             | 0.753               |
| 3   | BindNet(UniMol encoder) with pretrain  | 1.341          | 0.632             | 0.622               | 1.232          | 0.793             | 0.788              |
|     | **Replace pocket encoder**             |               |                   |                    |               |                   |                    |
| 4   | BindNet(esm2 encoder) wo pretrain      | 1.561         | 0.444             | 0.445              |               |                   |                    |
| 5   | BindNet(esm2 encoder) with pretrain    | 1.492         | 0.522             | 0.550               |               |                   |                    |

Upon examining the table above, it is evident that replacing the molecular or pocket encoder with Frad or ESM consistently results in superior performance for the pretrained version of Bindnet compared to the unpretrained version (idx0 vs idx1, idx4 vs idx5). Notably, Bindnet based on Frad continues to achieve SOTA results among other baselines, highlighting the flexibility of our framework.


It is essential to draw attention to the ESM2 encoder, which has nearly 3 billion parameters. Due to limitations in computational resources and time constraints, we were compelled to randomly select residues when the sequence number exceeded 500, and the model was trained for only 4 epochs.  Such constraints may have harmful effects on its overall performance. These constraints may have deleterious effects on its overall performance. We firmly believe that alleviating these constraints could significantly enhance its performance.

[1] Fractional Denoising for 3D Molecular Pre-training. ICML 2023

[2] Language models of protein sequences at the scale of evolution enable accurate structure prediction.

Reviewer comment (Reproducibility):
What is the "N-layer 3D-invariant Transformer" used in the paper? The authors should provide more details about this part.

Author response:
This question was also raised by reviewer kw5e. We apologize for any confusion, and we understand the need for additional clarification. We will provide a more detailed explanation in our revised paper.

The block in the N-transformer-based interaction module remains consistent with the previously pre-trained molecular encoder. To enhance clarity, we have incorporated a figure within the link https://anonymous.4open.science/r/BindNet_rebuttal-88C1/interaction_module.jpeg, illustrating the module structure to facilitate a better understanding.

As depicted in the figure, we denote the attention weight of the last layer for the pocket encoder as $\mathbf{h}^{(0)} _{PP}$ and the corresponding node-level embedding as $\mathbf{h}^{(0)} _{P}$. Similarly, the molecular attention weight is represented as $\mathbf{h}^{(0)} _{LL}$, and $\mathbf{h}^{(0)} _{L}$ signifies the atom embedding of the molecule.
The interaction module receives both node-level embeddings and pairwise-level embeddings, the latter serving as the attention bias added to the attention weight. This fusion combines representations from the pocket and ligand. Specifically, we concatenate the pocket and molecular embeddings to form the atom-level input. For the pairwise-level embeddings, denoted as $\mathbf{h}^{(0)} _{PL}$, we initialize it with zero values and subsequently populate it with the values from $\mathbf{h}^{(0)} _{PP}$ and $\mathbf{h}^{(0)} _{LL}$ at the corresponding positions, as follows: $\mathbf{h}^{(0)} _{PL}[:L_p, :L_p]=\mathbf{h}^{(0)} _{PP}$ and $\mathbf{h}^{(0)} _{PL}[-L_m:,-L_m:]=\mathbf{h}^{(0)} _{PP}$. Where the $L_p$ and $L_m$ denotes the atom numbers of pocket and ligand respectively. 
After forwarding through N transformer layers, we obtain the final node-level embeddings $\mathbf{h}^{(N)} _{P}$ and $\mathbf{h}^{(N)} _{L}$, utilized for Mask Ligand Reconstruction, as well as the pairwise embedding $\mathbf{h}^{(N)} _{PL}$, employed for Atomic Pairwise Distance Map Prediction task.

Reviewer comment (Experiments):
How to choose the mask ratio in MLR? The 0.8 seems to be a very high value.

Author response:
We selected a higher mask ratio with the intention of relying solely on the pocket to reconstruct the molecular ligand that binds to it. This approach is reminiscent of the pocket-based generation setting[1][2]. However, our Masked Language Reconstruction (MLR) operates at the embedding level, and a larger mask ratio may prove beneficial in enabling molecular or pocket embeddings to encode more interaction-related information. To validate this hypothesis, we set the mask ratio to 0.3 and retrained Bindnet. The results demonstrate that a larger mask ratio indeed improves performance on LBA, as illustrated in the table below.

| Methods                  | RMSE  | Pearson | Spearman |
| ------------------------ | ----- | ------- | -------- |
| BindNet (mask ratio=0.3) | 1.370 | 0.619   | 0.608    |
| BindNet (mask ratio=0.8) | 1.340 | 0.632   | 0.620    |

While the table suggests that a larger mask ratio is more beneficial, it's important to note that determining **the optimal parameter may require further experimentation in a more fine-grained manner**. This process involves time-consuming pretraining, and we plan to conduct additional experiments in the future to identify the best candidate due to time limitations.

[1]:A 3D Generative Model for Structure-Based Drug Design

[2]:Pocket2Mol: Efficient Molecular Sampling Based on 3D Protein Pockets

Reviewer comment (Evaluation):
As the major argument of the paper is regarding the capturing of the **interactions** through a Transformer-based interaction module, analysis on the interactions and its consequence should have been provided (e.g., specific types of interactions captured by this new module), which are missing in the paper.

Author response:
Thank you for your suggestions. We have selected four pocket-ligand complexes as examples to demonstrate the information captured by our interaction module in the following links(https://anonymous.4open.science/r/BindNet_rebuttal-88C1/sch1.jpeg; https://anonymous.4open.science/r/BindNet_rebuttal-88C1/sch2.jpeg; https://anonymous.4open.science/r/BindNet_rebuttal-88C1/sch3.jpeg; https://anonymous.4open.science/r/BindNet_rebuttal-88C1/sch4.jpeg).

The left part of figure illustrates interactions computed using **Ligand Interaction Diagram Panel of Schrödinger**[Schrödinger, LLC, NY, USA], depicting three common interactions: **hydrogen interaction** represented by the magenta line, **π-π interaction** denoted by the green line, and **salt bridge** visualized as a gradient color line transitioning from red to blue.

The right part of figure represents the attention heatmap between pocket and ligand from BindNet. The original attention in BindNet is atom-atom level, which is too long to present here due to the significant number of atoms in the pocket, so we aggregate the attention of same residue in pocket to form a atom-residue level attention heatmap for a clearer visual representation.
From these examples, we can easily find that **BindNet pays more attention to the residue which can form non-covalent interactions with ligand**, sometimes even the specific ligand atom - residue pair. This implys BindNet might have the ability to capture the interaction force information between the pocket and ligand, which is crucial for the task such as ligand binding affinity prediction and virtural screening.

Reviewer comment (Writing):
While it contends that they present a Transformer-based interaction module, the explanation provided is notably insufficient.

Author response:
Thank you for your reminder. In the revised version, we will include a more detailed explanation. The block in the N-transformer-based interaction module remains consistent with the previously pre-trained molecular encoder. To enhance clarity, we have incorporated a figure within the link https://anonymous.4open.science/r/BindNet_rebuttal-88C1/interaction_module.jpeg, illustrating the module structure to facilitate a better understanding.

As depicted in the figure, we denote the attention weight of the last layer for the pocket encoder as $\mathbf{h}^{(0)} _{PP}$ and the corresponding node-level embedding as $\mathbf{h}^{(0)} _{P}$. Similarly, the molecular attention weight is represented as $\mathbf{h}^{(0)} _{LL}$, and $\mathbf{h}^{(0)} _{L}$ signifies the atom embedding of the molecule.
The interaction module receives both node-level embeddings and pairwise-level embeddings, the latter serving as the attention bias added to the attention weight. This fusion combines representations from the pocket and ligand. Specifically, we concatenate the pocket and molecular embeddings to form the atom-level input. For the pairwise-level embeddings, denoted as $\mathbf{h}^{(0)} _{PL}$, we initialize it with zero values and subsequently populate it with the values from $\mathbf{h}^{(0)} _{PP}$ and $\mathbf{h}^{(0)} _{LL}$ at the corresponding positions, as follows: $\mathbf{h}^{(0)} _{PL}[:L_p, :L_p]=\mathbf{h}^{(0)} _{PP}$ and $\mathbf{h}^{(0)} _{PL}[-L_m:,-L_m:]=\mathbf{h}^{(0)} _{PP}$. Where the $L_p$ and $L_m$ denotes the atom numbers of pocket and ligand respectively. 
After forwarding through N transformer layers, we obtain the final node-level embeddings $\mathbf{h}^{(N)} _{P}$ and $\mathbf{h}^{(N)} _{L}$, utilized for Mask Ligand Reconstruction, as well as the pairwise embedding $\mathbf{h}^{(N)} _{PL}$, employed for Atomic Pairwise Distance Map Prediction task.

Reviewer comment (Novelty):
The authors argue that the framework is flexible, allowing for the integration of various pre-existing encoders for proteins and ligands. However, for the distance matrix associated with the atomic pairwise distance map prediction, it appears to necessitate a pair representation in the Uni-Mol. In other words, the architecture seems to be tailored specifically for Uni-Mol.

Author response:
Thank you for your valuable suggestion. This particular question has been raised by all three reviewers. To answer this question, we have made significant modifications, specifically by replacing both the molecular encoder and pocket encoder. This change aims to demonstrate the adaptability of our proposed framework to a variety of pre-existing encoders.

On the molecular side, we have substituted the **molecular encoder** from UniMol to the pretrained encoder provided by **Frad**[1], which is a molecular pretraining model with a focus on quantum-related tasks. Frad's backbone is built on torchmd-net, which is a gnn and does not output pair representations like 'unimol.' Integrating Frad into our framework is seamless as we can simply use the concatenation of corresponding node-level embeddings as pairwise representations. We conducted pretraining of the interaction module based on Frad for 8 epochs, considering time limitations. We reevaluated the performance of LBA, and additionally, we present results for the setting where the interaction module is unpretrained, as shown in the table below.

On the protein side, we replaced the **pocket encoder** with the **ESM2**[2] encoder and altered the protein input for pocket structure to the protein sequence. Pretraining was also conducted based on the ESM2 encoder. Similarly, we include results for the setting where the interaction module is unpretrained based on the ESM2 encoder, as detailed in the table below.
| idx | Methods                                | LBA(30%) RMSE | LBA(30%) Pearson | LBA(30%) Spearman | LBA(60%) RMSE | LBA(60%) Pearson | LBA(60%) Spearman |
| --- | -------------------------------------- | ------------- | ----------------- | ------------------ | ------------- | ----------------- | ------------------ |
|     | **Replace molecular encoder to Frad**  |               |                   |                    |               |                   |                    |
| 0   | BindNet(Frad encoder) wo pretrain      | 1.417         | 0.562             | 0.543              | 1.62          | 0.582              | 0.571              |
| 1   | BindNet(Frad encoder) with pretrain    | 1.386         | 0.612              | 0.597              | 1.341         | 0.761              | 0.762               |
| 2   | BindNet(UniMol) wo pretrain            | 1.434         | 0.565             | 0.541               | 1.357         | 0.753             | 0.753               |
| 3   | BindNet(UniMol encoder) with pretrain  | 1.341          | 0.632             | 0.622               | 1.232          | 0.793             | 0.788              |
|     | **Replace pocket encoder**             |               |                   |                    |               |                   |                    |
| 4   | BindNet(esm2 encoder) wo pretrain      | 1.561         | 0.444             | 0.445              |               |                   |                    |
| 5   | BindNet(esm2 encoder) with pretrain    | 1.492         | 0.522             | 0.550               |               |                   |                    |

Upon examining the table above, it is evident that replacing the molecular or pocket encoder with Frad or ESM consistently results in superior performance for the pretrained version of Bindnet compared to the unpretrained version (idx0 vs idx1, idx4 vs idx5). Notably, Bindnet based on Frad continues to achieve SOTA results among other baselines, highlighting the flexibility of our framework.


It is essential to draw attention to the ESM2 encoder, which has nearly 3 billion parameters. Due to limitations in computational resources and time constraints, we were compelled to randomly select residues when the sequence number exceeded 500, and the model was trained for only 4 epochs.  Such constraints may have harmful effects on its overall performance. These constraints may have deleterious effects on its overall performance. We firmly believe that alleviating these constraints could significantly enhance its performance.

[1] Fractional Denoising for 3D Molecular Pre-training. ICML 2023

[2] Language models of protein sequences at the scale of evolution enable accurate structure prediction.

Reviewer comment (Experiments):
If the system is indeed flexible, there should be experimental evidence presented with other encoders.

Author response:
Thank you for your valuable suggestion. This particular question has been raised by all three reviewers. To answer this question, we have made significant modifications, specifically by replacing both the molecular encoder and pocket encoder. This change aims to demonstrate the adaptability of our proposed framework to a variety of pre-existing encoders.

On the molecular side, we have substituted the **molecular encoder** from UniMol to the pretrained encoder provided by **Frad**[1], which is a molecular pretraining model with a focus on quantum-related tasks. Frad's backbone is built on torchmd-net, which is a gnn and does not output pair representations like 'unimol.' Integrating Frad into our framework is seamless as we can simply use the concatenation of corresponding node-level embeddings as pairwise representations. We conducted pretraining of the interaction module based on Frad for 8 epochs, considering time limitations. We reevaluated the performance of LBA, and additionally, we present results for the setting where the interaction module is unpretrained, as shown in the table below.

On the protein side, we replaced the **pocket encoder** with the **ESM2**[2] encoder and altered the protein input for pocket structure to the protein sequence. Pretraining was also conducted based on the ESM2 encoder. Similarly, we include results for the setting where the interaction module is unpretrained based on the ESM2 encoder, as detailed in the table below.
| idx | Methods                                | LBA(30%) RMSE | LBA(30%) Pearson | LBA(30%) Spearman | LBA(60%) RMSE | LBA(60%) Pearson | LBA(60%) Spearman |
| --- | -------------------------------------- | ------------- | ----------------- | ------------------ | ------------- | ----------------- | ------------------ |
|     | **Replace molecular encoder to Frad**  |               |                   |                    |               |                   |                    |
| 0   | BindNet(Frad encoder) wo pretrain      | 1.417         | 0.562             | 0.543              | 1.62          | 0.582              | 0.571              |
| 1   | BindNet(Frad encoder) with pretrain    | 1.386         | 0.612              | 0.597              | 1.341         | 0.761              | 0.762               |
| 2   | BindNet(UniMol) wo pretrain            | 1.434         | 0.565             | 0.541               | 1.357         | 0.753             | 0.753               |
| 3   | BindNet(UniMol encoder) with pretrain  | 1.341          | 0.632             | 0.622               | 1.232          | 0.793             | 0.788              |
|     | **Replace pocket encoder**             |               |                   |                    |               |                   |                    |
| 4   | BindNet(esm2 encoder) wo pretrain      | 1.561         | 0.444             | 0.445              |               |                   |                    |
| 5   | BindNet(esm2 encoder) with pretrain    | 1.492         | 0.522             | 0.550               |               |                   |                    |

Upon examining the table above, it is evident that replacing the molecular or pocket encoder with Frad or ESM consistently results in superior performance for the pretrained version of Bindnet compared to the unpretrained version (idx0 vs idx1, idx4 vs idx5). Notably, Bindnet based on Frad continues to achieve SOTA results among other baselines, highlighting the flexibility of our framework.


It is essential to draw attention to the ESM2 encoder, which has nearly 3 billion parameters. Due to limitations in computational resources and time constraints, we were compelled to randomly select residues when the sequence number exceeded 500, and the model was trained for only 4 epochs.  Such constraints may have harmful effects on its overall performance. These constraints may have deleterious effects on its overall performance. We firmly believe that alleviating these constraints could significantly enhance its performance.

[1] Fractional Denoising for 3D Molecular Pre-training. ICML 2023

[2] Language models of protein sequences at the scale of evolution enable accurate structure prediction.

Reviewer comment (Experiments):
Did the other baseline models use the primary state for ligands in experiments?

Author response:
For **binding affinity tasks** (LBA and LEP, Section 4.1), where complex data is provided as input, all baseline models utilize the torsion state (molecular structure in the binding complex) of ligands. In contrast, for **virtual screening tasks** (DUD-E, DUD-AD, Section 4.3) and the **docking task** (Section 4.2), the complex structure is either unavailable or can only be used as a label for prediction. In these cases, all methods use the primary state of ligands as input.

Notably, for the binding affinity tasks, unlike other structure-based methods, BindNet can discern the dependency on torsion state. Whether considering the torsion state or primary state, neither influences the state-of-the-art (SOTA) performance of BindNet, as confirmed by ablation studies in Section 5.2.