Paper: Self-supervised Pocket Pretraining via Protein Fragment-Surroundings Alignment (ICLR-2024)

Reviewer comment (Reproducibility):
The evidence for the construction of the pseudo-ligand is not clear.

Author response:
## Response to evidence for the construction of the pseudo-ligand and using peptides to replace small molecules 

As we discussed in the paper, pseudo-ligands share similar sizes as real ligands, and they also make similar non-covalent interactions with the pocket. To further support our proposal, we would like to cite another paper **"A defined structural unit enables de novo design of small-molecule–binding proteins"**[1], published in Science 2020, which supports that **intra-protein interactions are similar to protein-ligand interactions.** That is the reason we use peptides to **represent** small molecules to let the pocket encoder learn the interaction information. 

Also, we'd like to clarify that our intention is not to **"replace"** small molecules with peptides. It's a pretraining framework that leverages abundant protein-only data and uses peptides to **simulate** pocket-small molecule interactions for enhanced pocket representations. Following this pretraining phase, **the model can be finetuned with datasets that include real small molecules, ensuring its effectiveness in practical applications**.

To demonstrate the validity of constructing pseudo-ligands, we provide visualizations in **Appendix F** showing that interactions between pockets and our pseudo-ligands are similar to those with true ligands. These figures reveal shared interaction types like **hydrogen bonding**, **$\pi-\pi$ stacking**, and **salt bridge**, depicted with color-coded dashed lines for each interaction type. This should help clarify the rationale and evidence supporting our approach.

[1] Polizzi, Nicholas F., and William F. DeGrado. "A defined structural unit enables de novo design of small-molecule–binding proteins." Science 369, no. 6508 (2020)

Reviewer comment (Experiments):
Why do the authors choose peptides to replace small molecules, and is this choice reliable? Have the authors considered other potential ways to further close the gap between pseudo-ligands and real ligands, either through data processing strategies or by fine-tuning on downstream tasks?

Author response:
## Response to evidence for the construction of the pseudo-ligand and using peptides to replace small molecules 

As we discussed in the paper, pseudo-ligands share similar sizes as real ligands, and they also make similar non-covalent interactions with the pocket. To further support our proposal, we would like to cite another paper **"A defined structural unit enables de novo design of small-molecule–binding proteins"**[1], published in Science 2020, which supports that **intra-protein interactions are similar to protein-ligand interactions.** That is the reason we use peptides to **represent** small molecules to let the pocket encoder learn the interaction information. 

Also, we'd like to clarify that our intention is not to **"replace"** small molecules with peptides. It's a pretraining framework that leverages abundant protein-only data and uses peptides to **simulate** pocket-small molecule interactions for enhanced pocket representations. Following this pretraining phase, **the model can be finetuned with datasets that include real small molecules, ensuring its effectiveness in practical applications**.

To demonstrate the validity of constructing pseudo-ligands, we provide visualizations in **Appendix F** showing that interactions between pockets and our pseudo-ligands are similar to those with true ligands. These figures reveal shared interaction types like **hydrogen bonding**, **$\pi-\pi$ stacking**, and **salt bridge**, depicted with color-coded dashed lines for each interaction type. This should help clarify the rationale and evidence supporting our approach.

[1] Polizzi, Nicholas F., and William F. DeGrado. "A defined structural unit enables de novo design of small-molecule–binding proteins." Science 369, no. 6508 (2020)

Reviewer comment (Writing):
Section 3.1, second paragraph, line 4, what do the N Terminal and C Terminal refer to?

Author response:
## Response to the question on N Terminal and C Terminal

When a peptide bond is formed (green), one amine group and one carboxy group are left untouched. Therefore, in a linear polypeptide, there would always be a free amine group and a free carboxy group, and they are called the N terminal(blue) and the C terminal(red) respectively in biochemistry. You can find the visualization in **Appendix H**.

Reviewer comment (Experiments):
Why fixed the molecule encoder in contrastive learning, i.e., the encoder that encodes the pseudo-ligand.

Author response:
## Response to fixing the molecule encoder in contrastive learning

From an **effectiveness** standpoint, as you mentioned before, there are unavoidable discrepancies between pseudo and true ligands. If we didn't fix our molecule encoder during pretraining, then the conditions of Theorem 3.1 might not be met. The ablation results presented in the previous response clearly indicate a performance decline when the molecular encoder is not fixed during contrastive learning, particularly in the context of pocket matching. This supports our assertion that the pocket encoder struggles to acquire chemical knowledge without the stability provided by a fixed molecular encoder.

From an **efficiency** standpoint, a fixed molecule encoder also offers significant advantages. It reduces memory usage and accelerates training speed because all molecule embeddings can be pre-computed before training. This eliminates the need for feed-forward and back-propagation operations of the molecule encoder during training, leading to efficiency gains. Additionally, adapting to different molecule encoders becomes more straightforward when they are not co-trained during the pretraining process. This approach allows for more flexibility and ease in integrating various encoding methodologies **even if the molecule encoder is not accessible for training**. That is to say, our approach can even be used with undifferentiable molecular representations.

Reviewer comment (Experiments):
Could ProFSA be extended to other tasks like protein-protein interaction prediction? How might the pipeline and contrastive approach need to be adapted?

Author response:
## Response to extending to other tasks like protein-protein interaction prediction

Thank you for pointing out the potential application of our framework to protein-protein interaction (PPI) tasks. We have to point out that the interactions between proteins and proteins are far different from proteins and molecules. Specifically, the interaction areas are usually much larger, but shallower, and involved functional groups are less diverse due to limited types of natural amino acids. Therefore, several changes should be made to extend our ProFSA method to PPI tasks:

1. As for data creation, unlike the localized and specific nature of pocket-ligand interactions, protein-protein interactions are **more global and involve broader protein features**. This requires a shift in the data creation process to split entire protein structures into domains instead of short fragments and to capture potential interactions between them. 

2. The interaction interface between protein and protein is much larger. We might need to change from atomic-level to residue-level representations to efficiently capture interaction information.

3. For contrastive learning training, ProFSA uses a fixed pretrained molecular encoder to cope with various small molecules and their atomic representation. However, for protein-protein interaction, it is not necessary as we could only model 20 types of amino acids at the residue level.

Despite several changes that need to be done for PPI tasks, we find that we could divide protein complexes into local interactions. When the complex structures and monomer structures are provided, we have achieved a reasonable result on a flexible PPI affinity prediction benchmark with our current model in a zero-shot way. 

| Method            | Spearman $\uparrow$  |
|-------------------|----------------------|
| SchNet            | 0.072 ± 0.021        |
| DimeNet++         | 0.171 ± 0.054        |
| EGNN              | 0.080 ± 0.038        |
| TorchMD           | 0.117 ± 0.008        |
| GET               | **0.363 ± 0.017**    |
| ProFSA(zero-shot) | _0.248_              |

Surprisingly, as a **zero-shot** method, ProFSA is able to outperform other **supervised learning** models except for GET, a newly proposed unified model. This demonstrates our approach is also able to capture protein-protein interaction information despite it being designed for protein-ligand interaction. 

A detailed explanation of this experiment is shown in **Appendix C6**.

Reviewer comment (Experiments):
One weakness is that the proposed method is only evaluated on limited tasks.

Author response:
In our paper, the primary objective is to assess the effectiveness of our pretraining method and the quality of the trained pocket representations. To achieve this, we focused on two types of downstream tasks: pocket-only tasks (pocket property prediction and pocket matching) and pocket-ligand interaction tasks (ligand binding affinity prediction).

We appreciate the suggestion to apply our method to the protein-ligand binding pose prediction task. However, it's important to note that our current evaluation framework is designed to specifically assess **pocket representations**. For the tasks we chose, the architecture is straightforward: either a **zero-shot** evaluation or a **simple MLP** for mapping embeddings to predictions. In contrast, protein-ligand binding pose prediction often involves **complex methodologies**. For instance, state-of-the-art methods like DiffDock[1] require training a diffusion generative model and a separate confidence model, while other approaches like EDM-Dock[2], rely on reconstructing ligand conformations from predicted distance maps. These methods are not end-to-end and do not directly align with our objective of evaluating pretrained pocket representations.

Nevertheless, we recognize the potential of our method in enhancing existing binding pose prediction techniques. To integrate our approach, we would need to modify our framework. Using the same data creation strategy, we could train a binding pose prediction model with our preprocessed data, which could then be further fine-tuned using real pocket-ligand pair data from sources like PDBbind. Additionally, our data could be used to train a side-chain packing model, allowing for side-chain flexibility during docking. Thank you again for your advice, and we will leave protein-ligand binding pose prediction as a separate future work for our method.

In response to the need for evaluating our method on a broader range of tasks, we have extended our analysis to include two additional downstream tasks: LEP (Ligand Efficacy Prediction), and PPA (Protein-Protein Affinity Prediction).

result for LEP:

| Method      | AUROC $\uparrow$ | AUPRC $\uparrow$ |
|-------------|--------------|--------------|
| ATOM3D-GNN  | 0.681        | 0.598        |
| GeoSSL      | 0.776±0.03   | 0.694±0.06   |
| Uni-Mol     | 0.782±0.02   | 0.695±0.07   |
| ProFSA      | 0.840±0.04   | 0.806±0.04   |

GeoSSL and Uni-Mol, both pretraining methods, yield comparable results. However, ProFSA outperforms these methods, demonstrating the advantage of our pocket pretraining approach.

result for PPA:

| Method            | Spearman $\uparrow$  |
|-------------------|----------------------|
| SchNet            | 0.072 ± 0.021        |
| DimeNet++         | 0.171 ± 0.054        |
| EGNN              | 0.080 ± 0.038        |
| TorchMD           | 0.117 ± 0.008        |
| GET               | **0.363 ± 0.017**    |
| ProFSA(zero shot) | _0.248_              |

As a **zero-shot** method, ProFSA is able to outperform other **supervised learning** models except for GET, a newly proposed unified model. This demonstrates our approach is able to capture protein-protein interaction information despite it being designed for protein-ligand interaction modeling.

You can find detailed experiment settings and results in **Appendix C5 and C6**.

[1] Corso et al., "Diffdock: Diffusion steps, twists, and turns for molecular docking.", ICLR 2023.
[2] Masters et al., "Deep learning model for efficient protein–ligand docking with implicit side-chain flexibility." Journal of Chemical Information and Modeling 63, no. 6 (2023).

Thank you for your insightful feedback. In response to your concerns regarding the scope of our tasks, we have expanded our research in the revised paper. We now include additional tasks such as **ligand efficiency prediction** and **protein-protein affinity prediction**, detailed in **Appendix C5 and C6**. These tasks were incorporated to enhance the robustness of our method. We acknowledge that they are currently in the appendix due to space constraints, but we are open to relocating them to the main text if it would be beneficial.

We greatly appreciate the time and effort you have invested in reviewing our work. Your feedback has been invaluable in refining our research. Please let us know if there are any other aspects of our paper that you would like us to address or clarify. We look forward to your further feedback.

Reviewer comment (Experiments):
Will the proposed method work on other tasks, such as protein-ligand binding pose prediction?

Author response:
We appreciate the suggestion to apply our method to the protein-ligand binding pose prediction task. However, it's important to note that our current evaluation framework is designed to specifically assess **pocket representations**. For the tasks we chose, the architecture is straightforward: either a **zero-shot** evaluation or a **simple MLP** for mapping embeddings to predictions. In contrast, protein-ligand binding pose prediction often involves **complex methodologies**. For instance, state-of-the-art methods like DiffDock[1] require training a diffusion generative model and a separate confidence model, while other approaches like EDM-Dock[2], rely on reconstructing ligand conformations from predicted distance maps. These methods are not end-to-end and do not directly align with our objective of evaluating pretrained pocket representations.

Nevertheless, we recognize the potential of our method in enhancing existing binding pose prediction techniques. To integrate our approach, we would need to modify our framework. Using the same data creation strategy, we could train a binding pose prediction model with our preprocessed data, which could then be further fine-tuned using real pocket-ligand pair data from sources like PDBbind. Additionally, our data could be used to train a side-chain packing model, allowing for side-chain flexibility during docking. Thank you again for your advice, and we will leave protein-ligand binding pose prediction as a separate future work for our method.

[1] Corso et al., "Diffdock: Diffusion steps, twists, and turns for molecular docking.", ICLR 2023.
[2] Masters et al., "Deep learning model for efficient protein–ligand docking with implicit side-chain flexibility." Journal of Chemical Information and Modeling 63, no. 6 (2023).

Reviewer comment (Experiments):
Is there any new methods on the POCKET MATCHING task? If so, please include them in comparison.

Author response:
For the pocket matching task, CoSP is a newly proposed baseline which is published in ECML PKDD 2023. Alongside CoSP and Uni-Mol, we selected some of the most effective baseline results presented in the CoSP paper. We also tried to evaluate recent methods like PocketAnchor (Li et al., Cell Systems 2022) on our own since they are not tested on the pocket matching task. However, since the result didn't outperform other baseline machine learning methods(Uni-Mol, CoSP, and DeeplyTough), we decided not to include it in our final baseline comparison. We can include it in the camera-ready version if needed.

Reviewer comment (Novelty):
I'm not fully satisfied with the Related Work section. More work should be presented, such as [1], [2] and [3].

Author response:
Thanks for providing us with more insightful related works that could support our arguments. We actually have cited these three papers in the section on ligand binding affinity experiments, but we are also happy to add these in Related Works section 2.2. You can find the change in pdfdiff.

Reviewer comment (Experiments):
Why is COSP introduced as a component of the pocket pretraining method in Section 2.2, but not included as a baseline in Table 3?

Author response:
Because best to our knowledge, they did not release their code and they did not test their method on the ligand binding affinity dataset.

Reviewer comment (Reproducibility):
I am confused about how ProFSA works without the distributional alignment mechanism. In this context, what determines the length of the pocket representation?

Author response:
With or without distributional alignment, pockets are always defined by the given protein fragment with a fixed distance cutoff (6Å in our works, following the UniMol setup). The distributional alignment process merely samples these pairs to match the sizes of real ligands and pockets. Without distributional alignment, fragments are uniformly sampled from 1 to 7 residues. Pockets are similarly determined as with residues within the range of 6Å around peptide fragments.

Reviewer comment (Novelty):
The technical novelty is limited. The pocket encoder is borrowed from Uni-Mol. The contrastive loss is the vanilla form of classical contrastive learning.

Author response:
We appreciate the time and effort you have dedicated to reviewing our paper, and we are grateful for your constructive feedback and thoughtful evaluation of our work. We would like to address your comments about the technical novelty of our work and provide additional clarification on certain aspects of our paper.

## Response to technical novelty

Firstly, we acknowledge your observation that we didn't use any fancy models compared with typical **model-centric** works. While we respect your assessment, we would like to highlight that our primary focus in this paper is a **data-centric** pretraining method to introduce groundbreaking improvements in the field of protein pocket pretraining. Our work aims to address a critical challenge of data scarcity in the field of protein pocket representation by constructing large-scale synthetic data that facilitates the pretraining of models, ultimately enhancing the accuracy and robustness of protein pocket representations. We believe that the research community can conduct fast following-ups on our **released dataset** with more sophisticated models, and even extend our pipelines to other tasks like docking or drug design. We also would like to point out that we have provided a novel method to **distill knowledge from well-trained molecule models** to protein models. Though employing the same contrastive loss, our approach is different from existing contrastive learning models since it uses a fixed molecular encoder. The motivation here is to use a well-trained molecular encoder on a relatively larger dataset to **guide** the training of the protein encoder. For example, some quantum-chemistry properties that are difficult to compute for large systems like proteins could be distilled from molecules, by using some quantum-chemistry-aware molecular encoders such as Frad. Though our framework seems to be simple, it is non-trivial to make it work. We have made several efforts to solve the unavoidable discrepancies between true ligands and pseudo ligands. The ablation study shows that our efforts are effective and necessary:

|  | Kahraman$\uparrow$ | Tough M1$\uparrow$ | Fpocket$\downarrow$ | Druggability$\downarrow$ | Total SASA$\downarrow$ | Hydrophobicity$\downarrow$|
| --- | --- | --- | --- | --- | --- | --- |
| ProFSA | 0.7870 | 0.8178 | 0.1238 | 0.1090 | 31.17 | 12.01 |
| w/o alignment | 0.7614 | 0.7589 | 0.1265 | 0.1108 | 34.79 | 14.86 |
| w/o fix mol encoder | 0.6905 | 0.7337 | 0.1247 | 0.1094 | 32.17 | 12.20 |

Regarding your positive acknowledgment of our "simple but effective model for the right question," we are pleased to hear that our approach resonates with the objective we set out to achieve. We designed our model with **simplicity** in mind, prioritizing effectiveness and practical utility for the specific problem domain. This deliberate choice aligns with the notion that sometimes the most impactful solutions are elegantly straightforward. Also, we intentionally borrowed a pocket encoder from Uni-Mol to make a **fair comparison** with it, which strongly supports the power of our dataset. Similar to the molecule encoder, our pocket encoder can be changed to any other model due to our flexible framework. Notably, **we didn't load the pretrained weights of Uni-Mol pocket encoder**. We only use the same backbone architecture and the pretraining was completely done on our processed data with our training strategy. 

Notably, many simple but effective approaches have been recognized by top-tier conferences and journals. A prime example of this is in the field of protein language models, like the ESM series and ProtTrans. These studies adapted the Transformer architecture and masked language modeling techniques from natural language processing to protein sequences. While they didn't introduce groundbreaking techniques, their substantial contributions to protein modeling are evident, with publications in prestigious conferences and journals like NeurIPS, ICML, TPAMI, PNAS, and Science. Another example is the widely acclaimed CLIP paper presented at ICML 2021. CLIP, while not employing novel techniques, stands as a hallmark of data-centric deep learning. Its use of contrastive learning enabled training on expansive web-sourced text-image datasets, moving beyond the constraints of meticulously curated databases like MS-COCO. This aligns with our approach to addressing **data scarcity challenges** by facilitating training on large-scale datasets, mirroring our strategy for overcoming similar hurdles in the protein pocket modeling domain.

We appreciate your efforts to remain neutral and understand the challenges in decision-making. We believe further discussion will highlight our paper's contributions. Your feedback and insights in the upcoming discussion will be invaluable for refining our work.

Reviewer comment (Theory):
The bound of Theorem 3.1 is trivial. The authors claim that the bound naturally exists for these representations extracted by pretrained molecule models. However, it's a bit counterintuitive, because many models not pretrained on molecule datasets also fulfill this prior. So, can these models be used for this task? **I strongly suggest removing this part from the paper**.

Author response:
## Response to the bound of Theorem 3.1 

We are not very clear about the point of your question. We have two understandings and they are replied as follows.

If you mean $ ||t − t ^{(0)}|| < 1/2 l_T$ is trivial, please note that it is only a condition of the theorem. As you said, many other encoders can also satisfy this condition. This shows that our method is applicable to many different molecular encoders with or without pretraining as we proved in **Table 4**. Moreover, the condition is only a necessary condition. We also need the pretraining contrastive loss to be sufficiently small to guarantee a small contrastive loss between pockets and real ligands. In fact, in the process of our early exploration, we noticed that some encoders cannot achieve a low pretraining loss, indicating the entire conditions of the theorem are nontrivial.

If you mean that the conclusion of theorem 3.1 is trivial, we want to emphasize that our conclusion is $\lim_{L_i(t, s) =0}L_i(t^{(0)}, s) =0$. (We write it in a $\epsilon$-$\delta$ language in paper. They are equivalent forms.) Our conclusion shows the loss containing real ligands is consistent with the loss containing pseudo ligands that we optimized in pre-training. 

In contrast,  a trivial result from $||t − t ^{(0)}|| < 1/ 2l_T$ is that $\lim_{L_i(t, s) =0}L_i(t^{(0)}, s) <B$, B is a bound related to l_t and the representations. It does not guarantee a consistent loss and is substantially different from our result.
Therefore, our theorem is nontrivial and reveals the transfer ability of our contrastive pre-training from the pseudo-ligand domain to the real ligand domain.

As for this theoretical result, we are open to further discussion.

Reviewer comment (Experiments):
In the dataset creation process, the authors consider the distribution of ligand size and pocket size when designing the dataset. However, molecules possess more properties that can also lead to imbalance. It would be better to, at least, add some discussion about this issue.

Author response:
## Response to molecules possess more properties that can also lead to imbalance

It is correct that molecules have more properties than their size, like logP, Hbond donor and acceptor number, and rotatory bond number. As small molecule drugs are mostly designed to penetrate barriers like gut or cell membranes, they usually have much larger logP values, which means more hydrophobic. For the same reason, they usually have fewer hydrogen bond donors and acceptors, and only minimal essential ones are kept for specified interactions. Also, to minimize the entropy effect upon binding and to increase binding affinity, rotatory bonds are also unfavored. However, it is impossible to mimic such features with peptides, as the backbone of peptides is intrinsically hydrophilic and flexible. As we showed in a new figure, our pseudo-ligands are less similar to real ligands in those properties even with size alignment. You can find the figure in the new pdf in **Appendix G**.

We believe that the aforementioned divergence is the main limitation of poor zero-shot performance in predicting hydrophobicity scores (**Table 1**). However, as demonstrated in previous publications like CoSP, we could leverage real ligand-pocket pairs from PDBBind or BioLip database to further finetune our network, as an extension of the pipeline. In this way, we could handle property mismatches but still enjoy the power of our large-scale pretraining.

Reviewer comment (Experiments):
In the dataset creation process, during the second stage of the data construction, the approach to defining pockets needs further explanation or an ablation study.

Author response:
## Response to the approach to defining pockets 

We define the pocket following the UniMol setup. We found our model can also adapt to an 8Å setup in the toughM1 experiment 

In particular, we define the pocket for each protein-ligand pair as residues of the protein that have at least one atom within the range of 6Å from a heavy atom in the ligand. To further explain our design, we have done an ablation study on different choices of thresholds of 4Å, 6Å, and 8Å:

|  | Kahraman$\uparrow$ | Tough M1$\uparrow$ | Fpocket$\downarrow$ | Druggability$\downarrow$ | Total SASA$\downarrow$ | Hydrophobicity$\downarrow$|
| --- | --- | --- | --- | --- | --- | --- |
| 4Å | 0.7062 | 0.7549 | 0.1240 | 0.1095 | 28.29 | 13.07 |
| 6Å | 0.7870 | 0.8178 | 0.1238 | 0.1090 | 31.17 | 12.01 |
| 8Å | 0.8322 | 0.8292 | 0.1256 | 0.1125 | 34.83 | 12.92 |

Since the 8Å threshold corresponds with the pocket definition in the Kahraman and Tough M1 datasets, it leads to optimal results. Our decision to use a 6Å threshold was made to align with the methodology of pretraining data creation by Uni-Mol, facilitating a fair comparison. Notably, even with the 6Å threshold, we achieved strong results in the pocket-matching task, which serves as a testament to the effectiveness of our approach.