Paper: Protein Language Models Enable Accurate Cryptic Ligand Binding Pocket Prediction (ICLR-2024)

Reviewer comment (Novelty):
First and foremost, the novelty is quite limited. As a reviewer of ICLR, one of the top AI conferences, I expect to see great contributions to the deep learning community. However, the PLM algorithms including ESM-2, AnKh, ProtBert, and others are very mature techniques. The multi-task training with a secondary structure prediction is kind of interesting but still trivial. The author ought to provide more discussion and deeper insight regarding the benefit of this multi-task scheme in the main text. To summarize, the author just examine an existing methodology (i.e., PLMs) on some publicly available datasets (PDBBind + ESM-2 SSP) with a straightforward training paradigm (fixed embedding and [CLS] for prediction).

Author response:
Thank you for your time and consideration in reviewing our work.

This is an application paper, and as such it is not the intent to showcase a new algorithm.  This is within scope as outlined in the call for papers.  BERT was not a new algorithm either, it was a new application, trained with publicly available data.

Outperforming SOTA by such a large margin, on such an impactful application in small molecule pharmaceutical discovery is a noteworthy contribution to ML that should encourage many avenues of future work and be a net benefit for the human race.  

Regarding SSP, it was not consistently a benefit, however this is important to report for future investigators.

Reviewer comment (Experiments):
PLMs have shown promise in solving biological problems. However, it utilizes no structural information. Since we do have structural data in PDBBind, it really cannot convince me that PLMs are the best solution to predict cryptic pockets. (i) Please examine and compare more baselines (I am not familar with this specific task, and the references listed may be wrong) [A][B]. (ii) Machine-learning-based docking method can be also transferred to predict the pocket, such as EquiBind, and DiffDock. Please examine them. (iii) Please equip geometric networks (e.g., PocketMiner) with PLMs and verify the performance. See [C] for more details. I suppose PLMs + geometric models should outperform PLMs.

Author response:
The closest related work is PocketMiner, which has the goal of identifying residues--but not any ligand--that are likely to participate in or comprise cryptic pockets.

We mentioned as future work the combination of this approach with geometric approaches.

Equibind and Diffdock are fundamentally different approaches concerned with finding a binding pose given a protein and a ligand, which is a question that this work is not trying to answer.  If this were a docking application then they would be more relevant.

Reviewer comment (Reproducibility):
As a machine learning paper, it is very necessary to offer adequate experimental details for readers to reproduce the results (at least in the Appendix). For instance, how many computational resources (e.g., GPUs) have been used in the training? What is the optimizer type and how high is the learning rate?

Author response:
Regarding implementation and computational details, please see the methods section for details on the optimizer.  The learning rate required some experimentation and was not the same for all model types presented.  The model sizes are described and models were trained on single GPUs (no DDP or similar necessary).

Reviewer comment (Experiments):
In Section 5.2, the author uses the average embedding as a pseudo-[CLS] token for ProtT5-XL. However, in the official document of ESM-2, it is also recommended to use the average embedding as the representation of the entire protein. Have the author tried this style for training?

Author response:
Regarding specific questions:
1.  No ESM-2 embeddings were not averaged, although that could be added to the current results.

Reviewer comment (Presentation):
A minor point is that the authors adopt an unusual arrangement to organize the sections, which accords with the template of Spring Nature Journals. From my humble point of view, a standard format of AI conference is to put the method section before the result and conclusion sections.

Author response:
Regarding specific questions:
2.  The methods section can be rearranged to be after the background.

Reviewer comment (Novelty):
The authors are encouraged to design some methods that are best for this specific task.

Author response:
While future work is envisioned to create bespoke algorithms, this work already constitutes a significant advance in this area.

Reviewer comment (Novelty):
Is this the first work that use the concept of "cryptic pocket"? Seems no previous work about this term is referenced in Sec 2.1. The authors should either (a) discuss more previous study of cryptic pocket, or (b) give more accurate definition and explanation about this concept.

Author response:
Regarding cryptic pockets, section 2.1 is explicitly about them.  The references of PocketMiner and CryptoSite provide additional background, and are studies aimed at identifying them.

Reviewer comment (Novelty):
The novelty appears limited. The main technique in this study is "PLM as feature," which emerged in 2018 (BERT).

Author response:
While the combination of PLM with MLP/MHA (with our without SSP) is not novel from an algorithms point of view, it is a unique approach that outperforms SOTA by a significant margin for an impactful application.  BERT, as you mention, itself is not novel from an algorithms point of view either.

Reviewer comment (Evaluation):
Incorporating SSP doesn't seem consistently better than other methods.

Author response:
SSP is still worth reporting to assist future investigators in their efforts.

Reviewer comment (Writing):
There's still room for improvement in the writing. There are too many technical details that aren't informative, such as comparative discussions on LR, MLP with MHA-x, with or without CLS. Also, the results across the tables lack consistency. The organization of the paper (i.e., "Results -> Conclusion -> Method") seems unusual to the ML audience.

Author response:
Happy to put methods after background.

Reviewer comment (Novelty):
I believe the main weakness of the paper is that, in terms of the proposed method, the paper does not provide a significant novel contribution to the broader ICLR community. This is understandable since the purpose of the paper is to showcase that PLMs could be beneficial in identifying cryptic pockets in proteins, but I wonder whether ICLR is the best venue for such work to be published and gain visibility.

Author response:
We appreciate your concern regarding algorithmic novelty and venue.  Our thinking is that if something simple significantly outperforms SOTA for an impactful application then that's an important contribution.  It also paves the way for future work to build upon the lessons learned, one of which being the marginal benefit of SSP in this context.

Reviewer comment (Experiments):
The authors mention that ESP provides more localized positive cryptic pocket predictions as compared to PocketMiner, whose positive predictions are more broadly distributed. Given the knowledge that cryptic pockets tend to have local structures, could such prior knowledge be injected into the prediction model, for example, as a regularizer (where local positive predictions are encouraged during model training, while negative ones are discouraged)?

Author response:
That's definitely an interesting idea, to regularize based on the overall size and number of residues comprising ground truth cryptic pockets.  This work focused on using the simplest possible process, and would be a relevant baseline for introduction of a regularizer as you describe.

Reviewer comment (Evaluation):
In Section 5.3, it is mentioned that the results are reported for the *best* model from 7 trials. I was expecting that for each specific configuration (i.e., PLM/head/task), the *average* performance (and its *standard deviation*) across the seven random trials would get reported, not the *maximum* performance.

Author response:
While average and standard deviation are commonly used in ML literature, because this is a small molecule pharmaceutical application, the performance of the best model is the actual limiting factor in real-world application.  Evaluating methodologies by their best model is therefore most informative regarding practice.

Reviewer comment (Experiments):
The number of parameters for ESP in Table 7, especially with the MLP 1024 and MHA heads, is orders of magnitude larger than the number of parameters of PocketMiner. This is not even taking into account the number of PLM parameters (which are taken to be frozen in this paper). How does ESP perform compared to PocketMiner for a comparable number of parameters?

Author response:
It's definitely a valid point regarding the vast expansion of the trainable parameter space compared to PocketMiner.  However, for Ankh, even logistic regression makes an improvement to ROCAUC.  This does not take into account the parameters of the PLM.  However, it's not clear that PocketMiner can meaningfully scale to larger numbers of trainable parameters in the sense that graph methods have scaling issues regarding run time, and more parameters doesn't always mean more efficacy.  This is an interesting discussion to include in more detail in future work.

Reviewer comment (Writing):
I may have missed these, but there are certain acronyms in the manuscript that are not defined anywhere in the paper (such as AUC, APS, LR, CLS). Please review the manuscript and make sure all acronyms are defined the first time they are used.

Author response:
We are happy to explicitly define those acronyms, either in the main text or an appendix.