Paper: Deep Confident Steps to New Pockets: Strategies for Docking Generalization (ICLR-2024)

Reviewer comment (Evaluation):
The paper mentioned a lot of works in terms of neural network architecture and data augmentation. However, I could not see their connections to the main topic of generalization. In Table 1, all those attempts failed to improve DIFFDOCK performance in the DockGen benchmark in terms of success rate (<2A).

Author response:
The reason why we tested these different approaches is that a few of the most widely used techniques to improve generalization used with neural networks are (1) improve inductive biases in neural architecture, (2) get more data, (3) perform data augmentation. Indeed, these strategies do not seem to provide significant improvements in this setting (with arguably the exception of data increase). We do report these "failed attempts" because we believe it gives the reader a better understanding of the tradeoff between different approaches. We clarified this point in the manuscript.

Reviewer comment (Reproducibility):
Although a code repo is provided, I can not find any details about how baselines are benchmarked with the DockGen dataset. Specifically, when new data-split is provided, it is important to report whether baselines are re-trained or not. I am also confused by unclearified marks like EX. 64.

Author response:
Thank you for raising these points. We tried to clarify them below and in the manuscript. The DockGen benchmark was designed specifically to be compatible with the dataset and splits the baselines were originally trained on i.e. the (validation and) test sets contain complexes that belong to different clusters from those seen in PDBBind's training and validation set. Therefore none of the baselines required retraining. Ex. indicates the level of search exhaustiveness that can be regulated in search-based methods.

Reviewer comment (Experiments):
The effect of confidence bootstraping is not approperately evaluated. Although the performance is demonstrated on 8 clusters, it is documented that there are 63 clusters in the test set. It is not convincing to claim the proposed method is effective before a more systematic and quantitative evaluation.

Author response:
As a reminder, the model is finetuned on individual protein clusters. We have selected the 8 clusters in the test set by including all clusters that have 6 or more complexes. Most clusters contain 2 or less complexes, and it is difficult to fine-tune on clusters with very little structural or binding data (on top of being computationally very expensive to finetune 63 different models). Although we only include 8 clusters out of 63 in the test set, the 8 clusters actually contain 85 out of 189 complexes in the test split. We clarified these points in the manuscript.

Reviewer comment (Experiments):
Why is ECOD used for data split? Best to my knowledge, SCOP(e) and CATH are more widely acknowledged.

Author response:
We largely expect ECOD, SCOP(E), and CATH to give similar splits (see the introduction of [1]). We used ECOD also because it is automated and applied to every PDB entry.

[1] R. D. Schaeffer, J. Zhang, L. N. Kinch, J. Pei, Q. Cong, N. V. Grishin, Classification of domains in predicted structures of the human proteome. Proc. Natl. Acad. Sci. U. S. A. 120, e2214069120 (2023)

Reviewer comment (Evaluation):
1QXZ and 5M4Q have a sequence similarity of 22% but not 16%. I blast it myself.

Author response:
Thank you for pointing this out. We obtained the 16% value with the pw2.align.globalxx routine of biopython. However, we recognize that BLAST is a more widely accepted score and have updated the manuscript accordingly.

Reviewer comment (Evaluation):
Why do traditional methods like SMINA not perform well on the DockGen benchmark?

Author response:
We believe it is for a combination of reasons: (1) they also contain some parameters that have been fitted to protein domain clusters that are unlikely to be contained in our benchmark (i.e. the receptors are harder than for example those in PDBBind), (2) the receptors present in our test set are on average larger than those present in PDBBind's test set making the docking prediction potentially harder. We now plot these statistics and discuss this point in Appendix A.2.

Reviewer comment (Experiments):
There is one recent work that achieves strong performance on unseen targets (though split by UniProt ID), the authors are strongly encouraged to compared the performance on that model: FABind, https://arxiv.org/abs/2310.06763

Author response:
Thank you for the suggestion. We tried running FABind on the DockGen benchmark but have so far been unsuccessful. The repository does not give precise instructions on how to preprocess the data (in README the inference on custom complexes section is referred as "coming soon"). It does mention that the preprocessing is the same as TANKBind, but when we tried using the preprocessed data we generated for TANKBind we ran into some errors. We will reach out to the authors to understand how to run the model and will update the results in the future.

Reviewer comment (Evaluation):
More analysis needed on what protein features DOCKGEN probes that past benchmarks missed.

Author response:
Although it is hard to point out specific protein features that are captured in DockGen but not in PDBBind's test set, the main property that DockGen is able to probe is the capacity of generalization to new binding modes. To highlight this we conduct a careful analysis of the binding sites in the various datasets in Appendix A.2 comparing the similarities that the different benchmarks have with the training set. 

In particular, we extract the binding sites of different complexes and then compare them both at a sequence/chemical (using the BLOSUM62 amino acid substitution matrix to quantify the similarity of aligned residues) and structural (using the TMalign output for any aligned residues) level. The results, shown in Figure 5, show significant overlap in the pockets in PDBBind and very little similarity when looking at DockGen pockets. More details on this analysis can be found in Appendix A.2.

Reviewer comment (Experiments):
Ablation studies could better isolate benefits of different components of Confidence Bootstrapping.

Author response:
Thank you for raising this suggestion. We have now included a number of the ablation results of different components of the method (use of buffer, buffer size, use of training examples, sampling schedule…) in Appendix E.

Reviewer comment (Writing):
The mathematical formulas in the paper should be corrected. Some parentheses are missing, and there are some symbols that were not introduced in the text, e.g. $p_{0t}$. I understand that the equation in Section 4.1 should correspond to the equation introduced in the cited publication (Song et al., 2021), but the new undefined symbols and missing parentheses make this equation hardly readable.

Author response:
Thank you for pointing this out. We fixed the equation and the symbols.

Reviewer comment (Experiments):
In Table 1, the results of DiffDock with modification explained in Section 4.4 and without dataset extension should also be presented.

Author response:
Thank you for the suggestion we added this row to the table. Unfortunately, the model has not finished training yet so we have put the performance after approximately 70% of the expected training time and we will update the result with the final performance.

Reviewer comment (Evaluation):
In this benchmark, the only evaluation metric (besides computation time) is the percentage of the poses with RMSD below 2 or 5 A. Given the recent criticism of ML-based docking models, it would be advised to include conformation quality metrics like those proposed in PoseBusters [1].

Author response:
We agree with the reviewer and the authors of [1] that the quality of poses is an important metric. However, [1] shows that although the poses coming out of the diffusion model do not pass all quality metrics most of the time, they do pass them while remaining accurate typically if a quick energy minimization is applied. In [1] (Figure 5) the performance of DiffDock <2A goes down (in their benchmark) only from 38% to 35% if quality checks are also applied, assuming poses are relaxed (instead of 12% without relaxation). Therefore we believe that it makes sense to apply these checks in combination with the relaxation. To perform this test, we reached out (twice) to the authors of [1] to get the code used for the relaxation procedure, however, they said they will only release it in the future. Once this code is made available we'd be very interested in adding these metrics as the reviewer suggested!