Paper: Rigid Protein-Protein Docking via Equivariant Elliptic-Paraboloid Interface Prediction (ICLR-2024)

Reviewer comment (Evaluation):
The major concern about the method lies in the comparison with traditional template-based methods, e.g., HDOCK. As shown in Tables 1&2, HDOCK outperforms other methods by a significant margin, suggesting it may already effectively address the rigid-body docking challenge. Although the ElliDock’s inference speed is much faster than HDOCK, its performance is unsatisfactory. Given real-world applications, the value of faster inference time seems negligible, especially if the docking challenge doesn't necessitate high-throughput situations. Please correct me if I’m wrong.

Author response:
Thank you for the comment. While we respectfully agree with your point that HDOCK is a strong baseline in performance, we believe that models with faster inference time are more suitable in high-throughput scenarios. For example, in drug design, when studying whether inserting, editing or deleting sequence fragments in different regions of a protein affects the binding ability and binding site. If ElliDock is used in this case to simulate the binding process among the edited proteins, its fast speed will nicely support a large number of simulated docking processes within an acceptable time, such that we can locate the desirable protein candidate from its huge amino acid space.

Reviewer comment (Experiments):
Despite the considerable depth in the methods section discussing the rationale and procedure of using an elliptic paraboloid for approximation, it remains unclear how this looks like on real-world data. It would be good if there are some visualization experiments showing the learned paraboloid interfaces.

Author response:
Sorry for the inconvenience that the revised part of the paper was not marked in red previously, now we have resubmitted our revised paper. For the surface-based baselines you mentioned, we have added the discussion of MaSIF in the Introduction section. We have also modified the notation issue. Meanwhile, we provide visualization of predicted elliptic paraboloid interfaces in the appendix. Please let us know if you have other concerns.

Reviewer comment (Writing):
The paper's notations appear cluttered and confusing. For example, iEq.(8) denotes the other protein as $H_{-p}$, which is unconventional. In Eq. (12), the FC layer maps representation to $4$-dim without explanation. Meanwhile, Eq. (13) introduces the hyperparameter $M$ without elaboration. The bracket notation further complicates understanding. For better clarity, consider excluding the $p$, since most operations apply similarly to both proteins, and employ subscript notation for indexing.

Author response:
Thank you very much for raising these issues. We have made corrections to the notations you mentioned. In Eq. (8), we use $\neg{p}$ to denote the other protein. In Eq. (12), the first three elements of $F_1$ and $F_2$ are used to construct the elliptical paraboloid, and the fourth element is used for rotation refinement, which are both mentioned in the paper. $M$ is a hyperparameter introduced in Eq. (13), which is further specified in the paragraph under Eq. (16). Regarding the problem of excluding $p$, we have formulas that only apply to a single protein (e.g., Eq. (8)), formulas that require information interaction (e.g., Eq. (7)), and formulas that require both proteins (e.g., Eq. (14)), so we believe that the subscript $p$ can make the expression clearer.

Reviewer comment (Experiments):
A deeper exploration and connection with surface-based methodologies, such as MaSIF, would be beneficial. It would be insightful to include a comparison with it in the benchmarking section.

Author response:
Thank you for introducing MaSIF as a surface-based methodology. From a methodological point of view, MaSIF models a real surface with biochemical properties, making use of protein surface chemical features such as hydropathy and free electrons; ElliDock models a fictional elliptical paraboloid as a surface, which does not represent chemical properties, yet represents the position information of the pockets as well. On the other hand, MaSIF can obtain its fingerprint from protein monomers, which is conducive to studying the properties of the monomers, while ElliDock interacts with the features of receptors and ligands during the docking process, which may improve the expressiveness on complexes. 

To address your concern, we have added the content to the Introduction section in the revised version.

Reviewer comment (Theory):
In Eqs. (3)(4), the coefficient $\alpha_{j\to i}$ appears to merely normalize the inner product of $q$ and $k$, without taking into account the keys from other edges. This approach seems to limit the attention mechanism's capacity to capture interactions among edges or nodes. Given this, wouldn't it be more straightforward to directly map the message $m_{j\to i}$ to a scalar coefficient $\alpha_{j\to i}$?

Author response:
We believe that the inner product of $q$ and $k$ has a larger representation space than direct mapping of the message $m$, because the former is a mapping of $\mathbb{R}^{H\times H}$ to $\mathbb{R}$, and the latter is a mapping of $\mathbb{R}^{H}$ to $\mathbb{R}$. Please correct me if I'm wrong.

Reviewer comment (Experiments):
In Sec. 4.3, there's a mention that pre-trained models like HDOCK could be prone to data leakage issues. From my understanding, HDOCK functions as a template-based model and doesn't utilize learning in its scoring function. Could you provide a more detailed explanation regarding the concerns of data leakage?

Author response:
Yes, we can show the evidence of the possible data leakage of HDock. From the paper of HDock [1], we find that HDock may have data leakage in two parts: first, during the docking procedure, HDock needs to search the whole PDB (thus may also include the testing data in our experiments) for homologous templates and build the 3D structures through homology modeling, and HDock will update the PDB data through version; second, the protein-protein scoring function [2] used by HDock, referred to as ITScore-PP, was derived using the crystal structures of a training set of 851 protein–protein dimeric complexes containing true biological interfaces. 

[1] Yan, Y., Tao, H., He, J., & Huang, S. Y. (2020). The HDOCK server for integrated protein–protein docking. Nature protocols, 15(5), 1829-1852.

[2] Huang, S. Y., & Zou, X. (2008). An iterative knowledge‐based scoring function for protein–protein recognition. Proteins: Structure, Function, and Bioinformatics, 72(2), 557-579.

Reviewer comment (Theory):
In Eq. (12), while $F_p$ is identified as a 4-dimensional vector, its fourth element seems not to be used in the main paper.

Author response:
Sorry for the confusion. The fourth element of $F_p$ is used during rotation refinement, which is shown in Section 3.5.

Reviewer comment (Theory):
The authors demonstrate a significant improvement in inference time for their method compared to baseline approaches, yet the exact source of this efficiency enhancement remains somewhat unclear. An in-depth complexity analysis would be helpful in shedding light on the reasons behind this improvement. Typically, if a method directly predicts rotation and translation matrices, inference time should be as swift as the execution of these operations.

Author response:
Thanks. Here we briefly analyze reasons for the differences between the inference time of different models. First of all, ElliDock and EquiDock are both regression models and generate the transformation in one step from end to end. Therefore, there is no order of magnitude difference between them in speed. From Table 1, we show ElliDock is $\textasciitilde$2x faster than EquiDock, which is more likely due to the amount of model parameters, number of model layers, etc.. DiffDock-PP is a diffusion-based model, with a denoise process of T steps (T is a hyperparameter) in the inference procedure. Each inference step requires re-running the model, so there is a speed gap of 10$\textasciitilde$100 times compared with the regression model. HDock is a template-based model that not only obtain the templates of receptors and ligands through sequence alignment, but also needs to evaluate the energy of the predicted complex based on the scoring function and perform multi-step updates accordingly. Meanwhile, HDock can only run on the CPU, limiting its speed even more. Multimer is a deep learning model, yet it requires multiple sequence alignment (MSA) during training and inference, and its pre-trained models have a large number of parameters (GB level), so the inference time is extremely long.

Reviewer comment (Reproducibility):
Furthermore, sharing the splits of the second benchmark dataset used in your experiments would be advantageous, as it would enable others to replicate the results more easily and conduct experiments with the same splits in the future. This transparency promotes reproducibility in the research community.

Author response:
We have already uploaded the bash file for downloading and spliting the dataset. Please check README in our source code for more details. All the code sources will be released after our paper is accepted.

Reviewer comment (Theory):
Would you kindly offer substantiating evidence or theoretical findings to validate the assumption that elliptic-paraboloids accurately align with the ground-truth docked pockets?

Author response:
Thanks! We provide more explanations on DB5.5 here. DB5.5 is a well established docking database. From our point of view, IRMSD measures the ability of the model to align to pockets. In Tables 1 and 2, the performance of our model on IRMSD is better than that of models trained from scratch, indicating that our method is better at finding pockets. We are open to discussing if you still have any questions.

Reviewer comment (Evaluation):
Can you provide a more comprehensive explanation regarding the reasons behind the improved inference time compared to other methods?

Author response:
Thanks. Here we briefly analyze reasons for the differences between the inference time of different models. First of all, ElliDock and EquiDock are both regression models and generate the transformation in one step from end to end. Therefore, there is no order of magnitude difference between them in speed. From Table 1, we show ElliDock is $\textasciitilde$2x faster than EquiDock, which is more likely due to the amount of model parameters, number of model layers, etc.. DiffDock-PP is a diffusion-based model, with a denoise process of T steps (T is a hyperparameter) in the inference procedure. Each inference step requires re-running the model, so there is a speed gap of 10$\textasciitilde$100 times compared with the regression model. HDock is a template-based model that not only obtain the templates of receptors and ligands through sequence alignment, but also needs to evaluate the energy of the predicted complex based on the scoring function and perform multi-step updates accordingly. Meanwhile, HDock can only run on the CPU, limiting its speed even more. Multimer is a deep learning model, yet it requires multiple sequence alignment (MSA) during training and inference, and its pre-trained models have a large number of parameters (GB level), so the inference time is extremely long.

Reviewer comment (Reproducibility):
Could you consider releasing the splits of the second dataset utilized in your experiments to facilitate reproducibility efforts within the research community?

Author response:
We have already uploaded the bash file for downloading and spliting the dataset. Please check README in our source code for more details. All the code sources will be released after our paper is accepted.

Reviewer comment (Theory):
The paraboloid formulation is elegant, but I would like to see some discussion as to why this is the right choice compared to other considered options.

Author response:
Thank you for recognizing that our work. We choose elliptic paraboloids other than planes, spheres or complex manifolds to model the binding interface, as elliptic paraboloids are appropriately defined in 3D space and their SE(3) transformations can be devised in a closed form. On the one hand, if using planes, spheres or other symmetric 2D manifolds to model the interface, it will easily cause the docking ambiguity issue since there are usually different ways to transform a 2D manifold from one pose to another.  On the other hand, if using complex manifolds, it may introduce bias and higher computational complexity to obtain the docking solutions.  Under these two requirements, we believe that the elliptical paraboloid is a good trade-off.

Reviewer comment (Novelty):
The argument of why ElliDock's formulation is better than EquiDock seems to be developed sufficiently well, but I would like to see some more discussion contrasting with DiffDock-PP and AlphaFold-Multimer.

Author response:
Sorry for the insufficient explanations. We know that our work ElliDock and EquiDock are both regression models, which allows an easier comparison between two methods. On the contrary, DiffDock-PP is a diffusion-based model, it successfully builds a model that meets the degree of freedom required by the task. The introduction of noise in diffusion steps gives DiffDock-PP the ability to find multiple binding pockets, while it also leads to 10~100 times of the inference time compared to regression models. Alphafold-Multimer is based on Alphafold, its task setting is the structure prediction from sequence instead of the rigid protein-protein docking, thus the method design is different from other methods mentioned above.

Reviewer comment (Evaluation):
The inference time is reported in Table 1, but not in Table 2. Could you please report the values for Table 2?

Author response:
Nice suggestion! We have additionally updated the inference time in Table 2 in the revision. We believe that the inference time should be approximately proportional to the scale of the test set, so we only reported the results of one test set previously.

Reviewer comment (Experiments):
Could you please report the inference time in Table 3. It would be interesting to see which losses could be removed for even more speed-up, but without accuracy loss.

Author response:
Thank you for the comment. We would like to clarify that the losses only play a role in the training process and have little impact on the inference time during validation and testing. Hence, it is not very informative to report the inference time in Table 3 (the time is almost the same no matter which loss we remove).

Reviewer comment (Experiments):
Since DiffDock-PP is a diffusion-based model it is capable of modelling multiple conformations. How does ElliDock fare when the ligand-protein pair has multiple possible conformations?

Author response:
Thank you for raising this question. Like EquiDock and other regression-based methods, our Ellidock is not specifically designed to handle the situation when there are multiple binding sites for one single pair of receptor and ligand. However, for the datasets, particularly the antibody-antigen binding tasks, the specificity of the antibody is almost only determined by a small special region of itself (complementarity determining region, CDRs). In actual scenarios, it might be difficult for the same pair of antibody and antigen to have multiple binding sites.

Reviewer comment (Theory):
Could you argue a bit about why ElliDock's training objective is more suitable than DiffDock-PP's?

Author response:
In our perspective, since ElliDock and DiffDock-PP are different types of models, the training objectives are difficult to compare directly. The training target of DiffDock-PP is mainly the score function of rotation and translation, which pays more attention to the distribution changes of the overall molecule. ElliDock is based on interface fitting and tends to focus more on pockets in the design of training objectives.

Reviewer comment (Evaluation):
Why do you think ElliDock outperforms AlphaFold-Multimer on SAbDab?

Author response:
Thanks. Considering only the SAbDab data set, since the pre-training data of Multimer comes from the general protein domain, it probably performs poorly on the antibody-antigen domain. In Table 2 we show that ElliDock has better performance on both CRMSD and IRMSD metrics. As for the DockQ metric, it can be seen from Figure 5 that ElliDock has better prediction results on most test samples. However, Multimer has achieved extremely amazing accuracy on a few samples, which may be attributed to the powerful representation ability of Alphafold.

Reviewer comment (Evaluation):
the improvement is minor on DB5.5 when comparing with EquiDock and DiffDock-PP and much worse than Multimer and HDock.

Author response:
We would like to further emphasize the advantages of our Ellidock. 

First, as mentioned in the paper, we use the pre-trained models of Multimer and HDock, both of which use the data in the PDB for training (Multimer) or template selection (HDock). Since DB5.5 is included in PDB, we assume that Multimer and HDock perform extremely well on DB5.5 due to possible data leakage. The small scale of DB5.5 might also limit the learning ability of the model.

Second, inference time is another important factor in addition to the performance metric. Clearly, ElliDock is much faster than HDock, DiffDock-PP, and Multimer. In our responses to other reviewers, we believe that models with faster inference time are more suitable in high-throughput scenarios. For example, in drug design, when studying whether inserting, editing or deleting sequence fragments in different regions of a protein affects the binding ability and binding site. If ElliDock is used in this case to simulate the binding process between the editted proteins, its fast speed will nicely support a large number of sumulated docking processes within an acceptable time, such that we can locate the desirable protein candidate from its huge amino acid space.

Reviewer comment (Experiments):
[1] and [2] are stronger baselines on solving antibody-antigen complex.

Author response:
Thank you for introducing two baselines for antibody-antigen complex. Unfortunately, the codes of both baselines have not been published, it is difficult for us to use them as experimental benchmarks. We will add further discussion in the revised paper.

Reviewer comment (Experiments):
HDock performs much better than ElliDock. I guess the major reason is the input rigid structure is native unbounded structure. However, using native unbound structure as docking method's input is not realistic. What's the performance if you dock predicted structures.

Author response:
While we also respect HDock for its promising performance, we have some concerns about its possible data leakage, as already expressed in our paper. From the paper of HDock [3], we find that HDock may have data leakage in two parts: first, during the docking procedure, HDock needs to search the whole PDB (thus may also include the testing data in our experiments) for homologous templates and build the 3D structures through homology modeling, and HDock will update the PDB data through version; second, the protein-protein scoring function [4] used by HDock, referred to as ITScore-PP, was derived using the crystal structures of a training set of 851 protein–protein dimeric complexes containing true biological interfaces.

Reviewer comment (Theory):
the pair of elliptical paraboloids are with the same shape. How to handle large protein or samll proteins?

Author response:
There could be some misunderstanding here. For different protein pairs, due to the model design of global information interaction, the shape of the elliptical paraboloid is determined by both the receptor and the ligand, so the interface shape can be adapted to the scale of the protein.