Paper: EBMDock: Neural Probabilistic Protein-Protein Docking via a Differentiable Energy Model (ICLR-2024)

Reviewer comment (Evaluation):
The empirical results of the proposed method are not as strong as expected, as it fails to outperform traditional methods and AlphaFold-multimer. This raises questions about the effectiveness of the proposed approach in comparison to existing techniques.

Author response:
The AlphaFold method, which is based on multiple sequence alignments (MSAs), is difficult to improve in terms of efficiency and cannot be used for large-scale virtual screening. Even if their performance is good, there are significant limitations when dealing with large-scale data. Our approach aims to improve accuracy while maintaining high inference efficiency, although there is currently no method that can achieve this effectively. Therefore, even though our work may not match AlphaFold's, our work has made progress and provided new insights for researchers.

Reviewer comment (Experiments):
The use of only 100 samples for the DIPS-Het test set is questionable. It is unclear if there are any constraints that prevent the application of the proposed method to larger datasets. The authors should provide more information on this aspect and, if possible, test their method on a more extensive dataset.

Author response:
We select 100 samples for testing because methods like PatchDock, HDock, and AlphaFold-Multimer have too long inference time. These baselines limit us from conducting a fair comparison on a more extensive dataset. We have shown that our method is highly efficient and relatively stable, allowing scalability to large-scale datasets in our paper. Here, we present our results on the entire DIPS-Het test set (1153 samples). We apologize for the misunderstanding caused and hope that our clarification can alleviate your concerns about our method.

| Method  | CRMSD ($\downarrow$) mean/median/std  | IRMSD ($\downarrow$)  mean/median/std |   DockQ ($\uparrow$) mean/median/std |
| -- | -- | -- | -- |
| EBMDock(5)  | 9.18/8.12/6.23 | 9.12/7.23/5.94 |   0.21/0.11/0.20  |
| EBMDock(5)-interface | 2.93/1.65/3.87 | 2.14/1.53/1.54 |  0.62/0.68/0.71 |

Rigid protein-protein docking results on samples from the DIPS-Het test set.

Reviewer comment (Experiments):
The omission of xTrimoDock [1] from the baselines is a notable oversight. According to [1], xTrimoDock can outperform both HDock and AlphaFold-Multimer, and it is also relatively fast. The authors should consider including xTrimoDock in their comparison to provide a more comprehensive evaluation of their proposed method.

Author response:
We conduct comparative testing on the antibody-antigen dataset released in the xTrimoDock[1]. Unfortunately, we are unable to access their publicly available code, which results in differences in the training data and may lead to some unfair comparisons. Since we can not reproduce their experiments, the results related to xTrimoDock[1] in the table are directly extracted from their paper. Also, the inference time is not listed due to the different machines we used and it's not reproducible.

Results on antibody-antigen dataset released in xTrimoDock  (68 samples). Note that DIFFDOCK-PP fails on 4 samples (CRMSD metric exceeds 50Å). The results we presented are obtained after removing these outliers.

| Method      | CRMSD ($\downarrow$) | DockQ ($\uparrow$) |
| ----------- | -------------------- | ------------------ |
| HDOCK                |      15.779±6.364                |           0.090±0.187         |
| AF-Multier           |        13.650±5.886              |             0.108±0.172       |
| EquiDock             |     18.468±2.706                 |           0.043±0.017          |
| xTrimoDock |   10.090±7.817            |        0.220±0.232            |
| DIFFDOCK-PP(5)       |   23.817±7.041       |   0.018±0.064           |
| EBMDOCK(5)           |       14.870±4.930              |            0.054±0.022        |
| EBMDOCK(5)-interface |      6.511±5.126              |           0.324±0.218       |

Reviewer comment (Evaluation):
While the performance of EBMDock-interface is impressive, it relies on the ground-truth binding interface, which might not be a fair comparison to other baselines. This could lead to misunderstandings regarding the method's actual performance. I suggest that the authors clarify this point and consider presenting the results of EBMDock-interface separately in a standalone table to avoid potential confusion.

Author response:
Thanks for your nice suggestion, we will modify this part to avoid confusion. Furthermore, the highlight of our work lies in proposing and utilizing learnable energy for docking, as well as further integrating EBM training and Langevin dynamics sampling for structure prediction. We have not dedicated our efforts to enhancing the interface recognition algorithm. Therefore, we provide results under the assumption that we have a rough knowledge of the interaction interface.

Reviewer comment (Evaluation):
It is intriguing to observe that AF-Multimer outperforms rigid PP docking, especially considering that AF2-Multimer relies solely on sequence inputs and does not incorporate any structural information. Could the author provide an explanation for this phenomena?

Author response:
AlphaFold-Multimer demonstrates higher accuracy compared to structure-based protein complex prediction methods. Here are some possible reasons that can explain this phenomenon:

+ AlphaFold has been trained on a large-scale dataset, while our method's training set only contains around 10,000 data points. AlphaFold benefits from a larger training scale, which allows it to learn more robust representations of protein structures.

+ AlphaFold utilizes sequence homology and evolutionary information for structure prediction. By analyzing a vast amount of protein sequences, the model can identify sequence patterns and co-evolutionary signals that are crucial for understanding protein structure and interactions.

+ AlphaFold requires extensive time for inference to search for homologous information and optimize the structure. Moreover, as the sequence length increases, the inference time and optimization difficulty for AlphaFold also significantly increase.

In summary, the higher accuracy of AlphaFold-Multimer compared to structure-based methods may be explained by its larger training scale, utilization of sequence homology and evolutionary information, as well as the time-consuming nature of its inference and structure optimization processes, which become more challenging with longer sequences.

Reviewer comment (Theory):
While the energy function is modeled as statistical potential using distance likelihoods, the formulation seems rather simplistic. Could more complex energy formulations further improve accuracy?

Author response:
Our energy-based learning framework has good extensibility. As long as an energy term is differentiable, it can be incorporated into our framework easily. Through the experimental results, we find that the HDOCK and AF-Multimer achieve good performance on old proteins because they directly use information from training sets such as templates and sequence alignments. We can design new differentiable energy terms to measure how well the docking poses conform to the templates. We can also design energy terms based on physics, for example, van der Waals dispersion/repulsion, directional hydrogen bond energy, Coulomb electrostatic energy, and desolvation free energy. The calculation of these terms may be expensive, but we can approximate them using deep learning methods. Although these terms may increase the computational cost, we believe they can improve the performance. We will explore these possibilities in future work.

Reviewer comment (Experiments):
For the MDN, why choose a 6-component GMM? Is performance sensitive to the number of mixtures?

Author response:
The performance is not that sensitive to the number of mixtures, we have tried the number range from 4 to 10. We simply take an intermediate value, so we set the number to 6.

Reviewer comment (Experiments):
Have the authors considered extending this to flexible protein docking? What changes would be needed?

Author response:
Flexible docking poses a greater challenge compared to rigid docking, and as far as our knowledge extends, there are currently no machine learning-based approaches capable of effectively addressing the complexities inherent in flexible protein docking problems, especially rigid docking remains an area of extensive research. 

To further extend our method to flexible docking, we need additional degrees of freedom, such as incorporating torsion angle sampling on local protein scaffolds. If so, the complexity of the sampling process would intensify, thereby augmenting the difficulties associated with optimization.

Reviewer comment (Evaluation):
What is the performance on binding site and contact prediction, before pose prediction and docking? How does this performance compare to binding site and contract prediction classifiers?

Author response:
> ***Q1: "What is the performance on binding site and contact prediction, before pose prediction and docking? How does this performance compare to binding site and contract prediction classifiers?"***

|  | AUC | AP |
| --- | --- | --- |
| BSP |  0.86   |  0.67   |
| CP    |   0.95  |   0.58  |

As for the performance of the binding site and contact prediction, the details can be seen in Appendix B. As for other methods typically designed for predicting the binding site of the interface, such as SCANNET、MASIF, we are sorry for not being able to compare with them fairly. Since they only consider the binding sites on one protein, not considering the docking counterpart.

Reviewer comment (Evaluation):
Do the authors have any ideas about what explains the performance gap to HDock and how they could improve the EBMDock protocol? The results suggest that, based on DockQ scores, DiffDock-PP, EquiDock, and EBMDock without providing the ground truth binding site are not usable for pose prediction. This is a valuable result, but it also motivates proposals for substantial improvements.

Author response:
> ***Q2: "Do the authors have any ideas about what explains the performance gap to HDock and how they could improve the EBMDock protocol? The results suggest that, based on DockQ scores, DiffDock-PP, EquiDock, and EBMDock without providing the ground truth binding site are not usable for pose prediction. This is a valuable result, but it also motivates proposals for substantial improvements."***

HDock, PatchDock, and AlphaFold-Multimer use different training data compared to our models. They might have utilized parts of our test sets for extracting templates or as training samples, which may lead to the optimistic results. Therefore, we select 68 antibody-antigen samples from the Protein Data Bank which were released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK.

| Method    | CRMSD ($\downarrow$) | DockQ ($\uparrow$) |
| -------------------- | -------------------- | ------------------ |
| HDOCK   |  15.779±6.364    |   0.090±0.187   |
| AF-Multier  |   13.650±5.886     |   0.108±0.172       |
| EBMDOCK(5)   |    14.870±4.930     |   0.054±0.022        |
| EBMDOCK(5)-interface |      6.511±5.126     |  0.324±0.218       |

We acknowledge that the results are currently not usable, but the protein-protein docking is still a problem far from being solved. Our energy-based learning framework can be improved in many aspects. 

The HDOCK and AF-Multimer achieve good performance on old proteins because they directly use information from training sets such as templates and sequence alignments. We can design new differentiable energy terms to measure how well the docking poses conform to the templates.  We can also design energy terms based on physics, for example, van der Waals dispersion/repulsion, directional hydrogen bond energy, Coulomb electrostatic energy, and desolvation free energy. The calculation of these terms may be expensive, but we can approximate them using deep learning methods. They can be incorporated into our framework easily. Besides, our backbone can be easily replaced by new representation learning methods such as those based on the protein surface. For a particular category of docking problems such as antibody-antigen docking, we can train EBMDock on these proteins to learn a more refined energy function, leading to a usable model. We will explore these possibilities in future work.

Reviewer comment (Evaluation):
The performance of EBMDock is much worse than AlphaFold-multimer (AFM) (DockQ: 0.16 vs 0.48). Although EBMDock runs much faster than AFM, the performance is far from AFM. As a result, EBMDock is not going to be useful in practical setting since its prediction is mostly inaccurate.

Author response:
The AlphaFold method, which is based on multiple sequence alignments (MSAs), is difficult to improve in terms of efficiency and cannot be used for large-scale virtual screening. Even if their performance is good, there are significant limitations when dealing with large-scale data. Our approach aims to improve accuracy while maintaining high inference efficiency. Even though the performance may not match that of AlphaFold, our work has made progress and provided new insights for researchers.

Also, we have claimed that HDock, PatchDock, and AlphaFold-Multimer use different training data compared to our models. They might have utilized parts of our test sets for extracting templates or as training examples, which may lead to an optimistic result. Therefore, we select 68 antibody-antigen samples from the Protein Data Bank which were released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK. We will add these results to our manuscript and hope they can ease your concerns.

| Method               | CRMSD ($\downarrow$) | DockQ ($\uparrow$) |
| -------------------- | -------------------- | ------------------ |
| HDOCK                |      15.779±6.364                |           0.090±0.187         |
| AF-Multier           |        13.650±5.886              |             0.108±0.172       |
| EBMDOCK(5)           |       14.870±4.930             |            0.054±0.022        |
| EBMDOCK(5)-interface |      6.511±5.126              |           0.324±0.218       |

Reviewer comment (Experiments):
EBMDock is only evaluated on rigid-body docking setting, where the ground truth structure of each protein is given. Therefore, its comparison with AFM is not fair because it does not assume the binding site is known and it folds both proteins from sequence directly.

Author response:
We acknowledge that there may be some unfair comparisons when comparing with AlphaFold, but this is due to the inherent differences in our settings. In fact, other structure-based prediction methods also struggle to make a fair comparison with AlphaFold. When it comes to protein docking, we usually have known monomer structures and predict the resulting conformation after docking, rather than predicting the overall structure based solely on the sequence. Although AlphaFold has outperformed all other methods so far, its efficiency makes it challenging for large-scale data. Additionally, while AlphaFold can predict the folded structure even for two proteins that cannot dock, EBMDock identifies the difficulty of docking by observing high energy levels after docking. It is worth noting that as the length of protein sequences increases, the speed of AlphaFold further slows down, and the accuracy noticeably decreases. However, our method primarily incurs increased memory usage, while other metrics remain relatively stable. We hope these points will ease your concern.

Reviewer comment (Evaluation):
In the case where ground truth is not available, we still need to use AF2 to predict the structure of each protein. As a result, the entire workflow for EBMDock won't be much faster than AFM.

Author response:
The setting of our task is to predict complex structures from monomeric structures. While it is true that protein structure is determined by its sequence, in many cases, it is not always necessary to start with the sequence to predict the monomeric structure. Experimental techniques conducted in wet lab can also be employed to determine the structure, and the accuracy achieved through experimental methods undoubtedly surpasses that of prediction. In fact, the monomeric data used for training and testing in our study are derived from experimental measurements rather than predictions. Moreover, to the best of my knowledge, traditional docking software(HDock, etc), including those utilizing deep learning approaches(DIFFDOCK-PP, etc), typically initiate their predictions from monomeric structures, and these endeavors are indeed meaningful.

Reviewer comment (Theory):
Gradient of rotation $\nabla_R E(R, t)$ is not defined clearly.

Author response:
We are sorry we didn't make it clear in the paper. We will explain it in detail below.
In practice, we use the Euler Angle to characterize the rotation. Let $R=[\alpha, \beta, \gamma]^T$, the rotation matrix $R_M$ is calculated as follows:
\begin{aligned}
R_M &= R_{z}(\gamma)R_{y}(\beta)R_{x}(\alpha) \\\ &= \begin{pmatrix}
        \cos(\gamma) & -\sin(\gamma) & 0 \\\        \sin(\gamma) & \cos(\gamma) & 0 \\\        0 & 0 & 1
        \end{pmatrix}
        \begin{pmatrix}
        \cos(\beta) &  0 & \sin(\beta) \\\        0 & 1 & 0 \\\        -\sin(\beta) & 0 & \cos(\beta)
        \end{pmatrix}
        \begin{pmatrix}
        1 & 0 & 0 \\\        0 & \cos(\alpha) & -\sin(\alpha) \\\        0 & \sin(\alpha) & \cos(\alpha)
        \end{pmatrix}
\end{aligned}

The coordinates of the unbound ligand and receptor are $X_L$ and $X_R$, respectively. The docking pose is $(R_M X_L+t, X_R)$. For a residue pair $(li, rj)$ on the binding interface, the distance of them is $d_{i,j}= ||R_M X_{Li} + t -X_{Rj}||^2_2$. Then the energy can be calculated by equation (8) and (9) in the paper. After that, we can get $\nabla_{R}E(R,t)$ by backpropagation.

Reviewer comment (Reproducibility):
Equiformer-PP architecture is not described.

Author response:
We are sorry that we have not elaborated on the architecture of EquiformerPP. We add this part in Appendix E.1, see new pdf for details.

Reviewer comment (Theory):
The method only considers the position of alpha carbon in each residue, which can be a crucial limitation. Being able to model the geometry of full backbone atoms or even sidechain atoms is necessary in real-world scenarios.

Author response:
Since internal structures of residues of the same class are relatively similar and modeling on residue level can be computationally efficient, we choose to use residue-level modeling in our experiment. Our method can be easily extended from residue level to atom level. To extend EBMDock to the atomic level, we only need to modify the nodes in the protein graph from representing a residue to representing an atom. Instead of learning the statistical potential of residue distances, we would learn the statistical potential of atomic distances. The rest of the network structure and training methods remain unchanged. Also, the challenge of protein-protein docking always lies in the prediction of the backbone structure, and we can use existing tools such as Rosetta to complete the side chain conformation. We hope these points will ease your concern.

It should be noticed that most deep learning methods, such as Equidock and DIFFDOCK-PP(our baseline), are also based on modeling at the residue level.

Reviewer comment (Evaluation):
The authors include the DockQ score as one of their evaluation metrics. The original DockQ score includes the percentage of native contacts preserved in the predicted structure as one of criteria. However, the current method ignores all backbone atoms except for alpha carbon. The author should make this point clear.

Author response:
The original definition of DockQ indeed involves the use of all-atom representation, but here we substitute it with alpha-C and calculate it in the same manner for all baseline methods. This means our comparison is still fair although we used an alpha-C version of DockQ. Thanks for pointing this out, we have modified our manuscript to make this point clear.

Reviewer comment (Evaluation):
The proposed method falls short of the AlphaFold-Multimer on both evaluation tasks. Although the authors have mentioned that the results of AFM might be optimistic (test set may overlap with the training set of the AFM), it would be better if they could report the results of both models on another benchmark set composed of proteins released after the cutoff date of the AFM's training set.

Author response:
We select 68 antibody-antigen samples from Protein Data Bank which are released after October 2022 and have not been used to train AlphaFold-Multimer or as templates in HDOCK. Since the template library constructed by HDOCK may have encountered the test set mentioned in the article, we also include it here for a relatively fair comparison. 

| Method               | CRMSD ($\downarrow$) | DockQ ($\uparrow$) |
| -------------------- | -------------------- | ------------------ |
| HDOCK                |      15.779±6.364                |           0.090±0.187         |
| AF-Multier           |        13.650±5.886              |             0.108±0.172       |
| EquiDock             |     18.468±2.706                 |           0.043±0.017          |
| DIFFDOCK-PP(5)       |   23.817±7.041       |   0.018±0.064           |
| EBMDOCK(5)           |       14.870±4.930              |            0.054±0.022        |
| EBMDOCK(5)-interface |      6.511±5.126              |           0.324±0.218       |

The result here demonstrates that both HDOCK and AF-Multimer exhibit a significant decrease in performance on unseen cases. Although EBMDOCK's results also decline, they are close to those of AF-Multimer and surpass HDOCK in CRMSD. Additionally, when EBMDOCK is able to obtain rough docking interface information, the docking results are far superior to both AF-Multimer and HDOCK. We will add these results to our manuscript and hope they can ease your concerns.

Reviewer comment (Experiments):
Training an EBM with contrastive divergence where negative samples are generated by Langevin dynamics on the fly is quite time-consuming. May I know how much time it takes to train the EBMDock and have you tried any other techniques to reduce the cost?

Author response:
We take almost three days to train our EBMDock till converge on a machine with i9-10920X CPU, RTX 3090 GPU, and 128G RAM, which we think is acceptable. We have tried to use a fixed number of MCMC steps (typically fewer than required for convergence) to reduce the cost, since running Langevin MCMC till convergence to obtain an unbiased sample from $x \sim p_{\theta}(x)$ can be computationally expensive. We set the number of steps to 15.

Reviewer comment (Experiments):
Another straightforward idea of rigid Protein-protein docking is that we first train an inter-residue distance predictor and then recover the docking poses that best fit the predicted inter-residue distance map via gradient descent. Have you tried comparing EBMDock with such a model?

Author response:
Yes, we have tried to predict the distance map of residues directly instead of using the Gaussian mixture model. However, directly fitting the distance has poor performance and the model lacks good generalizability.