Paper: Thermodynamics-inspired Structure Hallucination for Protein-protein Interaction Modeling (ICLR-2024)

Reviewer comment (Theory):
The PDC module, while a very interesting idea, is on shaky ground mathematically.

Author response:
-- We appreciate your diligence in scrutinizing the mathematical foundation of our PDC module and also acknowledge that the thermal fluctuations and epistemic uncertainty of neighboring atoms certainly should be dependent. However, if we assume that the positions of particles are highly correlated to their neighbors, the mathematical analysis of their uncertainty becomes intricate. Just as you mentioned, we first need to hypothesize the cross-covariance of each pair of entities in the molecular system so that the self-covariance of each atom can be calculated. A major challenge lies in computing geometric variables like distances and angles. For instance, once $\mathbf{x}_i$ and $\mathbf{x}_j$ 

follow dependent Gaussian distributions, the squared norm of their difference $d_{ij}$  will deviate from a generalized chi-square distribution $\chi^2(.)$. Due to our mathematical limitations, explicitly defining a set of natural parameters to model the exact distribution of $d_{ij}$ is not feasible. Additionally, portraying higher-order geometries, such as the angle between vectors $\mathbf{x}_i - \mathbf{x}_j$ and $\mathbf{x}_i - \mathbf{x}_k$, becomes more challenging. We really understand and align with your perspective that close atoms have a significant impact on each other in the 3D space, and plan to address this in future work by introducing a more nuanced architecture. It is undeniable that separating individual particles into isolated elements provides great convenience for us to model complex molecular systems. More importantly, the interdependency between close entities will be captured via the following message-passing-based mechanism when training geometric networks.

Reviewer comment (Theory):
The authors assume all atom positions to be independent, which is a very questionable assumption as the thermal fluctuations and epistemic uncertainty of neighboring atoms certainly should be dependent.

Author response:
-- We appreciate your diligence in scrutinizing the mathematical foundation of our PDC module and also acknowledge that the thermal fluctuations and epistemic uncertainty of neighboring atoms certainly should be dependent. However, if we assume that the positions of particles are highly correlated to their neighbors, the mathematical analysis of their uncertainty becomes intricate. Just as you mentioned, we first need to hypothesize the cross-covariance of each pair of entities in the molecular system so that the self-covariance of each atom can be calculated. A major challenge lies in computing geometric variables like distances and angles. For instance, once $\mathbf{x}_i$ and $\mathbf{x}_j$ 

follow dependent Gaussian distributions, the squared norm of their difference $d_{ij}$  will deviate from a generalized chi-square distribution $\chi^2(.)$. Due to our mathematical limitations, explicitly defining a set of natural parameters to model the exact distribution of $d_{ij}$ is not feasible. Additionally, portraying higher-order geometries, such as the angle between vectors $\mathbf{x}_i - \mathbf{x}_j$ and $\mathbf{x}_i - \mathbf{x}_k$, becomes more challenging. We really understand and align with your perspective that close atoms have a significant impact on each other in the 3D space, and plan to address this in future work by introducing a more nuanced architecture. It is undeniable that separating individual particles into isolated elements provides great convenience for us to model complex molecular systems. More importantly, the interdependency between close entities will be captured via the following message-passing-based mechanism when training geometric networks.

Reviewer comment (Theory):
It is not possible to write down the self-covariance of the difference of random variables without knowing the cross-covariances. However, even assuming the positions to be independently distributed, then if their mean is updated as in Eq 6, then Eq 7 should read $${\sigma_{x_i}^{(l+1)}}^2 = \left[1 + \frac{1}{|N(i)|}\sum_{j \in N(i)} \phi_\mu(m_{j \rightarrow i})\right]^2{\sigma_{x_i}^{(l)}}^2 + \frac{1}{|N(i)|}\sum_{j \in N(i)}{\sigma_{x_j}^{(l)}}^2\phi_\mu(m_{j \rightarrow i})$$ i.e., with $\phi_\mu$ instead of a different $\phi_\sigma$, and distributing and squaring the $x_i$ terms because $x_i - x_j$ is not independent of $x_i$.

Author response:
-- Thanks for your advice on the update of coordinate variance! It is rational to use the same operation $\phi_\mu$ for renewing both mean and variance rather than employing separate operations (i.e., $\phi_\mu$ and $\phi_\sigma$). In addition, we concur with your suggestion to square the $\mathbf{x}_i$ terms since the vector $\mathbf{x}_i - \mathbf{x}_j$ is dependent on $\mathbf{x}_i$. Experiments show that this form of position variance computation performs slightly better in the mutant effect prediction task (without pretraining on PDB). We have included this new Equation in the Appendix of our revised manuscript and would like to express our gratitude for your insight on PDC-EGNN. We believe the introduction of positional uncertainty is a promising direction, worthy of further exploration and the proposal of advanced architectures.  
|  Method  | Per-Structure | Per-Structure |
|:--------:|:-------------:|:-------------:|
|          |    Pearson    |    Spearman   |
|  Equ. 7  |     0.4475    |     0.4102    |
| New Equ. |     **0.4490**    |    **0.4153**    |

Reviewer comment (Theory):
My general concern is that the authors have not given sufficient treatment to the mathematical subtleties of the PDC module, and hence, the module is not actually constrained to model the atomic uncertainties as the authors claim; rather, to the network ${\sigma_{x_i}^{(l+1)}}^2$ just looks like some other latent variable which it can use to help model the positional updates. It would be better to call the module “loosely inspired” by the modeling of uncertainty. If the authors nevertheless claim that this is a generally helpful modification to EGNN, this is a claim that requires significantly more thorough evaluation (on many different tasks ideally) than is given here.

Author response:
(3) We really understand your concern about the justification for the correspondence between learned "variance" and positional uncertainty. We agree that our loss term primarily influences output positions without directly enforcing the network to capture uncertainty information.  However, it's essential to note that our PDC-module design (the final name is to be determined) is theoretically grounded in the concept of atomic uncertainty. To elucidate, posit that all atoms adhere to a Gaussian distribution and derive the geometric attributes such as distance and angles, which are also represented as distributions. Then our PDC-module is proposed to allow the propagation of these distributions and update their natural parameters during training. From this standpoint, we firmly believe **this latent variance $\Sigma$ corresponds to and at least encodes a form of uncertainty**. 

We acknowledge your desire for straightforward proof that exhibits the strong correspondence between our proposed "variance" and uncertainty. However, experimentally observing and documenting particle uncertainty within macromolecules, such as proteins, is exceedingly challenging, if not impossible. All existing data in PDB or Skempi v2 are deterministic and uncertainty-free conformations, making it difficult to quantitatively measure uncertainty and evaluate our architecture's effectiveness in capturing atomic uncertainty. 

As a solution, we resort to computational methods like molecular dynamics (MD) simulations to simulate the atomic motions. Notably, MD simulations approximate atomic motions by Newtonian physics and can capture the sequential behavior of molecules in full atomic details at a very fine temporal resolution, quantifying how much various regions of the molecule move at equilibrium and what types of fluctuations they undergo. We run short-time MD simulations for all complex structures and calculate the Root Mean Square Fluctuation (RMSF) alongside the entire trajectory. It is worth mentioning that **RMSF is a numerical measurement similar to RMSD, but instead of indicating positional differences between entire structures over time. It is a calculation of individual residue flexibility, or how much a particular residue moves (fluctuates) during a simulation [A, B]. Therefore, we use RMSF as a measurement of the atomic uncertainty**, and then navigate three approaches to demonstrate the capability of our PDC module to encode uncertainty using this MD-based data.

First, we explore three sorts of variance initialization mechanisms. Specifically, i) all $\Sigma$ are identity matrices; ii) $\Sigma$ are learnable; iii) $\Sigma$ are initialized by RMSF from MD simulations. Results show that MD-based initialization achieves the best Spearman (0.4287), outweighing the learnable one (0.4102) and identity matrix (0.4043), emphasizing the efficacy of incorporating simulated uncertainty into the PDC module. This implies that simulated uncertainty is the optimal choice for this variance, and learned variance ideally should move towards this simulated uncertainty.

Reviewer comment (Reproducibility):
There are several critical missing details in the methodology and experiments (see Questions below).

Author response:
(1) Methodology Details 

-- To elucidate, the backbone module $h_\rho$ can take the form of any conventional geometric neural network (e.g., GVP-GNN, EGNN, SE(3)-Transformer, Graph-Transformer). Here, we adopt a one-layer Graph Transformer [A, B] to extract general representations of proteins. The refinement model $f_\theta$ needs to output both updated features and coordinates, and, therefore, **we use the PDC module as $f_\theta$** in our experiments. Lastly, the head predictor $g_\tau$ is a simple linear layer that accepts the concatenation of both wide-type and mutation-type representations and forecasts the free energy change. The total model size of Refine-PPI is approximately 6M. 

[A] Shan, Sisi, et al. "Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization." Proceedings of the National Academy of Sciences 119.11 (2022): e2122954119.

[B] Luo, Shitong, et al. "Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures." Advances in Neural Information Processing Systems 35 (2022): 9754-9767.

Specifically, we investigate three sorts of initialization mechanisms for $\sigma$. First and naively, we turn all $\sigma$ to be equivalent to one (a unit length). Second, we depend on physical principles and utilize molecular dynamic (MD) simulations to attain the short motion trajectories (10 nano-seconds) of these complexes in the 3D space. Then we calculate the root-mean-square fluctuation (RMSF) of each amino acid and take this value as the initial input of $\sigma$. Third, we adopt a learnable strategy to initialize $\sigma$. To be explicit, an embedding layer is created for each category of 20 residue types to a 3-dimensional continuous vector. This routine learns the variance of different components completely from the data. 

The performance of different initialization approaches is listed below, and it can be found that constant initialization is the worst. Besides, the MD-based methodology outperforms slightly better results than the embedding-based one. However, since MD simulations are time-consuming and costly, it is prohibited to implement MD during the inference stage each time. As a consequence, we use the third sort of initialization method in our paper. In the revised manuscript, we provide a more detailed explanation of the initialization process in the Appendix. 
|       Method       | Per-Structure | Per-Structure |
|:------------------:|:-------------:|:-------------:|
|                    |    Pearson    |    Spearman   |
|     Unit Length    |     0.4422    |     0.4043    |
|   MD Simulations   |    **0.4522**   |   **0.4287**  |
| Learnable Variance |     0.4475    |     0.4102    |

-- Thanks for your advice. We have provided proof in the Appendix. It would be helpful for readers who are interested in the equivariance property of our model. 

-- Thanks for your question about the gradient backpropagation. Notably, since the MMM task and the $\Delta\Delta G$ task share the same encoder $h_\rho$, the MMM task here is an auxiliary training task instead of a pretraining task. We have tried to allow gradient backpropagation from the $\Delta\Delta G$ task to the structure refinement module $f_\theta$ but found the training process unstable with slightly worse performance. This phenomenon can be attributed to the small number of available experimental data in the downstream Skempi v2 dataset. It can be expected that if given more ground truth complex structures, the training procedure would be stabilized and the allowance of gradient backpropagation can lead to enhanced results.

Reviewer comment (Theory):
The authors claim (Eq 1) to recover the structural distribution of the masked residues, but the training objective is risk minimization (Eq 3), not any kind of distributional modeling objective.

Author response:
-- Thanks for your question regarding the structural distribution recovery. You are right that our aim is to recover the structural distribution of the masked residues, namely, $p(\{\mathbf{x}^{\textrm{WT}}\}^{m+r}_{i=m-l} \big| \tilde{\mathcal{G}}^{\textrm{WT}}, a_m, \theta, \rho )$. 

In order to supervise the MMM task, the exact training object would be measuring and minimizing the gap between the predicted structural distribution $Q$ and the ground truth structural distribution $P$. A widely used measure is the KL-divergence, writtern as $D_{KL}(P||Q) = \mathbb{E}_{x\sim P}\big[log \frac{P(X)}{Q(X)}\big]$. Then if our goal is to learn a model $f:\mathcal{X} \rightarrow \mathcal{Y}$, our objective becomes 

${\textrm{argmin}}_ \theta  \mathbb{E}_ {x,y\sim\mathcal{D}} [-logQ_ \theta (y|x)].$

For classification problems, the cross-entropy loss is exactly what KL divergence minimizes. On the contrary, for regression task, minimizing the negative log likelihood (NLL) of this normal distribution is clearly equivalent to the mean-squared-error loss, namely, Equ.3 in our paper. For more information, please refer to this wonderful blog: https://dibyaghosh.com/blog/probability/kldivergence.html.

Reviewer comment (Theory):
The authors state that MMM “encourages graph manifold learning with the denoising objective”, but there is no further discussion or elaboration on how “graph manifold learning” is accomplished.

Author response:
-- We appreciate your clarification on the use of terminology. Please allow us to address the point you raised about the claims related to the graph manifold learning. To be specific,  By using an implicit mapping from corrupted data to clean data, the MMM objective encourages the model to learn the manifold on which the clean data lies — we speculate that the deep learning model learns to go from low probability graphs to high probability graphs. In the denoising-based initialization case, Refine-PPI learns the manifold of the input data. When node targets are provided, the model learns the manifold of the target data (e.g. the manifold of atoms at equilibrium). We speculate that such a manifold may include commonly repeated substructures that are useful for downstream prediction tasks. A similar motivation can be found in prior denoising-based molecular pretraining methods [A, B, C].

[A] Godwin, Jonathan, et al. "Simple gnn regularisation for 3d molecular property prediction & beyond." ICLR 2022.

[B] Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." Journal of machine learning research 11.12 (2010).

[C] Song, Yang, and Stefano Ermon. "Generative modeling by estimating gradients of the data distribution." Advances in neural information processing systems 32 (2019).

Reviewer comment (Writing):
There is repeatedly misleading use of the term “thermodynamics” when the authors mean “uncertainty.” The former term should be reserved when explicitly referring to physically meaningful quantities like energy, entropy, and free energy.

Author response:
--  We appreciate your guidance on the distinction between "thermodynamics" and "uncertainty." We will rectify the repeated misuse of the term "thermodynamics" in situations where "uncertainty" is more appropriate. This correction will ensure a more accurate and precise representation of our concepts.

Reviewer comment (Evaluation):
The statement “The pictures show that particles in the interface have smaller variation compared to those in the edges of proteins” is not backed quantitatively, and, as discussed, there is no reason to believe that the learned $\sigma^2$ actually corresponds to uncertainty.

Author response:
-- -- We acknowledge your concern about the need for quantitative support for certain claims, particularly in relation to the variation of particles at the protein interface. To address this, we conducted short trajectories of all structures in Skempi V2, calculating the Root Mean Square Fluctuation (RMSF) of each residue. Additionally, we determined the magnitude of $||\boldsymbol{\sigma}_{\mathbf{x}_i}||^2$, which is the learned variance of our uncertainty module. A detailed comparison of these values, categorized by residues at and not at the interface, is presented in the table below. Notably, the ground truth RMSF at the interface is significantly smaller than that observed elsewhere. 

Concurrently, the learned  $\boldsymbol{\sigma}$ exhibits a parallel pattern, where $||\boldsymbol{\sigma}_{\mathbf{x}_i}||^2$ at the interface is much smaller. This quantitative analysis serves to substantiate the claim that the learned variance indeed corresponds to uncertainty. We appreciate your suggestion to quantify the uncertainty, as it has enhanced the rigor of our analysis!

|     | Interface | Non-Interface |
|:-----------------------------:|:---------:|:-------------:|
|     RMSF in MD Simulations    |   0.4945  |     0.9735    |
| Magtitude of Learned Variance |   0.6072  |     0.8940    |

Reviewer comment (Evaluation):
The statement “It can be found that generally, a small error of wide-type structure reconstruction leads to a more accurate $\Delta\Delta G$ prediction” is not supported, as there is no such correlation in Figure 4B.

Author response:
-- We acknowledge the discrepancy you pointed out regarding the statement on a correlation between a small error in wide-type structure reconstruction and a more accurate prediction in Figure 4B. Upon further examination, we realize the oversight in the interpretation. We will detele the corresponding text for clarity and accuracy.

Reviewer comment (Evaluation):
The model only slightly outperforms previous state-of-the-art RDE-Net on a subset of metrics. It seems that overall performance of Refine-PPI and RDE-Net is similar.

Author response:
(1) We appreciate your honesty in highlighting the weaknesses of our work and understand your concern regarding the marginal improvement over the previous state-of-the-art model, RDE-Net. However, it is worth noting that RDE is pretrained on the additional protein structure dataset, PDB-REDO, which contains more than 130K refined X-ray structures in Protein Data Bank. Meanwhile, our Refine-PPI adopts no pretraining strategy and is directly trained on Skempi v2 with only 345 wide-type structures. Refine-PPI enjoys no benefits of unsupervised pretraining but achieves competitive or even better performance than existing algorithms, underscoring the superiority of our architecture design. As mentioned in Appendix C, several prior studies [A, B, C] have demonstrated that structural pretraining is beneficial to dramatically expand the representation space of deep learning models, and it is promising to pretrain Refine-PPI with more experimental protein structures and transfer the knowledge to this mutation effect prediction task. If we leverage the same PDB-REDO for pretraining, the per-structure Spearman correlation will significantly increase from 0.41 to 0.44 (see Table below for a clear comparison). 

Moreover, as declared in both our paper and RDE-Net, the correlation for one specific protein complex is often of greater interest and importance in practical applications. It is preferred to attach more attention to the average per-structure Pearson and Spearman correlation coefficients rather than the overall metrics. 

|   Method   | Pretrain | Per-Structure | Per-Structure |   Overall  |   Overall  |   Overall  |   Overall  |   Overall  |
|:----------:|:--------:|:-------------:|:-------------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|            |          |    Pearson    |    Spearman   |   Pearson  |  Spearman  |    RMSE    |     MAE    |    AUROC   |
|   RDE-Net  |    Yes   |     0.4448    |     0.4010    |   0.6447   |   0.5584   |   0.5799   |   1.1123   |   0.7454   |
| Refine-PPI |    No    |     0.4475    |     0.4102    |   0.6584   |   0.5394   | **1.5556** | **1.0946** |   0.7517   |
| Refine-PPI |    Yes   |   **0.4561**  |   **0.4374**  | **0.6592** | **0.5608** |   1.5643   |   1.1093   | **0.7542** |


[A] Zhang, Zuobai, et al. "Protein representation learning by geometric structure pretraining." arXiv preprint arXiv:2203.06125 (2022).

[B] Chen, Can, et al. "Structure-aware protein self-supervised learning." Bioinformatics 39.4 (2023): btad189.

[C] Wu, Fang, et al. "Pre‐Training of Equivariant Graph Matching Networks with Conformation Flexibility for Drug Binding." Advanced Science 9.33 (2022): 2203796.

Reviewer comment (Reproducibility):
The description of probabilistic density cloud representation is a bit unclear. In particular, how are the $sigma$'s initialized? If they are initialized as zero, then they will stay zero all the time. Are they initialized by some physical calculations?

Author response:
(3) Regarding the initialization of probabilistic density cloud representation, we agree that our description needs clarification. Specifically, we investigate three sorts of initialization mechanisms for $\sigma$. First and naively, we turn all $\sigma$ to be equivalent to one (a unit length). Second, we depend on physical principles and utilize molecular dynamic (MD) simulations to attain the short motion trajectories (10 nano-seconds) of these complexes in the 3D space. Then we calculate the root-mean-square fluctuation (RMSF) of each amino acid and take this value as the initial input of $\sigma$. Third, we adopt a learnable strategy to initialize $\sigma$. To be explicit, an embedding layer is created for each category of 20 residue types to a 3-dimensional continuous vector. This routine learns the variance of different components completely from the data. 

The performance of different initialization approaches is listed below, and it can be found that constant initialization is the worst. Besides, the MD-based methodology outperforms slightly better results than the embedding-based one. However, since MD simulations are time-consuming and costly, it is prohibited to implement MD during the inference stage each time. As a consequence, we use the third sort in our paper. In the revised manuscript, we will provide a more detailed explanation of the initialization process (e.g., in the Appendix). 

|       Method       | Per-Structure | Per-Structure |
|:------------------:|:-------------:|:-------------:|
|                    |    Pearson    |    Spearman   |
|     Unit Length    |     0.4422    |     0.4043    |
|   MD Simulations   |    **0.4522**    |   **0.4287**    |
| Learnable Variance |     0.4475    |     0.4102    |

Reviewer comment (Evaluation):
For the first task of mutation structure prediction task, can you report the RMSD between predicted mutated structure and ground truth?

Author response:
(2) For the first task of mutation structure prediction, we apologize for not reporting the RMSD between the predicted mutated structure and the ground truth. This information is, indeed, very crucial. However, it is worth mentioning that obtaining the structures of mutant proteins is a persistent challenge, as they are often elusive to acquire. All mutant structures in Skempi V2 are not available and we only have 345 wide-type structures. Therefore, we are afraid that we are unable to report the exact structure prediction error between the predicted mutated structure and ground truth. Notably, the inaccessibility of mutant structures is the core motivation for our structure refinement module, which is first trained by a mask mutation modeling (MMM) task on available wide-type structures and then transferred to hallucinate the inaccessible mutant protein structures. During the MMM training phase, the RMSD between predicted wide-type structures and ground truth is 1.437, which will increase if we enlarge the length of the masked region. We hope this statistic can provide insight into the modeling accuracy and partially answer your question.

Reviewer comment (Evaluation):
Weak benchmark performance: the elevation of performance is too marginal (especially those in the Appendix), and the time complexity is not studied. No variance is reported.

Author response:
We appreciate your honesty in highlighting the weaknesses of our work and understand your concern regarding the marginal improvement over the previous state-of-the-art model, which has also been mentioned by Reviewer xfqG. However, it is worth noting that RDE is pretrained on the additional protein structure dataset, PDB-REDO, which contains more than 130K refined X-ray structures in Protein Data Bank. Meanwhile, our Refine-PPI adopts no pretraining strategy and is directly trained on Skempi v2 with only 345 wide-type structures. Refine-PPI enjoys no benefits of unsupervised pretraining but achieves competitive or even better performance than existing algorithms, underscoring the superiority of our architecture design. As mentioned in Appendix C, several prior studies [A, B, C] have demonstrated that structural pretraining is beneficial to dramatically expand the representation space of deep learning models, and it is promising to pretrain Refine-PPI with more experimental protein structures and transfer the knowledge to this mutation effect prediction task. If we leverage the same PDB-REDO for pretraining, **the per-structure Spearman correlation will significantly increase from 0.41 to 0.44** (see Table below for clear comparison). Notably, We emphasize the importance of per-structure metrics over overall metrics for practical applications.

|   Method   | Pretrain | Per-Structure | Per-Structure |   Overall  |   Overall  |   Overall  |   Overall  |   Overall  |
|:----------:|:--------:|:-------------:|:-------------:|:----------:|:----------:|:----------:|:----------:|:----------:|
|            |          |    Pearson    |    Spearman   |   Pearson  |  Spearman  |    RMSE    |     MAE    |    AUROC   |
|   RDE-Net  |    Yes   |     0.4448    |     0.4010    |   0.6447   |   0.5584   |   0.5799   |   1.1123   |   0.7454   |
| Refine-PPI |    No    |     0.4475    |     0.4102    |   0.6584   |   0.5394   | **1.5556** | **1.0946** |   0.7517   |
| Refine-PPI |    Yes   |   **0.4561**  |   **0.4374**  | **0.6592** | **0.5608** |   1.5643   |   1.1093   | **0.7542** |

Besides, as for the time complexity of Refine-PPI, we empirically verify that it has little difference from traditional deep learning-based methodologies. To be explicit, it spends RDE and Refine-PPI about 60 and 66 seconds, separately, to perform 3-fold inference predictions of all 6.7K entries in Skempi V2. The generation process of mutant structures is very fast, making it feasible for real-world applications

|   Method   | Inference Time | 
|:----------:|:--------:|
|   RDE-Net  | 60 s | 
| Refine-PPI | 66 s | 

[A] Zhang, Zuobai, et al. "Protein representation learning by geometric structure pretraining." ICLR 2023.

[B] Chen, Can, et al. "Structure-aware protein self-supervised learning." Bioinformatics 39.4 (2023): btad189.

[C] Wu, Fang, et al. "Pre‐Training of Equivariant Graph Matching Networks with Conformation Flexibility for Drug Binding." Advanced Science 9.33 (2022): 2203796.

Reviewer comment (Writing):
The usage of term "thermodynamics" and "hallucination" is hardly relevant to the proposed method and thus confusing. I would suggest the authors to use plainer descriptions.

Author response:
ii) Previous studies exemplified by Google’s DeepDream train networks to recognize faces and other patterns in images, and invert and adjust arbitrary input images to draw more strongly resemble faces or other patterns as perceived by the network. The generated images are often referred to as **hallucinations** because they may not faithfully represent any actual face, but what DL models view as an ideal face. Remarkably, this mechanism has also demonstrated success in the context of macromolecules. It has been shown that the information stored in the many parameters of the trained networks can be harnessed to design new protein structures featuring new sequences. A lot of recent works [A, B, C, D] published in top journals have used deep network hallucination to generate a wide range of functional proteins. In our algorithm, we take a similar methodology and explore whether networks trained on existing wide-type structures could be inverted to generate brand new “ideal” protein structures according to the mutant information.

[A] Anishchenko, ... & Baker, D. (2021). De novo protein design by deep network hallucination. Nature, 600(7889), 547-552.

[B] Wicky, ... & Baker, D. (2022). Hallucinating symmetric protein assemblies. Science, 378(6615), 56-61.

[C] An, Linna, et al. "Hallucination of closed repeat proteins containing central pockets." Nature Structural & Molecular Biology (2023): 1-6.

[D] Costello, Zak, and Hector Garcia Martin. "How to hallucinate functional proteins." arXiv preprint arXiv:1903.00458 (2019).

-------------------------------
However, while we find these terms conceptually appropriate, we fully understand that these words can be confusing for some readers. We would like to replace them with more precise and relevant descriptors for improved clarity. Do you have any suggestions or recommendations for the title? We are very pleased to modify it with your opinion.

Reviewer comment (Reproducibility):
How are $h_\rho, f_\theta, g_\tau$ built precisely? What are the $\phi$ functions in Eq 5-7?

Author response:
First and foremost, we apologize for any confusion. To elucidate, the backbone module $h_\rho$ can take the form of any conventional geometric neural networks (e.g., GVP-GNN, EGNN, SE(3)-Transformer, Graph-Transformer). Here, we adopt a one-layer Graph Transformer [A,B] to extract general representations of proteins. The refinement model $f_\theta$ needs to output both updated features and coordinates, and, therefore, we use PDC-EGNN as $f_\theta$ in our experiments. Lastly, the head predictor $g_\tau$ is a simple linear layer that accepts the concatenation of both wide-type and mutation-type representations and forecasts the free energy change. The total model size of Refine-PPI is approximately 6M. In addition, $\phi_e, \phi_h, \phi_{\mu}, \phi_\sigma$ are the edge, node, mean, and variance operations respectively that are commonly approximated by Multilayer Perceptrons (MLPs). We will provide explicit details on the construction of these notations to enhance understanding in our revised paper. Thanks for your question to improve the clarity. 

[A] Shan, Sisi, et al. "Deep learning guided optimization of human antibody against SARS-CoV-2 variants with broad neutralization." Proceedings of the National Academy of Sciences 119.11 (2022): e2122954119.

[B] Luo, Shitong, et al. "Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures." Advances in Neural Information Processing Systems 35 (2022): 9754-9767.

Reviewer comment (Theory):
How are $\sigma_{x_i}$s modeled? And how are those initialized? In Eq 4 they are matrices while in Eq 7 they seem to be vectors. In my opinion 3d variance should either be a scalar or a learnable SPD matrix. Using vectors does not satisfy SE(3) invariance because an ellipsoid with standard axis is presumed, and the results are thus varied when rotations are applied to the input structure. Thus the method is not "readily apparent" to be equivalent.

Author response:
(4)  We appreciate your clarification on the use of $\sigma$ and how to initialize them in our probability density cloud representation. 
i) You are correct that $\sigma$ are matrices in Equ. 4 while are vectors in Equ. 7. The main reason is that $\boldsymbol{\Sigma}_i\in \mathbb{R}^{3\times 3}$ is a diagonal covariance matrix indicating that different axes are independent of each other. Consequently, we utilize $\boldsymbol{\sigma}$ to denote the diagonal elements of the variance matrix $\mathbf{\Sigma}$ for simplicity.  we will incorporate this into our revised manuscript.

ii) Specifically, we investigate three sorts of initialization mechanisms for $\sigma$. First and naively, we turn all $\sigma$ to be equivalent to one (a unit length). Second, we depend on physical principles and utilize molecular dynamic (MD) simulations to attain the short motion trajectories (10 nano-seconds) of these complexes in the 3D space. Then we calculate the root-mean-square fluctuation (RMSF) of each amino acid and take this value as the initial input of $\sigma$. Third, we adopt a learnable strategy to initialize $\sigma$. To be explicit, an embedding layer is created for each category of 20 residue types to a 3-dimensional continuous vector. This routine learns the variance of different components completely from the data. 

The performance of different initialization approaches is listed below, and it can be found that constant initialization is the worst. Besides, the MD-based methodology outperforms slightly better results than the embedding-based one. However, since MD simulations are time-consuming and costly, it is prohibited to implement MD during the inference stage each time. As a consequence, we use the third sort in our paper. In the revised manuscript, we will provide a more detailed explanation of the initialization process (e.g., in the Appendix). 

--------------------
|       Method       | Per-Structure | Per-Structure |
|:------------------:|:-------------:|:-------------:|
|                    |    Pearson    |    Spearman   |
|     Unit Length    |     0.4422    |     0.4043    |
|   MD Simulations   |     0.4522    |     0.4287    |
| Learnable Variance |     0.4475    |     0.4102    |

Reviewer comment (Presentation):
There lacks a visualization of the "hallucinated" structures.

Author response:
(5) We acknowledge the absence of visualization for "hallucinated" structures. In the revision, we will include visualizations (see Appendix) to provide a clearer understanding of the proposed method.

Reviewer comment (Experiments):
On what data, precisely, is Refine-PPI trained? The description seems to point to a 3-fold cross validation, but the concrete splits should be specified. And, since all benchmark performances are "directly copied" from a preprint, the authors must justify that their evaluation scheme is exactly the same to all benchmarks.

Author response:
(6) We apologize for any confusion regarding the training data and evaluation scheme. In our evaluation, we adopt the same 3-fold cross-validation split as RDE-Net. To be specific, the numbers of training and validation samples for fold-0, fold-1, and fold-2 are 4777/1929, 4290/2416, and 4345/2361. We confirm that our splitting strategy completely accords with all benchmarks reported in the RDE paper. Besides that, we would like to clarify that RDE was officially accepted by last year's ICLR [A] and it is our fault to use its preprint version as the reference. 

[A] Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction. ICLR 2023. https://openreview.net/forum?id=_X9Yl1K2mD