Paper: Functional Geometry Guided Protein Sequence and Backbone Structure Co-Design (ICLR-2024)

Reviewer comment (Experiments):
For the experiments, how is the model trained to obtain the evaluation digits in Table 1 for each baseline. This is very important because as far as I acknowledged, all the baselines were simply pre-trained on general protein structure datasets such as PDB and CATH, plus the ProteinMPNN(IF). However, in the section 4.2, the author(s) mentioned that NAEPro, the proposed method, is trained on domain-specific dataset from beta-lactamase and myoglobin before inference stage. Do the author believe that such treatments for baseline comparison is fair? To contextualize, most of the protein design or engineering task (antibody/nanobody/enzyme; mutation effect) suffer from lack of domain-specific data, which therefore hinders the use of ML models. The presence of such specific dataset can have a significant effect on the performance of models. Therefore, the result shown in Table 1 may not as strong as it claimed.

Author response:
We are sorry for the confusion. We got the codes of all baselines from their GitHub repos, and then re-trained all the models separately on myoglobin and beta-lactamase datasets. The evaluation procedure among all the baselines and our model follows the same process, and thus we believe the comparisons in Table 1 are fair.

Reviewer comment (Evaluation):
For the evaluation metrics, the problem to be studied is functional protein (as it is claimed in the title) design instead of general backbone generation (eg., RFDiffusion) or sequence generation (eg., ProteinMPNN). Thus, I found the the five of the metrics fail to (or indirectly if they potentially do) reflect the performance of functional protein design or scaffolding. For the common practice of protein engineering, one may conduct wet-lab validation for the designed candidates (such as in ProteinMPNN) or use a fitness regression model to approximately indicate the success rate of the inference output. All these metrics in Table 1 & 2 show the somehow sequence-structure matching between the predicted and native, but are not very suitable for proteins such as enzyme.

Author response:
In our analysis (Section 5.2), we evaluated the function for binding the corresponding metallocofactors of the designed proteins. Specifically, we first randomly select 20 cases from the top100 sequences (pLDDT ranking). We then employ AlphaFold2 for protein structure prediction, followed by inputting these structures into AlphaFill [1] to predict the associated ligands. Notably, our results indicate that all designed myoglobins exhibit heme binding capability, while 18 beta-lactamases demonstrate the ability to bind zinc ions, resulting in metallocofactor binding rates of 100% and 90%, respectively. It is evident that our method can generate proteins that express the basic and important functions, i.e. binding the corresponding metallocofactors. To systematically illustrate the function of our designed proteins, we use docking tool Gnina[2] to compute the binding affinity between the designed metalloproteins and the corresponding metallocofactors. The results in the following table shows that our model achieves the best binding affinity scores on both metalloproteins, demonstrating the proteins designed by our model are highly potential to actively exhibit biochemical functions.

|Model| myoglobin (kcal/mol)|beta-lactamase (kcal/mol)|
|:-----|:----:|:----:|
|Hallucination|-7.23|-6.37|
|Inpainting|-9.32|-7.63|
|SMCDiff+ProteinMPNN|-8.79|-6.89|
|PROTSEED|-10.07|-7.68|
|FrameDiff+ProteinMPNN|-9.54|-7.21|
|RFDiffusion+ProteinMPNN|-9.76|-7.59|
|NAEPro|**-10.14**|**-7.80**|

Reviewer comment (Theory):
The potential theoretical flaw in the equivariance analysis in Section 3.5. To contextualize, the SE(3)-equivariance represents the equivariant property of functions with respect to elements in the special Euclidean group in R^3 which encompass only translation and rotation transformations (or roto-translation equivariance). Corresponding, the special orthogonal group mentioned in Theorem 3.1 also excludes the reflection matrices which has determinant equal to -1. However, in the Theorem 3.1., the author claimed SE(3)-equivariance yet let the proposed NAEL layer be rotation or reflection equivariant. Suppose it is a typo, however, assuming reflection-equivariant property can be problematic for protein structure, because the protein chirality can be important for helical structure.

Author response:
Thank the reviewer for pointing out this typo. Our model satisfies only roto-translation equivariance without reflection equivariance. We have accordingly updated the paper.

Reviewer comment (Novelty):
Significance: It is likely that previous works can be readily applied to the motif-conditioned setting, and the authors can easily adapt their method to antibody design tasks. It would be beneficial if the authors further elaborate on the significance of their work.

Author response:
We would like to clarify our goal more clearly. Our task is to co-design functional and novel protein sequence and structure. Previous work which is most similar to ours is Inpainting [1]. However, [1] provides the functional sites manually while we automatically mined the functional sites (plus the conserved sites as suggested by reviewer ****VH6U****) by MSAs. Previous methods considering only sequence design constrained by fitness value [2,3] or sequence design based on given backbone structure (inverse folding) [4] is not suitable for our task. For antibody design, antibody is always Y-shaped, while one of our goal is to design novel and diverse protein structure. For example, in Figure 4, the designed beta-lactamases have different fold categories and belong to three different sub-classes. In Figure 5 (b), the designed myoglobin has a RMSD of 3.943\AA with the most similar one in PDB, but it can still bind the corresponding metallocofactor heme in the AlphaFill simulation [5]. From this aspect, we think antibody design models can not be directly applied to our task without any modification and vice versa. 

[1] Scaffolding protein functional sites using deep learning.  Jue wang. et al. Science. 2022.

[2] Biological Sequence Design with GFlowNets. [Moksh Jain](https://arxiv.org/search/q-bio?searchtype=author&query=Jain,+M) et al. ICML 2022. 

[3] Proximal Exploration for Model-guided Protein Sequence Design. Zhizhou Ren et al. ICML 2022. 

[4] Robust deep learning based protein sequence design using ProteinMPNN. J. Dauparas et al. Science. 2022.

[5] Alphafill: enriching alphafold models with ligands and cofactors. Maarten L Hekkelman et al. Nature methods. 2023.

Reviewer comment (Reproducibility):
Code: The authors do not provide code for checking the soundness of the methods.

Author response:
We have provided the code and data in the supplementary material. Due to the memory limitation, we couldn’t upload the model checkpoints. However, the training process is highly efficient, and the reviewer could train the model if he/she is interested.

Reviewer comment (Experiments):
Experiment setting: The authors do not provide results on standard benchmarks, such as the CATH dataset, for fair comparison on both sequence and structure design.

Author response:
We would like to clarify our task more clearly. Our task aims to co-design functional and novel protein sequence and structure. In our method, we use MSAs to automatically find motifs to guarantee the protein function, which is achieved within the same protein family. Instead, CATH is a protein dataset consisting of proteins from diverse families, making the motif extraction difficult. Besides, CATH is usually used to evaluate inverse folding task, say protein sequence design based on fixed backbone structure, which is not the goal of our paper. We hope we didn’t misunderstand the reviewer’s meaning by “the original SMCDiff and FrameDiff results” as SMCDiff and FrameDiff haven’t been evaluated on CATH dataset. Since SMCDiff and FrameDiff focus on protein structure design, CATH is usually used to evaluate protein sequence design based on fixed backbone structure, which may have some discrepancies on the design goals.

Reviewer comment (Experiments):
Could you provide experimental results from the CATH dataset to compare with the original SMCDiff and FrameDiff results?

Author response:
We would like to clarify our task more clearly. Our task aims to co-design functional and novel protein sequence and structure. In our method, we use MSAs to automatically find motifs to guarantee the protein function, which is achieved within the same protein family. Instead, CATH is a protein dataset consisting of proteins from diverse families, making the motif extraction difficult. Besides, CATH is usually used to evaluate inverse folding task, say protein sequence design based on fixed backbone structure, which is not the goal of our paper. We hope we didn’t misunderstand the reviewer’s meaning by “the original SMCDiff and FrameDiff results” as SMCDiff and FrameDiff haven’t been evaluated on CATH dataset. Since SMCDiff and FrameDiff focus on protein structure design, CATH is usually used to evaluate protein sequence design based on fixed backbone structure, which may have some discrepancies on the design goals.

Reviewer comment (Experiments):
Similarly, could you provide head-to-head comparisons to ProteinMPNN, ESMIF, and PiFold on protein sequence design? Please follow the same setting.

Author response:
We provide protein sequence design given backbone structures on our two datasets in the following table. In this setting, we slightly modified our method to make it adaptable to the task as the reviewer asked. Specifically, we provided the whole backbone structure as the model input and designed the whole protein sequence. As the results show, even though we only provide  backbone structure without any motif residues, our model can still achieves higher AAR and pLDDT on myoglobin, and higher AAR on beta-lactamase.

**myoglobin**

|Model| AAR (%)|pLDDT |
|:-----|:----:|:----:|
|ProteinMPNN|81.37|81.9817|
|ESMIF|76.49|78.0986|
|PiFold|78.93|80.2912|
|NAEPro|**85.56**|**82.3871**|

$\beta$-lactamase

|Model| AAR (%)|pLDDT |
|:-----|:----:|:----:|
|ProteinMPNN|53.59|**70.6723**|
|ESMIF|57.49|61.3928|
|PiFold|63.38|64.9017|
|NAEPro|**68.39**|65.8218|

Reviewer comment (Reproducibility):
Could you provide the code for checking the results?

Author response:
We have provided the code and data in the supplementary material. Due to the memory limitation, we couldn’t upload the model checkpoints. However, the training process is highly efficient, and the reviewer could train the model if he/she is interested.

Reviewer comment (Novelty):
The identified problem or the motivation for designing such a method is not supported by existing literature (e.g., Questions 1-2).

Author response:
**Weakness 1 & weakness 2 & Q5: The identified problem or the motivation for designing such a method is not supported by existing literature.  Some designed modules counter the intuition in biology, such as "motif mining"**

Ans: We follow the design insight proposed by [1], which starts from the functional site and fills in additional sequence and structure to create a viable protein scaffold in a single forward pass. Different from [1], we provide the functional sites which are automatically mined from MSA as previous methods [2, 3] while [1] provides the functional sites manually. Besides, we design a new architecture called NAEPro to generate the whole protein sequence and structure based on the given partial protein fragments, while [1] achieves this goal by finetuning the RoseTTAFold. However, we agree with the reviewer’s opinion in question 5 that MSA discovers not only protein functional sites as addressed in [2,3] but also finds conversed sites which may not directly relate to the protein function. Therefore, we change all the term “motif/functional sites” in our paper to  “meaningful protein fragments”.

[1] Scaffolding protein functional sites using deep learning.  Jue wang. et al. Science. 2022.

[2] Emerging methods in protein co-evolution. David de Juan et al. Nature Review Genetics. 2013.

[3] Evolutionary information for specifying a protein fold. Michael Socolich et al. Nature. 2005.

Reviewer comment (Theory):
Some designed modules counter the intuition in biology (e.g., Questions 5).

Author response:
**Weakness 1 & weakness 2 & Q5: The identified problem or the motivation for designing such a method is not supported by existing literature.  Some designed modules counter the intuition in biology, such as "motif mining"**

Ans: We follow the design insight proposed by [1], which starts from the functional site and fills in additional sequence and structure to create a viable protein scaffold in a single forward pass. Different from [1], we provide the functional sites which are automatically mined from MSA as previous methods [2, 3] while [1] provides the functional sites manually. Besides, we design a new architecture called NAEPro to generate the whole protein sequence and structure based on the given partial protein fragments, while [1] achieves this goal by finetuning the RoseTTAFold. However, we agree with the reviewer’s opinion in question 5 that MSA discovers not only protein functional sites as addressed in [2,3] but also finds conversed sites which may not directly relate to the protein function. Therefore, we change all the term “motif/functional sites” in our paper to  “meaningful protein fragments”.

[1] Scaffolding protein functional sites using deep learning.  Jue wang. et al. Science. 2022.

[2] Emerging methods in protein co-evolution. David de Juan et al. Nature Review Genetics. 2013.

[3] Evolutionary information for specifying a protein fold. Michael Socolich et al. Nature. 2005.

Reviewer comment (Evaluation):
The method is not 'extensively evaluated' as claimed by the authors (they only selectively analyzed 2 proteins), and many results do not suggest that the proposed method is SOTA among baselines. Moreover, the evaluation does not include all relevant methods, which makes it too aggressive for the authors to claim in the Introduction that their method "achieves the HIGHEST...among ALL competitors" and "is faster than the FASTEST method".

Author response:
**Weakness 3 & Q1: The method is not 'extensively evaluated' as claimed by the authors (they only selectively analyzed 2 proteins), and many results do not suggest that the proposed method is SOTA among baselines. Moreover, the evaluation does not include all relevant methods, which makes it too aggressive for the authors to claim in the Introduction that their method "achieves the HIGHEST...among ALL competitors" and "is faster than the FASTEST method". Empirically, It has been proven by several recent research that the de novo method can generate new sequences for fixed backbone or desired functions [4-6].**

Ans: Sorry for the confusion. Our method achieves the best performance on 4/5 metrics on myoglobin and 3/5 metrics on beta-lactamase among all the compared representative baselines in our paper. Our model is faster than all baselines compared in our paper. We have revised all these statements in our paper (highlighted in red). For other relevant methods as the reviewer suggested in reference [4-6], we have already compared with [4]. [5] was posted on Oct. 3, 2023, which is later than the ICLR submission ddl, and [3] was posted on Aug. 14, 2023, which is within 3 months before ICLR submission ddl. We appreciate the reviewer’s suggestion, and we tried to compare with [3], but they haven’t released their code now. We emailed the authors of [3] to ask for code and they said their work is under review now and refused to provide code currently. We are trying to reimplement the model now and will update the results in our revised version.

[4] Watson et al., De novo design of protein structure and function with RFdiffusion (2023).

[5] Sumida et al., Improving protein expression, stability, and function with ProteinMPNN (2023).

[6] Zhou et al., Conditional Protein Denoising Diffusion Generates Programmable Endonuclease Sequences (2023).

Reviewer comment (Evaluation):
Results in Table 1 do not support the superiority of the proposed method. For instance, RMSD>1.7 \AA means the two structures are different for myoglobin. No matter if the predicted result is 4 \AA away or 10 \AA away, neither of them are reliable prediction. For pLDDT, the prediction on $\beta$-lactamase is ~66%, which is much lower than 80%-90% which is conventionally believed reliable for folding predictions. The TM-score, if smaller than 0.5, then the two structures are believed different. In this case, both 0.48 and 0.46 are considered a failed design, and it is meaningless to yield the 2% outperformance in this case.

Author response:
**Weakness 4:  Results in Table 1 do not support the superiority of the proposed method. For instance, RMSD>1.7 \AA means the two structures are different for myoglobin. For pLDDT, the prediction on lactamase is ~66%, which is much lower than 80%-90% which is conventionally believed reliable for folding predictions. The TM-score, if they are smaller than 0.5, then the two structures are believed different.** 

Ans: We would like to clarify our goal more clearly.  Our task is to design the sequence and structure of proteins with effective functions.  We calculated TM-score by AlphaFold2 as suggested by the reviewer, and the two proteins designed by our model both achieves an average score over 0.5 (myoglobin **0.5912** and beta-lactamase **0.5009**). However, as found by previous method [7],  protein pairs with a TM-score >0.5 are most likely in the same fold while those with a TM-score <0.5 are mainly not in the same fold. Therefore, low TM-score can not indicate the designed protein doesn’t have the corresponding function. As claimed in previous work [9,10], pLDDT between 70~90 are classified to be confident. We calculated pLDDT by AlphaFold2 as suggested by the reviewer and the two protein families designed by our model are confident on an average aspect (myoglobin **91.4139** and beta-lactamase **76.9550**).  Similarly, as observed in previous work[11,12], pLDDT has a weak correlation with protein function. Therefore, a moderate TM-score and pLDDT can not indicate a protein expresses no or poor function. As shown in figure 5(b), the designed myoglobin has a RMSD of 3.943\AA with the most similar one in PDB, but it can still bind the corresponding metallocofactor heme in the AlphaFill simulation [8]. To systematically illustrate the function of our designed proteins, we use docking method Gnina [13] to compute the binding affinity between the designed metalloproteins and the corresponding metallocofactors. The results in the following table shows that our model achieves the best binding affinity scores on both metalloproteins, demonstrating the proteins designed by our model are highly potential to actively exhibit biochemical functions.

|Model|myoglobin (kcal/mol)|beta-lactamase (kcal/mol)|
|:-----|:----:|:----:|
|Hallucination|-7.23|-6.37|
|Inpainting|-9.32|-7.63|
|SMCDiff+ProteinMPNN|-8.79|-6.89|
|PROTSEED|-10.07|-7.68|
|FrameDiff+ProteinMPNN|-9.54|-7.21|
|RFDiffusion+ProteinMPNN|-9.76|-7.59|
|NAEPro|**-10.14**|**-7.80**|

[7] How significant is a protein structure similarity with TM-score= 0.5? Jinrui Xu and Yang Zhang. *Bioinformatics. 2010.* 

[8] Alphafill: enriching alphafold models with ligands and cofactors. Maarten L Hekkelman et al. Nature methods. 2023.

[9] pLDDT Values in AlphaFold2 Protein Models Are Unrelated to Globular Protein Local Flexibility. Oliviero Carugo. Crystals. 2023

[10] Generating new protein sequences by using dense network and attention mechanism. Feng Wang et al. Mathematical Biosciences and Engineering. 2023.

[11] Using AlphaFold to predict the impact of single mutations on protein stability and function. Marina A. Pak et al. PloS one. 2023.

[12] Peptide binder design with inverse folding and protein structure prediction. Patrick Bryant et al. Nature Communications Chemistry. 2023.

[13] GNINA 1.0: Molecular docking with deep learning. A McNutt et al. Cheminformatics. 2021.

Reviewer comment (Theory):
Page 1: "The use of separate models for sequence and structure cannot ensure the consistency between the generated sequence and structure." I disagree with this statement. Generated protein sequences with higher than 30% identity to the wild-type protein merely change their geometry in space, while existing de novo methods usually generate new sequences with 30-90% sequence identity. Empirically, it has been proven by several recent research that the de novo method can generate new sequences for fixed backbone or desired functions [1-3].

Author response:
**Q1 & Q2: "The use of separate models for sequence and structure cannot ensure the consistency between the generated sequence and structure." "knowing the topology of a protein before design process is difficult and also cannot guarantee the designed proteins have the desired functions." I disagree with this statement.** **I disagree with this statement.** 

Ans: Thanks for the suggestion! We agree with the reviewer and updated the statement in our paper.

Reviewer comment (Theory):
Page 2: "knowing the topology of a protein before design process is difficult and also cannot guarantee the designed proteins have the desired functions." I disagree with this statement. Although AlphaFold2 cannot predict structures for all proteins with no error, it's a great reference, and designing and screening with it is sufficiently powerful in many research work, including those conducted wet-lab experiments.

Author response:
**Q1 & Q2: "The use of separate models for sequence and structure cannot ensure the consistency between the generated sequence and structure." "knowing the topology of a protein before design process is difficult and also cannot guarantee the designed proteins have the desired functions." I disagree with this statement.** **I disagree with this statement.** 

Ans: Thanks for the suggestion! We agree with the reviewer and updated the statement in our paper.

Reviewer comment (Writing):
Page 2: What is the meaning of "generally-encoded" for 20 types of amino acids?

Author response:
**Q3: what is the meaning of "generally-encoded" for 20 types of amino acids?** 

Ans: It means the 20 common amino acids. We have updated the paper accordingly.

Reviewer comment (Writing):
Page 3: "The selection of motif varies from setting to setting..." I cannot understand this example regarding the difference between motifs for active sites and binding sites. Can the authors please provide a clearer explanation?

Author response:
**Q4: "The selection of motif varies from setting to setting...". Can the authors please provide a clearer explanation?**

Ans: We mean the motifs have different meanings in different tasks. For de novo enzyme design [14], it means the binding sites where enzyme binds the metallocofactor and substrate. For de novo binder design [15], it means the binding sites where the protein binder binds the protein targets.

[14] De novo enzyme design using rosetta3. Florian Richter et al. PloS one. 2011.

[15] De novo design of protein interactions with learned surface fingerprints. Pablo Gainza. Nature. 2023.

Reviewer comment (Theory):
Page 5: "motif mining". I don't see the rationale of this module. MSA essentially discovers the conserved sites of a protein sequence by comparing them with other sequences from the same family. However, these conserved sites might not be directly related to its central function. For instance, many conserved sites are buried within the proteins, but functional sites are generally exposed at the surface of the protein. Moreover, the active sites are usually spatially close, but they may be distant sequentially.

Author response:
**Weakness 1 & weakness 2 & Q5: The identified problem or the motivation for designing such a method is not supported by existing literature.  Some designed modules counter the intuition in biology, such as "motif mining"**

Ans: We follow the design insight proposed by [1], which starts from the functional site and fills in additional sequence and structure to create a viable protein scaffold in a single forward pass. Different from [1], we provide the functional sites which are automatically mined from MSA as previous methods [2, 3] while [1] provides the functional sites manually. Besides, we design a new architecture called NAEPro to generate the whole protein sequence and structure based on the given partial protein fragments, while [1] achieves this goal by finetuning the RoseTTAFold. However, we agree with the reviewer’s opinion in question 5 that MSA discovers not only protein functional sites as addressed in [2,3] but also finds conversed sites which may not directly relate to the protein function. Therefore, we change all the term “motif/functional sites” in our paper to  “meaningful protein fragments”.

[1] Scaffolding protein functional sites using deep learning.  Jue wang. et al. Science. 2022.

[2] Emerging methods in protein co-evolution. David de Juan et al. Nature Review Genetics. 2013.

[3] Evolutionary information for specifying a protein fold. Michael Socolich et al. Nature. 2005.

Reviewer comment (Evaluation):
Page 7: "evaluation metrics": ESMFold, as observed by their authors, performs as well as AlphaFold2 when the folded sequence exists in nature. However, for a novel sequence, its prediction is worse than AlphaFold2. Consequently, evaluating novel sequences should use AlphaFold2 instead of ESMFold.

Author response:
**Q6: For a novel sequence, ESMFold’s prediction is worse than AlphaFold2. Consequently, evaluating novel sequences should use AlphaFold2 instead of ESMFold.**

Ans: Due to limited time and computing resources, we provide the AlphaFold2 results for ProtSeed, RFDiffusion+ProteinMPNN and our NAEPro. As the results show, even though ESMFold achieves lower scores than AlphaFold2, the tendencies (ranking) among different models are similar.  Particularly, our model achieves an average TM-score over 0.5 on beta-lactamase (ESMFold 0.4812) and pLDDT over 70 (ESMFold 66.8705), which gives stronger evidence that our model has the ability to design proteins which have stable structures.

**myoglobin**

|Model| pLDDT|TM-score|
|:-----|:----:|:----:|
|PROTSEED|77.9032|0.5621|
|RFDiffusion+ProteinMPNN|88.2130|0.5103|
|NAEPro|**91.4139**|**0.5912**|

**$\beta$-lactamase**

|Model| pLDDT|TM-score|
|:-----|:----:|:----:|
|PROTSEED|73.3176|0.4901|
|RFDiffusion+ProteinMPNN|**87.8092**|0.4806|
|NAEPro|76.9550|**0.5009**|

Reviewer comment (Writing):
Page 7: "Consistency - the RMSD between the designed structure and predicted one". I cannot understand this statement. What is the designed structure and the predicted structure? Which is from NAEPro? Where does the other one come from?

Author response:
**Q7: "Consistency - What is the designed structure and the predicted structure? Which is from NAEPro? Where does the other one come from?**

Ans: Consistency is calculated as the RMSD between the designed structure from NAEPro and the one predicted by ESMFold for the designed sequence from NAEPro.