Paper: De novo Protein Design Using Geometric Vector Field Networks (ICLR-2024)

Reviewer comment (Novelty):
The thesis of the paper would be improved by better contextualization relative to IPA. The authors should not shy away from acknowledging significant similarities, but highlight the key changes and the insights behind them. I would suggest a side-by-side algorithmic comparison.

Author response:
We appreciate the reviewer's constructive suggestions. Indeed, our current paper may not be reader-friendly for those unfamiliar with IPA. We will carefully address this issue and make substantial improvements to our writing in the next version. In the **global response**, we presented a comparison between VFN and IPA, and highlighted insights that were overlooked by the reviewer. Please refer to the global response for more details.

We agree with the reviewer and have incorporated the **updates to the paper** in **Appendix A.1** as per the reviewer's instructions. To facilitate the reviewer's examination, we have included this section in **Section 2 of the supplementary materials** we provided. **Please download the supplementary materials**, or review section A.1 in the updated paper for a clearer explanation. Here, for the sake of conciseness, we are only presenting crucial information beyond supplementary materials.

1) We vividly illustrate the comparison between IPA and VFN using pseudocode, as shown in Algorithm 1 and 2 in the supplementary material. This comparison is presented **side by side**. Due to the limitations of Markdown, we are unable to provide such vivid comparison here; please refer to our supplementary material.
2) From the intuitive comparison of pseudocode, VFN and IPA appear to be vastly different.
3) The reason we use IPA as an analogy for VFN in the paper is to facilitate better understanding for our readers. However, this does not imply a high degree of similarity between VFN and IPA. While VFN and IPA do share the same paradigm (virtual atoms), their specific methods are significantly different, as illustrated in the pseudocode.
4) We have revised and updated the paper as per the reviewer's instructions. In the next version, we will further enhance our presentation to better compare VFN and IPA.

Reviewer comment (Experiments):
The paper could be further strengthened by additional comparisons with IPA. Particularly, if we replace IPA in AlphaFold/ESMFold with VFN, does the performance persist? It should not be too hard to run this experiment since the structural module is not very large.

Author response:
This is a very meaningful experiment that we have attempted before. However, training AlphaFold2 is resource-intensive, requiring approximately **2 weeks** of training with **128 TPUs**, as demonstrated in the official paper. Moreover, intricate data processing methods are employed during this process. Currently, we do not have sufficient resources to conduct this experiment, even for fine-tuning AlphaFold2. Therefore, we leave this experiment for future work. But thanks for the suggestion.

Reviewer comment (Experiments):
The claims about a "universal encoder" are not well-supported. It would be nice to see actual experiments where sidechains are also involved.

Author response:
We think that experiments related to sidechains are also meaningful, so we conducted comparative experiments on the benchmark established within GearNet[1], as shown in the table below:

| |GearNet[1]|VFN|
|:-|:-:|:-:|
|Fold3D (Acc) | 52.17 | **55.98** |
|EC ($f_{max}$) | 76.22 | **77.74** |
|GO-BP ($f_{max}$) | 44.05 | **44.96** |

The above experimental settings were meticulously aligned, constituting rigorous ablation experiments. Due to time constraints, the training duration for both methods was set at 2/3 of the official setting. For detailed information on the specific benchmark, please refer to the GearNet paper. One advantage of VFN is that it **eliminates** the need for a **featurizer** to provide **hand-crafted features** such as atomic distances and directions. Instead, it simply requires **treating real atoms as virtual atoms** and feeding them into the vector field operator. This experimental outcome highlights **the potential** of our approach in future tasks related to **sidechain**.

Here, we begin by referencing the introduction to the relevant tasks in GearNet.

> GearNet: Enzyme Commission (**EC**) number prediction seeks to predict the EC numbers of different proteins, which describe their catalysis of biochemical reactions. The EC numbers are selected from the third and fourth levels of the EC tree, forming 538 binary classification tasks. Gene Ontology (**GO**) term prediction aims to predict whether a protein belongs to some GO terms. Fold classification (**Fold3D**) is first proposed in Hou et al. (2018), with the goal to predict the fold class label given a protein.

Those tasks are **full-atom**, and the majority are associated with protein functionality. Therefore, they constitute experiments related to side-chain interactions. In addition, many papers, e.g. [2], have been published based on the same tasks to investigate better representation methods related to side chains. The representation method of VFN is different from theirs, representing a potential future approach that can be integrated with existing methods. We leave this aspect of the research for future work.

The experimental results are shown below:
| |GearNet[1]|VFN|
|:-|:-:|:-:|
|Fold3D (Acc) | 52.17 | **55.98** |
|EC ($f_{max}$) | 76.22 | **77.74** |
|GO-BP ($f_{max}$) | 44.05 | **44.96** |

[1] Protein Representation Learning by Geometric Structure Pretraining
[2] Learning Hierarchical Protein Representations via Complete 3D Graph Networks

Reviewer comment (Novelty):
From the novelty standpoint, an argument can be made that the architecture is similar enough to IPA and / or PiFold to count against its technical significance.

Author response:
We believe that some insights regarding SE(3) invariance have been overlooked, leading to concerns related to novelty. Therefore, we elaborate on our insights in detail in the global response. We would greatly appreciate it if the reviewer could carefully read our global response to understand our insights. Our novelty and contributions mainly include the following points: 1) insights into **the atom representation bottleneck** in IPA, 2) **a novel method ensuring SE(3)-invariance**, and 3) **the vector field operator** (as depicted in equation 1-4). The VFN employs many **commonly used** operations, such as MLP-based attention (used in PiFold). However, these operations are **not the focus** of this paper, and the use of common operations should not undermine the novelty of our work.

Reviewer comment (Experiments):
Please describe more details on how the VFN-IF+ training split is constructed.

Author response:
We initially **filtered out** proteins that exhibit a high degree of similarity (>**80%**) to sequences in the **validation set** and **test set**. Therefore, our training split does not pose any concerns regarding data leakage. Subsequently, proteins with a resolution greater than 9A were also excluded. Ultimately, our training set consists of 120,934 proteins.

Reviewer comment (Experiments):
Table 5 shows results for FrameDiff+ProteinMPNN vs VFN-Diff+VFN-IFE. What was the exact setting for reporting these numbers reported here, relative to Table 4? Are there equivalent results for FrameDiff+VFN-IF or VFN-Diff+VFN-IF? Please show these for all experimental settings in Table 4.

Author response:
The settings for Table 5 are as follows: Noise Scale = 1, Num. step = 500, Num Seq. = 1. We do have results for VFN-Diff+VFN-IF, as shown below:

|Methods|scTM>0.5|RMSD<2|Diversity|
|:-| :-:|-:|-:|
|FrameDiff+ProteinMPNN |37.25%|6.88%|38.02%|
|VFN-Diff+VFN-IF |44.81%|11.48%|42.72%|
|VFN-Diff+VFN-IFE |**45.43**%|**11.73**%|**43.46**%|


**We are conducting experiments** on more settings as per the reviewer's suggestions. However, the runtime for these experiments is exceedingly long. We plan to release these results soon. However, the lack of uniformity in the experimental settings within this field poses challenges for such experiments. The reasons are as follows:

1) The ProteinMPNN in Table 5 utilized a **larger training set**, **additional data augmentation**, and an autoregressive decoder, which is **unfair to VFN-IF**. We are incorporating these tricks employed by ProteinMPNN into VFN-IF to obtain experimental results.

2) The experimental results in Table 5 indicate that, even in an unfair setting with **fewer training data** and **no data augmentation**, VFN can still **outperform** the FrameDiff pipeline.

3) The ProteinMPNN in **Table 1** is reproduced by PiFold, validated by peer review, and **strictly aligned with our setting**. This experiment demonstrates that VFN-IF can significantly outperform Protein MPNN in a fair setting (**54.74%** vs. 45.96%).

4) VFN-IF employs PiFold's one-shot decoder, leading to faster inference speed. However, it is not easy to sample multiple results, so the Num Seq. can only be set to 1.

5) One fact is that, in this experiment, inverse folding is currently **not** the primary bottleneck and does not result in significantly noticeable differences. The performance of this table is determined by the diffusion model. The final results will be very similar with Table 4.

Reviewer comment (Evaluation):
In Figure 1, why does FrameDiff suffer in scTM more on medium-length proteins than the longest proteins?

Author response:
We currently believe that this phenomenon is attributed to **data bias** in the PDB. Since this phenomenon is observed not only in FrameDiff but also in VFN and other published methods, the most likely cause is the bias in the training data. It is noteworthy that this phenomenon is particularly pronounced in scTM (Figure 1.B), while it is not evident in scRMSD (Figure 5). VFN also experiences this issue; however, VFN is more **robust** to data bias, and the problem is not as pronounced.

Reviewer comment (Experiments):
Why was it necessary to retrain FrameDiff, as claimed in the appendix?

Author response:
FrameDiff is trained on the PDB. On one hand, the data in the PDB may increase daily, and on the other hand, the author of FrameDiff does not release the sample IDs used in their training. **If we do not retrain FrameDiff, it will result in misalignment of the training data.** To avoid this issue, we have retrained FrameDiff to ensure that the training sets for all comparative experiments are consistent.

Reviewer comment (Experiments):
Please clarify if the PiFold and FrameDiff numbers are taken directly from the respective papers. Please affirm that the results and claims made here are indeed under *identical experimental conditions* relative to PiFold and FrameDiff, or if they have been modified, please be very direct about these modifications.

Author response:
Our comparative experiments with FrameDiff are **rigorously aligned without any tricks**. We directly adopted and reported the official results of PiFold, and our comparative experiments are also precisely aligned with PiFold. However, it is important to note that PiFold employs a featurizer, whereas VFN-IF **does not use a featurizer**. This constitutes an advantage for VFN, as PiFold relies on manually extracted distances and directions between atoms, which VFN does not require.

Reviewer comment (Experiments):
**The problem setting is unfair:** In the first paragraph of section 4, the author mentioned "In the protein diffusion part, the protein structure is designed and represented using backbone frames T. Subsequently, these backbone frames are fed into the inverse folding network to obtain the corresponding protein sequence for the designed structure." If I didn't understand wrongly, this means the author first designs the protein structure and then generates the protein sequence based on the designed structures. Therefore, it's kind of like a pipeline. However, in Table 1, for the sequence design task, the author only compared with the inverse folding models, that is, sequence design based on real structures instead of designed structures. I think a more fair comparison should compare to baselines like structure design model plus inverse folding model, such as FrameDiff + ProteinMPNN, RFDiffusion [1] + ProteinMPNN, and also some structure-sequence co-design models like ProtSeed [2].

Author response:
### 3)  The issues regarding the inverse folding benchmarks and diverse settings

We did provide a comparison of the settings (**VFN-Diff+VFN-IFE vs. FrameDiff + ProteinMPNN**), mentioned by the reviewer, in the appendix. Our method exhibits a **11.18% advantage** in designability. The experimental results are presented in the table below, and for details, please refer to **appendix** A.2 as well as Table 5 and Figure 10.

|Methods|scTM>0.5|RMSD<2|Diversity|
|:-| :-:|-:|-:|
|FrameDiff+ProteinMPNN |37.25%|6.88%|38.02%|
|VFN-Diff+VFN-IF |44.81%|11.48%|42.72%|
|VFN-Diff+VFN-IFE |**45.43%**|**11.73%**|**43.46%**|

In addition, we would like to emphasize the following:
1) Once again, **the purpose of VFN is not to propose a new protein design pipeline but rather to introduce a new protein design encoder.**
2) **The significance of the inverse folding benchmark we adopted.** Inverse folding is a crucial task, and the benchmark we adopted is widely recognized. At least **20** papers have conducted tests using the same or similar benchmarks on **real protein structures**, including **science** paper, ProteinMPNN, and ProtSeed mentioned by the reviewer (VFN-IF significantly outperforms ProtSeed, **54.7% vs. 43.8%**). Moreover, inverse folding can be widely used in various protein design pipelines, **not just protein diffusion**. The outstanding performance of VFN-IF is a **significant objective contribution**.
3) We partially agree with the reviewer's thoughts on benchmarks based on generated proteins. However, **such benchmarks face some obvious issues**. One fact is that current protein generation models still encounter many issues. Testing based on generated data introduces many **incorrect structures** and **biases** from the **generation model**. Additionally, the random seed of the generation model introduces random factors. Therefore, for inverse folding, such benchmarks may not be rigorous enough. The CATH benchmark adopted by the community may be better, so we follow the CATH benchmark.

Reviewer comment (Writing):
In figure 2, $T_{i\rightarrow j}$ means T_j to T_i, while in line below Equation 1, it means T_i to T_j. What does this term really mean?

Author response:
### 6)  The question regarding $\mathbf{T}_{i\leftarrow j}$

The reviewer is correct. There is a typo here. We confirm that $\mathbf{T}_{i\leftarrow j}$ signifies the transition from $\mathbf{T}_j$ to $\mathbf{T}_i$.

Reviewer comment (Writing):
Why the range of H can be negative, say -200 A?

Author response:
### 7)  The question regarding $\vec{\mathbf{h}}_k$

As $\vec{\mathbf{h}}_k$ is a vector, it possesses a direction. The positive or negative sign of $\vec{\mathbf{h}}_k$ signifies the direction of the vector.

Reviewer comment (Reproducibility):
$d_q$ is the number of channels in $g_{i,j}$. Does that mean the number of virtual atoms between node i and node j? What is the specific value used in this paper?

Author response:
### 8)  The question regarding $d_q$
Sorry, there is a typo here. The number of channels for $\mathbf{g}_{i,j}$ is $d_g$, not $d_q$; please refer to Eq.3. The reviewer's understanding of $d_q$ is correct, referring to the quantity of atoms. We used 16 atoms in the paper, which is much less than the quantity in IPA.

Reviewer comment (Writing):
Are the MLP in equation 7 and 4 the same one?

Author response:
### 9)  The question regarding Eq.7 and Eq.4

Those MLPs are **not** the same one. The two MLPs do not share parameters. We will emphasize this point in the next version.

Reviewer comment (Experiments):
Using the ESM as initialization and then testing the model on CATH benchmark may have data leakage issues. How did the author deal with this problem?

Author response:
### 10)  The question regarding ESM model

The use of ESM to assist inverse folding[3] has been **accepted by peer review** (**LM-design[3], ICML 2023 Oral**). We adopted exactly the same setting. Therefore, we believe this is **not a flaw** in our paper. While we acknowledge the risk of data leakage, we argue that it should be considered from a different perspective.

1) The objective of inverse folding is to map structures into sequences. Since ESM has not been trained on structures but only on sequences, this does not constitute a data leak in the strict sense.

2) The approach[3][4] of inverse folding based on ESM has demonstrated its efficacy in sequence design. Therefore, from this standpoint, it holds significance.

3) Even from the strictest perspective, LM-Design and VFN-IFE can be considered as retrieval-based sequence design methods. However, in the ESM training set, potentially thousands of sequences exhibit a high degree of similarity to the predictions of VFN-IF. **Retrieving** the correct sequences remains an ongoing challenge, and significant progress has been made by VFN-IFE in addressing this issue (**62.67%** vs. 55.65%, compared to LM-design[3]).

[3] Structure-informed Language Models Are Protein Designers. ICML 2023.  
[4] Knowledge-Design: Pushing the Limit of Protein Deign via Knowledge Refinement