Paper: KW-Design: Pushing the Limit of Protein Design via Knowledge Refinement (ICLR-2024)

Reviewer comment (Reproducibility):
The code associated with the paper is currently unavailable.

Author response:
**Reply to W1** During rebuttal, we deploy KWDesign to Colab for your check. It is on the way.

Reviewer comment (Evaluation):
The paper predominantly employs perplexity and recovery as metrics for evaluating the designed sequences. However, there is a chance that the designed proteins may not be soluble or may not fold correctly to the given backbone. It would be beneficial for the authors to incorporate additional metrics (e.g., scTM score, solubility) in their evaluation.

Author response:
**Reply to W3** Thanks for your good question. In the updated appendix, we compared the scTM score of different methods, where KWDesign consistently outperformed baselines.

Reviewer comment (Reproducibility):
Is there any fine-tuning done on the pretrained language model used in your approach?

Author response:
**Reply to Q1** The pretrained language models are frozen.

Reviewer comment (Evaluation):
In Section 4.3, Table 3 claims that “the predictive confidence score, an unsupervised metric, exhibits a strong correlation with recovery.” Could the authors provide a more detailed analysis, perhaps including the Spearman correlation between these two values?

Author response:
**Reply to Q2** Thanks for your insightful comments. To clarify, higher confidence generally correlates with higher recovery rates. Taking PiFold and KWDesign as examples, we analyze recovery statistics within specific confidence ranges. In the test set of CATH4.2, we record the confidence and recovery (0 or 1) for each residue in all proteins. Finally, we calculate the average recovery for residues falling within a particular confidence range, shown as follows:

>>| conf        | >0.0   | >0.1   | >0.2   | >0.3   | >0.4   | >0.5   | >0.6   | >0.7   | >0.8   | >0.9   | >0.95  |
>>|----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
>>| PiFold   | 51.58 | 51.60 | 55.43 | 63.04 | 69.87 | 75.99 | 81.75 | 86.66 | 90.96 | 95.09 | 97.44 |
>>| KWDesign | 60.25 | 60.26 | 62.08 | 67.16 | 72.71 | 78.00 | 83.08 | 87.63 | 91.76 | 95.49 | 97.48 |

>> The table shows that higher confidence generally correlates with higher recovery rates.

Reviewer comment (Reproducibility):
Regarding the virtual MSA, does the sequence order affect the resulting residue embedding? If not, what criteria are used to determine the sequence order?

Author response:
**Reply to Q3** As shown in Eq.(12), we employ a summation operation to merge features of virtual MSA sequences. Consequently, the order of the sequences does not affect the residue embedding. In other words, the residue embedding remains unaffected by the sequence order.

Reviewer comment (Presentation):
The method section is not very clear. In particular, Section 3.2 and 3.3 have a large portion of redundancy. For instance, I think “Knowledge extractor & Confidence predictor” in Section 3.3 seems to just repeat the content in “Virtual MSA” in Section 3.2. “Fuse layer” can be merged to “Virtual MSA” to make the presentation more concise. Besides, I don’t know how Eq (13) is related to Eqs (8) and (9). It looks like they are both about how to update the embeddings, but how does “PiGNN” process $e^{(l)}$ to make Eq. (13) consistent to Eqs (8) and (9)?

Author response:
> **Q1.1** The method section is not very clear. In particular, Section 3.2 and 3.3 have a large portion of redundancy. For instance, I think “Knowledge extractor & Confidence predictor” in Section 3.3 seems to just repeat the content in “Virtual MSA” in Section 3.2. “Fuse layer” can be merged to “Virtual MSA” to make the presentation more concise. 

>> **Reply to Q1.1** Thank you for your valuable feedback. We are not intended to confuse readers. Instead, we have structured the method section of KW-Design hierarchically, with Section 3.2 providing a high-level overview of the computing framework and Section 3.3 delving into the specifics of the knowledge-tuning module. The abstract-to-detailed approach result in some redundancy. Following your suggestions, we have simplified the sentences of Section 3.3. The introduction of the knowledge extractor and confidence predictor in Section 3.3 serves to ensure complete description of the knowledge-tuning module.

> **Q1.2** Besides, I don’t know how Eq (13) is related to Eqs (8) and (9). It looks like they are both about how to update the embeddings, but how does “PiGNN” process to make Eq. (13) consistent to Eqs (8) and (9)?
>> **Reply to Q1.2** We appreciate your suggestion! We have revised the manuscript accordingly. In the revised version, we clarify that PiGNNs function as the refine module in Eq. (9), which is responsible for transforming $z^{(l)}$ into $h^{(l+1)}$.

Reviewer comment (Theory):
In “Confidence-aware updating”, why do you need two MLPs to process the same input (one is the negative sign of another) for the gates? Any justification on Eq. (11)?

Author response:
> **Q2** In “Confidence-aware updating”, why do you need two MLPs to process the same input (one is the negative sign of another) for the gates? Any justification on Eq. (11)?
>> **Reply to Q2** The two MLPs in our method serve the purpose of computing positive and negative scores to control the updating or retaining of residue embeddings. 

>> - As the openreview could not render complex latex code, we rewrite $b = c^{(l+1)}_{s'}$,  $a = c^{(l)}_s$, representing the confidence vectors of refined sequences at the $l+1$ and $l$ knowledge-tuning layer, respectively. 
>> - In Fig. 3, we demonstrate that not all refinements have an positive impact on improving recovery. To address this, we have designed a gated filter that selectively activates the latest refinement or retrain the previous residue embeddings.
>> - In Eq.(11), $score^+ = \sigma(MLP_1 (b-a))$ measure the superiority of the latest refined embedding over the previous embedding. Similarly, we compute the negative score $score^- = \sigma(\text{MLP}_2 (a-b))$. We use different MLPs to calculate positive and negative scores because they represent different meanings in filtering refined embeddings $h^{(l+1)}$ or   previous embeddings $h^{(l)}$, respectively.
>> - Finally, we combine the refined residue embeddings and previous embeddings via Eq.(11), that is $ h^{(l+1)} \leftarrow h^{(l+1)} \odot score^+ + h^{(l)} \odot score^-$
>> We hope this clarification addresses your queries.

Reviewer comment (Reproducibility):
In Algorithm 1, I’m not sure how the optimal $\phi^{(l)}$ is determined and how the “patience of 5” is applied. Also, in line 12 (Algorithm 1), what does $n$ mean in $h_n^{(l+1)}$?

Author response:
> **Q3** In Algorithm 1, I’m not sure how the optimal $\phi^{(l)}$ is determined and how the “patience of 5” is applied. Also, in line 12 (Algorithm 1), what does $n$ mean in $h_n^{(l+1)}$ ?

>> **Reply to Q3**  Thank you for your question. In Algorithm 1, the determination of the optimal $\phi^{(l)}$ is done using a patience of 5. Here's how it works:
>1. We train the knowledge-tuning modules sequentially, updating $\phi^{(l)}$ at each training process $l$.
>2. During the training process of the $l$-th knowledge-tuning module, we monitor the average predictive confidence over the training set as a metric for evaluating the performance.
>3. We use early stopping with a patience of 5, which means that if the average predictive confidence does not improve for 5 consecutive training iterations, we stop the training process for that specific $l$-th module.

> In Algorithm 1, $h_n^{(l+1)}$ shoud be $h_b^{(l+1)}$, we have revised the manuscipt accordingly. Thank you for bringing it to our attention.

Reviewer comment (Presentation):
I wonder why not split Table 1 into two subtables: one for CATH 4.2 and another for CATH 4.3?

Author response:
> **Q4** I wonder why not split Table 1 into two subtables: one for CATH 4.2 and another for CATH 4.3?
>> **Reply to Q4** We put the results in the same tabel for saving the page space. As you can see, many results and analysis have been removed to the appendix due to the limted space.

Reviewer comment (Evaluation):
In Table 2, what does the “Worst” metric mean? Why does the proposed method perform worse than all other methods?

Author response:
> **Q5** In Table 2, what does the “Worst” metric mean? Why does the proposed method perform worse than all other methods?
>> **Reply to Q5** In Table 2, the "Worst" metric refers to the lowest recovery rate achieved on the test set, following the setting of PiFold. In the case of TS500, all the method could not achieve resonable recovery. It is possible that there are low-quality or difficult structures in the dataset.

Reviewer comment (Writing):
Minor typo issue. “settings Specifically” -> “settings. Specifically” on page 6. “3d CNN” -> “3D CNN” on page 3.

Author response:
> **Q6** Minor typo issue. “settings Specifically” -> “settings. Specifically” on page 6. “3d CNN” -> “3D CNN” on page 3.
>> **Reply to Q6** Thank you for pointing out the minor typos. We have made the necessary revisions in the manuscript.