Paper: Microenvironment Probability Flows as Proficient Protein Engineers (ICLR-2024)

Reviewer comment (Evaluation):
The pLDDT score is designed to evaluate the accuracy of structure predictions generated by AlphaFold. Using this metric to assess the authenticity of generated sequences and concluding that 'the model is likely to work in the real world' is logically inconsistent.

Author response:
**The pLDDT score is designed to evaluate the accuracy of structure predictions generated by AlphaFold. Using this metric to assess the authenticity of generated sequences and concluding that 'the model is likely to work in the real world' is logically inconsistent.**

Thank you for your thoughtful comments. Our choice to utilize pLDDT and RMSE in assessing the generated sequences is motivated by two key considerations. Firstly, the decision is aligned with practices found in relevant literature, where one or both of these metrics have been commonly employed.
Secondly, in the context of evaluating inverse folding or engineering models, researchers are particularly interested in ensuring the 'consistency' between structure-to-sequence and sequence-to-structure predictions. Given AlphaFold's proficiency in predicting protein structures, leveraging it to assess this consistency appears logical and is in line with the objectives of our study.

Reviewer comment (Experiments):
As someone that has undergone an actual wet-lab protein engineering campaign, the tools presented here should be defined as annotation or sequence generation. There are not clear steps to move these entities out of in silico predictions.

Author response:
Thank you for your comments, we emphasize this in the revision appendix “We recognize the disparity between machine learning-based sequence generation and experimental validation of computationally generated protein sequences. To address your question, we first develop methods, iteratively improve them, and submit them to machine learning conferences with the intent of sharing our in silico experimental result with the machine learning community. Once we feel our methods cannot be improved further in silico, we collaborate with experimental protein engineers to experimentally validate our methods. However, these experimental collaboration projects get submitted to protein journals such as Nature, Cell, ACS, journals rather than machine learning conferences.
We will move the discussions to the main text once we have an extra page.

Reviewer comment (Writing):
It would be great to define “recovery ratio” before using it a number of times in the work. Ideally the reader doesn’t have to hunt through a number of references to find its definition. This is also less familiar than the pLDDT or pTM scores.

Author response:
Thank you for your comments, we highlight the recovery ratio refers to the wildtype accuracy in the revision.

Reviewer comment (Presentation):
For all the tables (including Table 1), it’d be great to have up or down arrows if a “good” metric should be large or small, respectively.

Author response:
Thank you for your comments, we add these arrows in the updated version.

Reviewer comment (Evaluation):
The cosine similarity results in Table 3 are difficult to make conclusions from. What are meaningful, statistically significant distances for cosine similarity? Each of these models have quite different representations and architectures, many of which haven’t been designed for this task.

Author response:
Thank you for bringing this confusion to our attention. In Table 3, we generate sequences for 50 randomly selected proteins/backbones. For each of these backbone scaffolds, we run MPNN, PiFold, and MeFlow to generate 10 sequences, giving us a total of 500 sequences for each model. For the 10 sequences generated, we embed them with ESM2 and measure their cosine similarity in the ESM2 latent space. Since the sequences are being compared in solely ESM2 latent space, the cosine similarity of the sequence’s embedding for a given scaffold provides a fair comparison and a meaningful metric. We do this for each of the 50 randomly selected scaffolds with MPNN, PiFold, and MeFlow separately.
We will clarify this in the Table caption and paper to make this more clear once we have extra page.

Reviewer comment (Experiments):
“Experiment Settings To further verify the ability of protein engineering, we apply zero-shot ∆∆G prediction on FireProtDB….” This is sequence annotation.

Author response:
Thank you for your comment. Yes, ∆G is sequence annotation and ∆∆G is an annotation for a point mutation calculated by given ∆G of two sequences. We use these annotations to evaluate how well our model correlates with them.  In our context, ∆∆G represents the change free energy of folding based on a point mutation.

Reviewer comment (Experiments):
There are many different zero-shot mutation effect predictors, one of which is in the ESM family of models. I would like to see that baseline in section 4.2.

Author response:
Thank you for your comments, we list some supervised models and zero-shot models for the comparison. We present ESM-1b, ThermoNet and Stability Oracle in Table 6. ThermoNet is a well-known supervised trained ddG predictor, while Stability Oracle is a recently published SOTA model. We demonstrate that our zero-shot performance is even better than the supervised ThermoNet model.

Reviewer comment (Writing):
Nit: “and” is spelled wrong a number of times in the manuscript, sometimes as “nad” or “adn”.

Author response:
Thank you for your comments, we fix these typos.

Reviewer comment (Evaluation):
“As demonstrated in Table 3, we evaluate the diversity of different method. To assess diversity, we first consider the relative Hamming distance since there is no insertion or deletion operation in the design space.” - What about Levenshtein Distance? This is pretty common to deal with indels.

Author response:
Thank you for your comments. In our problem set, the sequence length is fixed (no insertions and deletions), and the position in the sequence corresponds to the position in the structure. In this case, there is no distinction  between Levenshtein Distance and Hamming distance and they can be considered equal.

Reviewer comment (Writing):
“Hence, we propose that differences within this hidden space could represent a ‘hidden’ dimension for important concepts, such as MSA, and therefore applying cosine similarity to measure diveristy in the ESM2 latent space.” I don’t think I fully follow or understand here.

Author response:
Thanks for the comments, we simplify this to ‘we measure the similarity with the pretrained ESM2 latent space’. Our key insight is based on the fact that ESM2 replaced the need of explicit MSAs in ESMFold. This enabled us to hypothesize that its latent space must  encompass rich information about MSAs.

Reviewer comment (Writing):
“we apply log(pmut/pwt) as the zero-shot score to predict whether a mutation is good.” What exactly do you mean by “good”? I would prefer if the phrasing was “if a mutation affects function”.

Author response:
Thanks for the comments, ‘good’ is confusing and we rewrite this to ‘if a mutation can potentially improve a protein phenotype, such as stability (∆∆G)”.

Reviewer comment (Writing):
“In real protein design problems, a given backbone can come to multiple potential sequences, and our interest is to identify these candidate sequences.” What does this mean? If this is a real problem biologists can encounter, can you provide and example or citation?

Author response:
As we mentioned in the third paragraph in the introduction section, It is quite common in nature for proteins that have a sequence similarity of at least ∼30% to have a structurally similar scaffold (Stern, 2013; Rost, 1999). Once we go below 30% sequence similarity, this fact begins to break down and is known as the “twilight zone” (Rost, 1999). This scaffold degeneracy is the underlying principle that drives the grouping of proteins into families and domains (e.g., Fox et al., 2014; Sillitoe et al., 2021; Paysan-Lafosse et al., 2023).

We refer the readers and the reviewers to all these related works we mentioned, for the background and related information. We will add a citation to this statement to help the reader.

Reviewer comment (Experiments):
I see in this work that ESM is used for creating embeddings and some of the other validations. Was this version of ESM blinded to CATH as well, as your splits have been done in model creation?

Author response:
Thanks for your valuable insight. To clarify, ESM is used as one way to measure the similarity in the hidden space. The ESM is trained on sequence similarity splitting while CATH is structure splitting, therefore, the ESM is not blind to the CATH sequences. We highlight this in the revision. However, we do not see this resulting in any issues since we are not training or evaluating the performance of ESM2.

Reviewer comment (Experiments):
In the mutation effect prediction task, can this method predict the effect of multiple mutations?

Author response:
As mentioned in the experiment settings of section 4.2, we construct zero-shot predictors based on the flow model. For the multiple mutations, we can get a zero-shot model by adding single-mutation effects as demonstrated in [1] and ESM-1b paper. However, this naive addition does not take into account epistatic interactions. To get an accurate multiple-mutation predictor, it’s necessary to do supervised training with multiple mutation data, which is not the key focus of this paper.

[1] Predicting a Protein’s Stability under a Million Mutations

Reviewer comment (Experiments):
Does sequence augmentation actually help with your method? If not, why include it?

Author response:
As we mentioned in Table 6, the augmentation improves the zero-shot regressor performance by a large margin (for example, on FireProtDB, the spearman is improved from 0.30 to 0.41). Considering the literature, it even out-performs the supervised models, e.g., [1]. We list these numbers for further comparison in revision, and therefore we think our augmentation helps the model.

[1]  Predicting changes in protein thermodynamic stability upon point mutation with deep 3D convolutional neural networks

Reviewer comment (Experiments):
How well does the “Classification” model in Table 1 compare in all the other experiments below?

Author response:
Thanks for your suggestion, we include it in several tables in the updated pdf, where it makes sense. Table 3 is about measuring the diversity while the classification model cannot directly generate diverse outputs. Table 4 is about sensitivity analysis about generation related hyper-parameters and has nothing to do with classification. In Table 5, we add the classifier results in the revision. In Table 6, we already demonstrate the classification model’s  zero-shot performance.

Reviewer comment (Presentation):
While I appreciate the number of metrics in Figure 3, I think it is a bit overkill. Also, the 1/rmse for Myolobin is off the chart.

Author response:
Thanks for your comments, as we demonstrated in the caption, all the detailed numbers are in the appendix. We also update figure for Myolobin.

Reviewer comment (Experiments):
The central claim that conditional generation using microenvironments allows for designing more diverse sequences that recover a target backbone structure is not sufficiently supported.

Author response:
We do not agree with the point “The central claim that conditional generation using microenvironments allows for designing more diverse sequences that recover a target backbone structure is not sufficiently supported”. In literature, researchers either compare recovery ratio (e.g., PiFold), or compare zero-shot single-mutation predictions (e.g., ESM). We do both settings, and demonstrate that our method is comparable or even better.

Reviewer comment (Reproducibility):
What atoms comprise the local environment? Are these all the backbone atoms?

Author response:
The microenvironment consists of the backbone atoms that are within 20Å of the Ca atom of the center residue. We will make edits in the language to make the definition of the inverse folding microenvironment more clear. We only task the backbone atoms to build the local chemical environment.

Reviewer comment (Evaluation):
The authors claim that their overarching goal is to explore conditioning on local atomic environments rather than global backbone structure for inverse design; the usual sequence recovery and diversity metrics do not show much difference between these paradigms. Have the authors considered tasks more related to remote homology detection?

Author response:
Thanks for valuable comments. We think an inverse folding model can be used as a zero-shot remote homology detector. However we have yet to run these experiments ourselves.

Reviewer comment (Experiments):
Data augmentation via Uniref100 is interesting but underexplored. Do the authors have ideas or additional experiments that could improve the results using this style of sequence augmentation?

Author response:
Thanks for valuable comments. We think generating augmentations with sequence similarity is a new and not-explored idea. We know similar sequences can come to similar scaffolds and we plan to do more future works on this topic.

Reviewer comment (Reproducibility):
What physical properties are used to characterize the atoms?

Author response:
As we mentioned in the appendix, Our included physical properties contain two channels, the partial charges and the surface or core information. The physical properties contain categorized partial charge (negative, neutral and positive) and whether the atom is buried.

Reviewer comment (Evaluation):
What are the correlation coefficients in Fig 2b and c? Is this similar to other methods? What references or evidence suggest that achieving comparable pLDDT and pTM scores indicate that a model will “work in the real world”?

Author response:
We update the manuscript and mention this is the Pearson correlation. We also add comments “In terms of Pearson correlation, the difference between our method and PiFold is not significant” in the updated pdf. We use pLDDT since ProteinMPNN [1] evaluate relative AlphaFold success rates in Figure 2. We notice this is not accurate and therefore remove this sentence “work in the real world” in the updated pdf.

[1] Robust deep learning–based protein sequence design using ProteinMPNN

Reviewer comment (Experiments):
What baselines are relevant for Table 5?

Author response:
Thanks for your comments, we further provide a classification model baseline in the revision. We mainly do the comparison in table 6, which shows training on our dataset, we get a better zero-shot model.

Reviewer comment (Reproducibility):
Is the Interface dataset publicly available?

Author response:
The dataset is not publicly available yet, we will make the dataset we used available after acceptance.

Reviewer comment (Writing):
It may improve clarity to give names to the Losses in equations 4 and 7 to refer to them throughout the paper.

Author response:
Thanks for your comments, we will modify this once we have more space for the main text.

Reviewer comment (Presentation):
Figure 3 is visually interesting but not particularly informative and largely redundant with Table 6.

Author response:
In Figure 3, we want to show more metrics to compare these two methods. Table 6 only shows two correlation metrics. Therefore, we do not think Figure 3 is redundant. A model with high correlation does not always refer to a model with high recall/precision.