Paper: Dynamics-Informed Protein Design with Structure Conditioning (ICLR-2024)

Reviewer comment (Novelty):
Section 3.1 could be included in Section 2, as it is established and common to use a delta function centered at the denoiser's mean to perform guidance, and it is not unique to the paper.

Author response:
Dear Reviewer 65X8,

Thank you for taking the time to read and reply to our response.

We apologise for an apparent miscommunication on our part: We did not mean to claim that guidance using Tweedie's formula is the novelty of our method (indeed, we cite reference [1] you mention in our response and the paper).

Rather, we believe the novelty of our method comes from combining an old-school bioinformatics technique (NMA) and making it "machine-learning fit" and by formulating the task of joint motif and dynamics conditioning via normal modes. We believe that the introduction of this idea will be novel and interesting to several practicioners in the field, and hope that our evaluation of it constitute a helpful first guide. 

In the revised manuscript that we just uploaded (apologies for taking this long, re-running the experiments & analysis to address the interesting questions raised in the reviews took us some time), you will find an evaluation of how well improving the objective transfers to inverse-folded and re-folded designs exhibiting the targeted lowest normal mode dynamics. We agree that further independent analysis via techniques outside of NMA (e.g. molecular dynamics or lab experiments) are desirable to establish more clearly for which problems this technique an be promising and see these as important avenues of further work. For now, we rely on the established body of literature studying normal mode analysis and which types of motion it is useful for (e.g. larger scale backbone movements such as hinge or shear motions) to give an intuition about the realm where our ideas are likely to be applicable and useful. We agree that visual inspection does not constitute a sufficiently compelling argument and hope you find the self-consistency NMA score we introduce in this revised version of the manuscript (appendix G) helpful as criterion in this regard.

We'd like to thank you again for your comments and for engaging in discussion, and hope you find the revised manuscript interesting. 

Kind regards,
the authors

Reviewer comment (Experiments):
The current experimental results are fairly limited. Showing that guidance leads to lower loss values is more of a sanity check than a full result.

Author response:
We also discuss the points about evaluation in the general response, since they were of interest to multiple reviewers. For a short summary here, in addition to visual inspections and evaluating the NMA loss, the current quantitative proxy evaluation you mention, we newly add a self-consistent NMA (scNMA - explained in the general response) evaluation to our evaluation pipeline and are currently running experiments to assess it. Akin to the self-consistency metrics which are used in structure conditioning, this metric will evaluate how well the targeted mode is captured by the inverse folded & re-structure-predicted protein. We agree that evaluations which include molecular dynamics simulations will be a further, more powerful test that are out of scope for this current work, but that we plan to perform in the future.

Reviewer comment (Evaluation):
The paper lacks a compelling independent evaluation to demonstrate that the samples have the desired dynamics. While such an evaluation would be challenging and may require expensive molecular dynamics simulations or lab work, establishing such a framework could be a significant contribution.

Author response:
In the revised manuscript that we just uploaded (apologies for taking this long, re-running the experiments & analysis to address the interesting questions raised in the reviews took us some time), you will find an evaluation of how well improving the objective transfers to inverse-folded and re-folded designs exhibiting the targeted lowest normal mode dynamics. We agree that further independent analysis via techniques outside of NMA (e.g. molecular dynamics or lab experiments) are desirable to establish more clearly for which problems this technique an be promising and see these as important avenues of further work. For now, we rely on the established body of literature studying normal mode analysis and which types of motion it is useful for (e.g. larger scale backbone movements such as hinge or shear motions) to give an intuition about the realm where our ideas are likely to be applicable and useful. We agree that visual inspection does not constitute a sufficiently compelling argument and hope you find the self-consistency NMA score we introduce in this revised version of the manuscript (appendix G) helpful as criterion in this regard.

Reviewer comment (Experiments):
How were the guidance scale values selected? Were they tuned for the protein structures you evaluated on (in which case they might be overfit to the tasks)? An independent evaluation model could also be useful here.

Author response:
1. The guidance scales were indeed fine-tuned per target type, as is also common elsewhere in the field [1,2]. The same guidance scales were used for all samples with strain targets (when multiple sampled proteins had varying lengths), the same for all random targets (again, multiple sampled proteins with varying lengths), and there were different guidance scales for each of the hinge targets (per each target, many proteins, same lengths). Please find references to the works showing that fine-tuning guidance scale is a common practice in the field.

[1] Dhariwal and Nichol, “Diffusion Models Beat GANs on Image Synthesis”,  https://arxiv.org/pdf/2105.05233v4.pdf

[2] Chung et al. , “Improving Diffusion Models for Inverse Problems using Manifold Constraints”, https://arxiv.org/pdf/2206.00941.pdf

Reviewer comment (Experiments):
Why is guidance only turned on in the middle of sampling? This is not standard practice to my knowledge.

Author response:
2. Why is the guidance scale tuned on in the middle of sampling? 
In our loss-guided diffusion, in the sampling at step t>0 we rely on a reasonably good prediction of sample $\hat{x}_0$ at t=0 via Tweedie’s formula. This is important to be in the realm of applicability of NMA, which is robust to slight changes in coordinates but not backbone topology. Early in the sampling the model estimate of the sample at t=0 is poor and fluctuates strongly such that starting conditioning too early has a negative impact on sample quality. We therefore tune the guidance scale on midway through the sampling process, when the sample has become more protein-like. We note that similar techniques (i.e. timestep dependent losses) are used by other works in the field, such as in FrameDiff [https://arxiv.org/abs/2302.02277], where the auxiliary loss on the atoms distances was switched on only when t<0.5*T_max.

Reviewer comment (Experiments):
The filtering described in section 5.1 is very ambiguous. The statement “At the start, we filter out the ‘low quality’ samples whose mean chain distance is outside [3.75, 3.85]” lacks information about what fraction of samples were filtered out.

Author response:
- “The filtering in section 5.1 is very ambiguous [...]” 
Apologies for any confusion caused. Out of 300 samples, for strain targets conditioned samples, 18 samples failed with NaNs, and 63 violated distance filter. Out of 300 samples, for random targets, 38 samples failed with NaNs, and for 62 samples violated distance filter. The unconditional samples never failed with NaN, but about 40-50 violated the chain distance filter. The NaN values occur due to what we previously referred to as “blow up”, explained in more detail in the next point.

Reviewer comment (Evaluation):
The statement “Remarkably, Genie outperformed other models such as ProtDiff (Trippe et al., 2023), FoldingDiff (Wu et al., 2022) or FrameDiff (Yim et al., 2023), and remains comparable to RFDiffusion (Watson et al., 2022)” raises the question of why this performance is remarkable. It is unclear if the determination is left to the reader, whether this evidence should be shown in an Appendix figure, and whether RFDiffusion is statistically significantly better.

Author response:
- Q: “Remarkably, Genie outperformed other models such as [..] Why is this remarkable?” -  Apologies for the confusing phrasing. We meant for it to be read more like “Note that”. This is updated in the manuscript. 
We mention the capabilities of Genie model as compared to other models to explain our design choice to use Genie in the experiments. In this work we were not investigating how to improve the sample quality in the unconditional generation, but we needed to use a reasonable unconditional model to test the conditioning methodology. We chose Genie because of its satisfactory performance that according to the Genie paper [https://arxiv.org/abs/2301.12485] outperforms other models, as well as because of its code availability.

Reviewer comment (Evaluation):
In the statement “the mean chain distance that should be close to 3.8 A;” it is unclear where the value 3.8 originates from.

Author response:
- Q: “ the mean chain distance that should be close to 3.8 A  Where does 3.8 come from?” - thank you for this comment, we should have clarified that we refer to the distances of subsequent alpha carbons in the backbone here. The alpha carbon atoms (CA) in subsequent amino acid residues in a protein are physically constrained to lie in a very narrow range of around 3.8 Angstrom [c.f. e.g. https://www.wiley.com/en-us/Biochemistry,+4th+Edition-p-9780470570951]. Here we consider the mean $C_\alpha-C_\alpha$ distance, that is known to be close to 3.8 A.

Reviewer comment (Evaluation):
It is unclear whether the values of NMA loss differ for strain or random targets, and whether the authors expect them to differ. If so, this seems like a straightforward statistical quantity to calculate.

Author response:
- Q: “Do the values of NMA loss differ for strain or random targets? Do you expect them to?” – Good question! As illustrated in Figure 2, the loss values are indeed fairly comparable between the two situations, however, we find it hard to give an intuition as to what to expect in these general cases. Instead, we would like to highlight that “strain” and “random” are meant mostly as more general demonstrations that the conditioning works for a variety of target definitions and substructures and that we use two kinds of targets in order to show that NMA-loss can be minimised regardless of how we choose the target. The “hinge” target formulation on the other hand is meant to capture the more application relevant situation of trying to design a certain motif with a targeted low mode behaviour.

Reviewer comment (Writing):
In the statement “In the analysis of the remaining samples, we considered the distributions of NMA-loss and scTM-score (Figure 5),” it appears that the second figure may instead be RMSD. If they are the same, this should be defined.

Author response:
- Q: ““In the analysis of the remaining samples, we considered the distributions of NMA-loss and scTM-score (Figure 5). Isn’t the second figure RMSD?” - apologies for the confusing phrasing here. The distribution of NMA-loss is presented at Figure 5, scTM-score is discussed later in the text. We updated the phrasing in the text.

Reviewer comment (Novelty):
The paper would benefit from a brief statement on how the authors envision their approach to be useful in practice. In settings where we have information about dynamics that we wish to impose on generated samples, we are presumably operating in a fairly constrained setting, where we wish to resample only parts of a protein. The authors should clarify the advantages of conditioning a diffusion model on dynamics compared to, for instance, using normal mode analysis (or coarse-grained molecular dynamics) to generate a structural ensemble and then using a model to predict amino acid identities conditioned on structure (i.e., some inverse folding model).

Author response:
Specifically to your point on **how we envision this method to be used**:  
If we have a selected motif (e.g. an enzymatic active site or pocket, binding cleft/pocket) that performs backbone rearrangements that are well captured by NMA during biological function (e.g. hinge or shear type backbone rearrangements), then this could be “grafted” onto a new scaffold that not only preserves the geometry of the motif, but also creates a scaffold backbone that promotes the relative motion of the motif that is needed to achieve the biological function. This idea is driven by the observation in the NMA community that some protein backbones pre-determine them for certain types of motion [https://pubmed.ncbi.nlm.nih.gov/16143512/], and that this motion is – for selected classes of proteins – the biologically relevant motion. 

We envision this to be useful to transfer functional motifs to different backbones which might have different additional binding domains to confer specificity, or lack thereof to promote promiscuity, while preserving the topology required for the functional motif’s primary motion. 
Besides coming from an existing protein, a functional motif with a dynamic behavior (e.g. a pincher motion) could also be the result of a protein engineering or design hypothesis.

Reviewer comment (Theory):
Page 4, eq (8) — Why is there no weighting factor (aka temperature) on the loss term? In other words, why would you expect this to be 1?

Author response:
- Page 4 eq (8) - in the current model specification, a weight factor in the loss function does not affect the final result
Consider our loss in Equation 1 after we cast it to the form required in the conditional score (Equation 15) with an added temperature $\gamma$:
$$
\lambda_t \nabla  exp(-l(\hat{x}_0(x_t) ,y)/ \gamma) = - \frac{\lambda_t}{\gamma} \nabla  l(\hat{x}_0(x_t) ,y)
$$
with the guidance-scale $\lambda_t$. After reparametrisation $\hat{\lambda}_t  = \frac{\lambda_t}{\gamma}$
$$
\lambda_t \nabla  exp(-l(\hat{x}_0(x_t) ,y)/ \gamma) = - \hat{\lambda}_t \nabla  l(\hat{x}_0(x_t) ,y)
$$
the temperature $\gamma$ has no effect as $\hat{\lambda}_t$ is tuned.

Reviewer comment (Writing):
Page 5, "x_C_M \in ... is the expected positions of conditioned residues at t=0 as sampling progresses" — This sentence was confusing. Doesn't "sampling progresses" imply a "t" different from "t=0" (and at t=0, don't you know the position?). In this case, shouldn’t this expected position be subscripted with a time t?

Author response:
- Page 5, "x_C_M \in ... is the expected position of conditioned residues at t=0 as sampling progresses" - We rephrase that entire sentence in the updated manuscript, but please find additional clarification here.
At each step in the sampling, which progresses from t=T_max to t=0, in order to calculate the NMA-loss or the structure loss we must obtain the model prediction of the residues of interest in the final backbone at t=0. By that we mean the “expected positions of conditioned residues at t=0“. Importantly, this is what differentiates the guidance paradigm we use here (reconstruction based guidance) from general classifier-based guidance – where a predictor network (or function) is needed that can predict using noisy data. While NMA is somewhat robust to noise, by predicting the current estimate $\hat{x}_0$ via Tweedie’s formula, we apply NMA to a more protein-like structure ($\hat{x}_0$ instead of $x_t$), moving us closer to the realm in which NMA is valid.

Reviewer comment (Experiments):
Page 7, "Occasionally the conditional sampling will ‘blow up’, which has been generally observed in many conditional diffusion models (Lou & Ermon, 2023)" — Does the rate of blowing up depend on the "guidance scale" used during sampling?

Author response:
- Page 7. "Occasionally the conditional sampling will ‘blow up’, which has been generally observed in many conditional diffusion models (Lou & Ermon, 2023)" - We apologise for the use of informal language and updated this statement in the manuscript, and your hypothesis that the “blow up” is connected to the guidance scale is indeed correct. By ‘blow up’ we refer to a process where a coordinate increases by orders of magnitude along the sampling trajectory, or even becomes NaN. This phenomenon tends to happen when the conditioning pushes a sample’s coordinates outside of the realm of observed samples for which the denoiser was trained (and therefore it predicts mostly random scores there). This phenomenon underlines that finding a right balance between the NMA-loss driven part of the score and the unconditional part of the score requires striking a delicate balance, one of our contributions. 
This “divergence” effect is not unique to the denoiser we use and it has been generally found that many other large diffusion models working with image data suffer from diverging features in the sampling. In images they usually combat this with the ‘thresholding trick’ - they clip the feature values to never exceed some pre-defined range. Within the image domain that results in samples with too much colour saturation. For instance, in Stable Diffusion, which uses weights to balance conditional score and the unconditional score, too much weight on the conditional score leads to mode collapse and out of distribution samples [https://arxiv.org/pdf/2311.00938.pdf].

Reviewer comment (Writing):
Page 2, "as well as we perform the visual inspection." — Something is wrong in this sentence. Rephrase.

Author response:
- Page 2, "as well as we perform the visual inspection." Something is wrong in this sentence. Rephrase. - We rephrased this sentence for more clarity. We wanted to highlight that in addition to using our custom loss function for monitoring the effectiveness of our method, we also manually inspected the agreement of target and sample displacement vectors for selected samples and concluded that the results from our loss function agree with manual inspection. To further provide you with quantitative metrics, we are adding an experiment studying whether the designed-for normal mode is exhibited after inverse folding (we call this self-consistency NMA “scNMA” – refer to the general response for more details).