From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper studies protein backbone 3D structure generation using diffusion models. All reviewers acknowledged the significance of this work in terms of methods and results. The issues raised during review have been largely resolved during rebuttals. Thus an accept is recommended.
justification_for_why_not_higher_score: There are minor issues that remain.
justification_for_why_not_lower_score: All reviewers acknowledged the significance of this work in terms of methods and results. The issues raised during review have been largely resolved during rebuttals. Thus an accept is recommended.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper1427/Reviewer_BXA4
title: Response to authors
comment: Thank you authors for the updates to the work as well as responses to myself and the other authors.

I have adjusted my score accordingly.

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Checking in
comment: We thank the reviewers again for their thoughtful reviews. We write again in hopes of sparking further consideration and discussion.

We believe that we have addressed all major reviewer concerns in our replies, and that the experiments and discussion we have added in response to reviewer comments have significantly improved the paper.  We ask that the reviewers consider these changes and reply back with any concerns they feel have not yet been addressed.

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to reviewer EPhs
comment: We thank the reviewer for their careful attention and constructive feedback. Please see our responses to all reviewers in the top-level comment and our individual responses below.

**“Topic is a little bit narrow…”**: We believe motif-scaffolding is a widely applicable technique of considerable importance. As stated by reviewer KTe1, “A system that performs well at the motif scaffolding task could be revolutionary for a variety of applications.” Motif-scaffolding can be applied to a wide range of applications such as vaccine and enzyme design or designing new protein-protein interactions [1].

**“ProtDiff […] is similar to existing EDM models”**:  We agree that the underlying architecture of ProtDiff is similar to existing EDM models, but respectfully do not view this as a weakness.  We believe that our work on ProtDiff forms an important technical contribution. But we did not intend to emphasize our architectural modifications specific to proteins (such as encodings of sequence position and sequence distance). Rather, we believe our contribution lies in our empirical demonstration that (with these modifications) ProtDiff is capable of generating backbone samples that structurally agree with AlphaFold2 predictions. To avoid implying that the architectural modifications are key contributions, we have further de-emphasized these details by moving the “existing EDM model”-like details to Appendix C.

**SMCDiff on synthetic data**: Thank you for this suggestion. We have updated the manuscript with Appendix I where we apply SMCDiff to inpainting MNIST images. We find SMCDiff produces visually better inpainting samples over the replacement method. We hope our new experiment demonstrates SMCDiff’s applicability to general data and that future research can explore SMCDiff in other tasks.

**Inpainting vs. conditional model**: Our approach has the benefits of training a single model for unconditional and conditional generation. Part of our contribution is a theoretical result that SMCDiff can achieve exact conditional samples with enough particles and an accurate unconditional diffusion model. While this formulation is elegant and leads to simpler training, it is possible training a conditionally trained generative model would do better. 

For our task scenario of motif scaffolding, another advantage of our approach is that one does not need to pick which part of the protein is the motif, and which is the scaffold, at training time. But instead one can use arbitrary conditionals at test time. While amortization is an option (using an array of different inpainting tasks) the need to choose that distribution is also a challenging problem. But we suspect conditional training is a worthwhile direction to explore.

**Citations**: Thank you for alerting us to this issue. We confirm that we are experiencing this issue. Oddly, we don’t have this issue for the version we currently compiled before upload to OpenReview. Our hope is that the updated pdf will work when downloaded from OpenReview, but we will follow up with a later reply if the bug persists for us. Please let us know if it persists for you.

[1] [Wang et al. Scaffolding protein functional sites using deep learning. Scaffolding protein functional sites using deep learning. Science, 377(6604):387–394, 2022.](https://www.science.org/doi/10.1126/science.abn2100)

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to Reviewer BXA4
comment: We thank the reviewer for their careful attention and constructive feedback. Please see our responses to all reviewers in the top-level comment and our individual responses below.

**Validity of AlphaFold2:** We agree checking the validity of AF2 without MSA for scaffold design is important to justify our evaluation. Motif design is not a part of our work. We address your questions in our top-level comment.

**Choice of 6exz and 5trv for motif scaffolding test cases:**  As noted in our reply to reviewer KTe1, we have presented results on 6exz and 5trv because we believe they are sufficient to demonstrate our main claims. However, the choice of these proteins in particular was largely arbitrary; construction of a more comprehensive benchmark set of motif scaffolding tasks is an important direction for future work.

**Self-consistency (Figure 3A):** We have trouble understanding this question. Could the reviewer clarify what they mean by “how do you know that your scTM is > 0.5, so it is “designable” ?
* If the reviewer means to ask how we compute scTM for a generated backbone: we (1) use ProteinMPNN to design a sequence, (2) Use AF2 (without MSA) to predict a structure corresponding to that sequence, and (3) Run TM-align to compute scTM as the TMScore between the generated and predicted backbones.
* Or if the reviewer means to ask how we know the generated backbone is “designable”: “Designable” is the term we have chosen to denote if a generated structure and sequence matches the AF2 predicted structure. We determine this based on whether scTM > 0.5, a threshold proposed to indicate when two structures are roughly the same [1]. 
Please let us know if this does not answer your question.

**Interpolation:** Figure 4 (as well as Appendix H.3) mainly serves to give us visual intuition for the relationship between the noise in the reverse diffusion and the generated structures. Recall that diffusion models perform sampling by iteratively transforming noise into data (see Figure 1). They inject additional noise $\epsilon^{(0:T)}$ at each step. We obtain the plots in Figure 4 by **interpolating between two independently sampled noise trajectories**, $\hat{\epsilon}^{(t)} = \sqrt{\alpha} \epsilon^{(t)} + \sqrt{1 - \alpha} \tilde{\epsilon}^{(t)}$, where $\alpha$ ranges within 0 to 1 at every time step. The plots give us a sense of what small differences in the underlying noise change about the structure itself.

**Invalid intermediate folds.** Figure 4 does indeed reveal the model sometimes samples invalid folds due to chain overlaps and clashes during interpolation. This is a known limitation of our method as mentioned in Section 6 and Figure 7. We believe improved modeling and training will help solve these limitations.

**Scaffold vs. motif.** Motifs are structural protein fragments imparting biological function while scaffolds are additional protein structures surrounding the motif to stabilize the motif’s shape and folding. For example, the Fab domain of an antibody can be thought of as having two parts: strands in a β-sandwich (the scaffold) which support the binding loops (the motif). We have added definitions of motifs and scaffolds to the Introduction.

**Section 2.1:** We have clarified this wording and added additional information about modified proteins in Appendix B.

**Section 5.2:** Thank you for catching this incorrect wording. We meant structure and have replaced the wording.

[1] [Zhang and Skolnick "TM-align: a protein structure alignment algorithm based on the TM-score" Nucleic Acids Research, 2005](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1084323/)

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to reviewer KTe1 (2/2)
comment: 
**Minor comments**:

* **Limitation of prior work**: To the best of our knowledge, the authors of existing papers do not offer explanations for why their ML methods do not scale beyond 20 residues [3].
The only method for going beyond 20 residues is hallucination, but hallucination relies on overly expensive Monte Carlo sampling. We suspect that a major advantage of both our method and hallucination over non-probabilistic methods is the ability to stochastically search the space of backbone structures and choose the best one; this space becomes larger and more complex at larger residue scales (and thus benefits from stochasticity in search). We have updated our introduction to reflect this discussion.

* **Structure and sequence relationship**: We thank the reviewer for pointing out this over-simplification in our initial submission.  We have updated the statement to make our choice of this modeling abstraction more explicit, and added additional discussion of how structure relates to sequence in Appendix B.

[1] https://en.wikipedia.org/wiki/EF_hand 

[2] [Lee et al. EquiFold: Protein Structure Prediction with a Novel Coarse-Grained Structure Representation. bioRxiv 2022](https://www.biorxiv.org/content/10.1101/2022.10.07.511322v1)

[3] [Wang et al. Scaffolding protein functional sites using deep learning.  Scaffolding protein functional sites using deep learning. Science, 377(6604):387–394, 2022.](https://www.science.org/doi/abs/10.1126/science.abn2100)

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to reviewer KTe1 (1/2)
comment: We are excited to hear that the reviewer enjoyed our paper! We thank the reviewer for their careful attention and constructive feedback. Please see our responses to all reviewers in the top-level comment and our individual responses below.

**SMCDiff**: Thank you for the insightful comment that SMCDiff could be applicable to the wider ML community in other settings. Thanks to this prompt and another reviewer’s suggestion, we have performed additional experiments on MNIST – yielding preliminary evidence of SMCDiff’s general applicability. We have highlighted these new experiments in our main response to all reviewers.

**Scale of experiments**: Most importantly, we hope to convince the reviewer that the experiments we have are sufficient to support our claims; then we turn to the challenges of further experiments and our new updates in the revision. To that first goal, recall our key claims and note that they are both existence claims and thus supported by a small number of examples:

1. Generative modeling has the ability to produce long and diverse scaffolds; our first experiment demonstrated the utility of SMCDiff to **produce multiple scaffolds for the same motif**. As a result, SMCDiff is capable of generalizing beyond the training set since each motif has only one natural scaffold. But the regime of very small motifs and long scaffold is still challenging.

2. This first successful demonstration of motif scaffolding via generative modeling is **NOT** state of the art, but rather fails on difficult examples with motifs not in the training set (see as well the second example we have added of this, on the EF-Hand motif).

We make additional supporting claims – such as that SMCDiff provides an advantage over the alternative inpainting approaches that approximate conditional sampling. This claim might benefit from additional benchmarking, so we have added new support; in addition to our previous case studies (6exz, 5trv, RSV), we now study MNIST and an additional example,  EF-hand (which we discuss next).

While larger scale experiments would have been possible, we have not included them because we mostly felt the value they might add did not justify the computational cost of the experiments (which is large because because we considered many different scaffold lengths, three different in-painting methods, and our use of many replicates). The exception to this point is that we did agree that there would be benefit from more diverse biologically meaningful examples. To that end, we have updated our manuscript with results of another motif, EF-hand [1], a calcium binding site, which presents the challenge of scaffolding non-consecutive residues which are also purely loops. Finally, we note that providing further biologically meaningful examples requires substantial expertise and curation; although outside the scope of this paper, we believe it would be a useful contribution to develop a benchmark suite of even more biologically meaningful examples.

**Length of proteins**: Our choice of training on proteins up to length 128 is due to modeling limitations of an E(3)-equivariant score network on C-alpha backbone atoms. The chance of left-hand helices increases with length and the overall self-consistency success rate decreases when we train on longer proteins. We noticed in early experiments that our results were much worse for length <128 when training up to length 256. We did not explore alternate neighborhood-based message passing in the present work. Utilizing a coarse-grained representation of the structure – as in concurrent work [2] – offers an exciting potential method for improving scaling to longer proteins. Extending to longer proteins with improved protein representations is an important direction of further research.

**Types of motifs**: The motif in Fig 2A is a contiguous helix, which may seem trivial, but is still challenging when the model has to sample long scaffolds as our results in Figure 3 show. Our new motif, EF-hand, increases the diversity of our motifs. In particular, it exhibits non-contiguity, a common characteristic of enzymatic motifs. We have added a new explanation of the challenges of our motifs to Section 5.2.

**Reliance on ProteinMPNN**: Indeed, ProteinMPNN is not a perfect method. Determining the overall self consistency rate with ProteinMPNN is an excellent suggestion. We are unable to compute self-consistency over all of PDB due to limited compute resources. We analyzed scTM over the training set of proteins length <128 and found 87% percent of these examples to fall in the designable (scTM>0.5) range. We have added this statistic to the manuscript in Section 5.1.

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to reviewer eV4U
comment: We thank the reviewer for their careful attention and constructive feedback. Please see our responses to all reviewers in the top-level comment and our individual responses below.

**Coordinate scaling**: We thank the reviewer for pointing out the issue of convergence to the isotropic Gaussian because we think it highlights some problems with the clarity of our wording, which we address in the revision. First, note that by tailoring the noise schedule to the data, diffusion models should always result in $x^{(T)}$ being nearly isotropic Gaussian. In particular, diffusion models proceed by iteratively adding noise based on a noise schedule $\beta^{(t)}$. At time $T$, we have:
$x^{(T)} \overset{d}{=} \sqrt{\bar{\alpha}^{(T)}} \ x^{(0)} + \sqrt{1 - \bar{\alpha}^{(T)}} \ \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$. 
In our case (with $\beta^{(1)}=10^{-4}, \beta^{(T)}=0.02, T= 1000$, and so $\bar{\alpha}^{(T)}= 3.2 * 10^{-5})$ the final time step has nearly no signal remaining.

However, there are practical concerns. If $x^{(0)}$ is standardized to start with, we can expect that most reasonable schedules and choices of $T$ will practically achieve an $x^{(T)}$ nearly isotropic Gaussian. To that end then, we start by effectively standardizing $x^{(0)}$. To double check that $x^{(T)}$ behaves as desired, though, we have now added an empirical test to the revision (in Appendix C) that indeed our $x^{(T)}$ is nearly isotropic Gaussian. We also moved our discussion of scaling to Appendix C and clarified our wording, to make sure this point isn’t a distraction to readers.

**Validity of Proposition 4.1**: Thank you for pointing out a lack of clarity in our statement.  We updated the Introduction and Section 4 to make clear the assumptions when we discuss Proposition 4.1. The significance of Proposition 4.1 is to demonstrate correctness guarantees over the previous replacement method [1], which was stated to only be “approximately correct” even when $p_\theta$ is exact. As a result, SMCDiff guarantees to sample from the correct conditional distribution given the exact $p_\theta$ and enough particles. In practice, we can never know $p_\theta$ exactly and will incur errors with conditional sampling. However, SMCDiff reduces the sources of error to only how well we approximate $p_\theta$ rather than also having systematic error from the algorithm, which is the case for the replacement method.

**Modeling chirality**:  We agree that EGCL is another limitation that prevents the model from being aware of reflections. We have clarified this in the Discussion (Section 6). Our motivation for ProtDiff comes from a previous state-of-the-art diffusion method on molecule generation, EDM [2]. Exploring alternative model architectures was not a focus of this work, but we believe that further exploration of architectures that break reflection symmetry may have potential to address this limitation.

[1] [Song et al. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.](https://arxiv.org/abs/2011.13456)

[2] [Hoogeboom et al. Equivariant diffusion for molecule generation in 3D. In International Conference on Machine Learning, 2022.](https://arxiv.org/pdf/2203.17003.pdf)

From: ICLR.cc/2023/Conference/Paper1427/Authors
title: Response to all reviewers
comment: We thank the reviewers for their careful attention to our paper and constructive feedback. We next summarize our contributions and address concerns shared across reviewers. We respond to individual reviewers in separate comments.

**Summary**: We believe that our framing of the motif-scaffolding problem as conditional sampling opens a route to leverage the exciting potential of generative machine learning. We propose the first concrete generative methodology for this problem by developing two new tools: an adaptation of recent equivariant diffusion models to proteins and a surprising new connection between diffusion models and sequential Monte Carlo ideas. Our method simultaneously (1) substantially increases the proposed scaffold size over previous methods (80 residues vs. fewer than 20 residues) and (2) samples diverse scaffolds instead of just a single scaffold.

We believe the technical contributions are significant and of high impact. As stated by reviewers,
* “Formulation of motif-scaffolding problem as a conditional sampling process based on denoising diffusion probabilistic models is novel.”
* “A system that performs well at the motif scaffolding task could be revolutionary for a variety of applications.”
* “The paper brings together disparate ML ideas (diffusion models, sequential Monte Carlo, protein design) in an elegant way.”

We next turn to shared concerns.

**Generality of SMCDiff beyond protein applications**: 

We are excited to hear multiple reviewers believe that SMCDiff will be of independent interest to the community. To check the general applicability of SMCDiff, we have conducted additional experiments empirically testing the method on MNIST, and we indeed find that it improves conditional sampling over the method in [1] when only an unconditional diffusion model is used. In our revision, we have added these experiments to Appendix I and reference them in our discussion (Section 6).

**Reliability of ProteinMPNN and AlphaFold 2 without Multiple Sequence Alignments**:

Multiple reviewers have expressed concerns regarding the reduced accuracy of using AlphaFold2 (AF2) without multiple sequence alignments (MSAs). An advantage of  **ProteinMPNN and AF2 without MSA is that this approach has been validated to work well in laboratory experiments** for de novo protein design applications [2, 3]. Part of this comes from empirical evidence that Af2 without MSA is highly accurate on de novo proteins as seen in Figure S7 of [4] where Af2 without MSA achieves >75 pLDDT on de novo proteins; moreover, without ProteinMPNN for sequence design, [5] has shown AF2 to provide highly-accurate MSA-free structure predictions on de novo designed proteins.  While benchmarking protein language model methods, such as OmegaFold [6] and ESMFold [7], with ProteinMPNN is a valuable study, it is not in the scope of our work.

We have **updated Section 5.1 with this information and associated references**.

[1] [Song et al. Score-based generative modeling through stochastic differential equations. In International
Conference on Learning Representations, 2021.](https://arxiv.org/abs/2011.13456)

[2] [Dauparas et al. Robust deep-learning based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022.](https://www.science.org/doi/full/10.1126/science.add2187)

[3] [Bennett et al. Improving de novo protein binder design with deep learning.
bioRxiv, 2022](https://www.biorxiv.org/content/10.1101/2022.06.15.495993v1)

[4] [Wang et al. Scaffolding protein functional sites using deep learning. Science, 377(6604):387–394, 2022.](https://www.science.org/doi/10.1126/science.abn2100)

[5] [Chowdhury et al. Single-sequence protein structure prediction using a language model and deep learning. Nat Biotechnol 40, 1617–1623 (2022).](https://www.nature.com/articles/s41587-022-01432-w)

[6] [Wu et al. High-resolution de novo structure prediction from primary sequence. bioRxiv 2022 ](https://www.biorxiv.org/content/10.1101/2022.07.21.500999v1 )

[7] [Lin et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv 2022](https://www.biorxiv.org/content/10.1101/2022.07.20.500902v1)

From: ICLR.cc/2023/Conference/Paper1427/Reviewer_eV4U
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In this paper, authors propose a E(3)-equivariant diffusion probabilistic model for both unconditional and conditional sampling of protein backbones, which can be used for the motif-scaffolding problem. Particularly, the conditional sampling is implemented as a particle filtering algorithm, where scaffolds that are more consistent with the motif are prioritized. Authors demonstrate that the proposed method generated valid scaffolds for the given motif, as verified by AlphaFold2’s predicted structures. Experiments on unconditional sampling indicate that this model captures the underlying distribution of protein backbones.
strength_and_weaknesses: Pros:
1. The formulation of motif-scaffolding problem as a conditional sampling process is novel, and authors propose a single diffusion model that can be used for both unconditional and conditional sampling to realize such motif-based scaffold design.
2. The re-weighting scheme in the particle filtering algorithm is based on the likelihood assessment of the diffusion model, which can better guide the conditional sampling process towards the promising direction.
3. Clear discussion on the modeling and data limitation of the proposed method.

Cons:
1. During the training of ProtDiff, the diffusion model of protein backbones, 3D coordinates are represented in nanometers, rather than Angstrom, so that the spatial scale is similar to an isotropic Gaussian with unit variance. I am not sure whether such scaling is sufficient to ensure that x^{T} at the last step is indistinguishable with an isotropic Gaussian, which is one of the fundamental assumptions in DDPM.
2. In Section 4.2, authors state that the proposed SMCDiff can provide arbitrary accuracy for conditional sampling, as guaranteed by Proposition 4.1. However, such proposition is based on the assumption that p_theta **exactly** matches the forward diffusion process, which might be too strong and even unrealistic.
3. In Section 5.1, authors mention that when prediction protein structure with AF2, no multiple sequence alignment (MSA) is used. However, as shown in several single-sequence PLM-based protein structure prediction papers, AF2 w/o MSA input works inferior to such PLM-based structure prediction, possible due to the gap between training and inference inputs. Authors may need to consider feed MSA into AF2 for more reliable predictions, or directly use PLM-based methods (e.g., ESM-Fold, OmegaFold and HelixFold-Single) for in silico evaluation.
4. In Section 6, authors state that only using 3D backbone coordinates as inputs to ProtDiff is the main cause that ProtDiff is reflection symmetric and may generate left-handed helices. However, I think the EGCL-based network architecture may also be one of root causes (even the biggest one), since EGCL only takes pairwise distance as spatial information, which is reflection invariant. In other words, ProtDiff cannot distinguish one structure and its mirrored counterpart by design, and this won’t be resolved even full-atom 3D coordinates are taken as inputs.
clarity,_quality,_novelty_and_reproducibility: Clarity: Mostly satisfying. Section 4 may need further refinement for easier understanding.

Quality: Good.

Novelty: The formulation of motif-scaffolding problem as a conditional sampling process based on denoising diffusion probabilistic models is novel.

Reproducibility: Code is included in the supplementary material (although I have not tried yet).
summary_of_the_review: The DDPM-based conditional sampling formulation of motif-scaffolding problem is novel. Authors propose ProtDiff and SMCDiff for both unconditional and conditional sampling of protein backbones. The validness of some assumptions (Cons 1 & 2) may need further discussion.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1427/Reviewer_KTe1
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: For many protein design applications, the user specifies a motif, and then needs to build a scaffold of additional amino acids that will act to stabilize the motif. This paper proposes an interesting technique for generating scaffolds conditional on motifs. First, a 3D diffusion model of protein structures is fit. Then, conditional 'inpainting' is performed to sample the 3D configuration of the scaffold given the motif using a particle filtering technique. Then, an fixed structure -> sequence model from prior work is used to produce a protein sequence. Given the accuracy of AlphaFold, the overall system can be evaluated without performing prohibitively expensive wet-lab experiments: simply check whether the AlphaFold-predicted structure of the output sequence contains the desired motif.  
strength_and_weaknesses: =Strengths=
The paper is very well grounded in current literature. 
The paper addresses a high-impact application.
The paper provides a clean use of sequential Monte Carlo (SMC) that will be of independent interest to the community.

=Weaknesses=
Using SMC for inpainting with diffusion models is a generic idea that could have high impact, for example, to computer graphics research. However, I worry that that target audience won't read this paper because it is quite focused on proteins. If the paper was more methods oriented 'hey, we have a generic SMC method for diffusion models…", it could perhaps have higher impact.

The experiments are careful and well-thought out, but mostly focus on a couple of small-scale case studies. I don't quite understand why the analysis of the conditional sampling (aka motif scaffolding) couldn't be done at a larger scale (see below). This will hurt the impact of this paper.

clarity,_quality,_novelty_and_reproducibility: I work in this research area. The paper's problem formulation is well-motivated. A system that performs well at the motif scaffolding task could be revolutionary for a variety of applications.

The paper is very well written and approachable. I also appreciate that it is frank about the shortcomings of the current experiments, etc. 

The paper brings together disparate ML ideas (diffusion models, sequential Monte Carlo, protein design) in an elegant way. 

Implementation for the paper's method is non-trivial (e.g., SMC is annoying to implement). I appreciate that the authors have released code.


summary_of_the_review: Overall, I really like this paper. I think the method is really interesting and it is presented well. However, there are clear concerns around the experiments, as they mostly focus on a couple of specific case studies and there are clear shortcomings of the current method (e.g. ability to capture chirality). Below are some questions mostly to clarify why more comprehensive experiments would be challenging.

=Scale of experiments=
I don't fully understand why sec 5.2 is only on a couple of example proteins. What is the limiting factor? Is it that we don't know good motifs to scaffold? That other natural targets for scaffolding have protein sequences that are too long? Were you compute bound? 

=Length of proteins=
I understand that scalability was a key concern, and thus you trained on sequences of length < 128. Was this a computational or modeling issue? Do you have to actually model every single amino acid in order to capture the gist of the 3d structure? I wonder if the diffusion model could be defined over a coarser-grained version of the structure.

=Types of motifs=
Fig 2a doesn't seem that challenging of a motif. Is it just a single helix? Is the motif consecutive in the primary sequence? For enzymes, for example, I'd imagine that the motif (the active site) would correspond to non-consecutive residues.

=Reliance on ProteinMPNN=
Surely there are realizable structures that ProteinMPNN can't produce a sequence for. It would be good to know if this is a bottleneck in your system. Do you know what the self-consistency rate of ProteinMPNN is in general? What fraction of PDB structures are such that structure -> sequence -> predicted_structure has high similarity between structure and predicted_structure?

=Minor comments=
It would be helpful in the intro to provide a bit more rationale for why prior work only considered 20 residues. It was hard for me to understand why the current work doesn't scale.

"A protein’s structure is a function of its sequence, so we may write x(s)." You should acknowledge at some point in the paper that the notion of a single static structure for a sequence is fraught.

I appreciate the negative result at the bottom of page 7!

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper1427/Reviewer_EPhs
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper studied protein generation problem, and specifically, motif-scaffolding conditional protein design problem. This paper first proposed a general protein 3D structure generation framework, which can generate high-quality protein backbones in 3D. Then the author proposes a new sampling algorithm for inpainting problem, which means part of the data is masked while the model aims to recover the missing part. Analytical experiments demonstrate the effectiveness of the proposed method for both unconditional and conditional protein design.
strength_and_weaknesses: Strength:

1. The paper is well organized. The domain-specific motif-scaffolding problem is clearly explained, and the method is easy to follow.
2. The paper nicely provides technical contributions from both biology and ML perspectives.
3. The paper conducts careful experiments to demonstrate the effectiveness of the proposed method.

Weakness:

1. The topic is a little bit narrow, which is even a subarea of general protein generation. Therefore, the paper may not be attractive to ML conferences.
2. The protein generation framework (ProtDiff) itself is similar to existing EDM models.
clarity,_quality,_novelty_and_reproducibility: The paper is pretty clear but still has several parts kind of confusing. See my questions for details.

Quality is very high, with paper well polished and experiments carefully designed.

Novelty is a little limited. But the task combined with the diffusion model solution is somehow new to the specific community.

Reproducibility is pretty good. Details for reproducing the experiments are well explained and the source code is also provided.
summary_of_the_review: The paper aims to solve the protein motif-scaffolding problem. The author mainly proposed ProtDiff and SMCDiff methods. ProtDiff is for protein generation in 3D, which is similar to previous EDM model for molecule generation. SMCDiff is a new proposed conditional sampling method. Experiments are well designed and sound, showing the model can generate reasonable and realistic structures.

Questions:
1. I think from the description of SMCDiff sampling, it is a general sampling algorithm that is not only for proteins. Then, whether you can also show the effectiveness of SMCDiff sampling for general data? I understand running experiments for large-scale images costs too much time, but maybe you can test it on some synthetic data?
2. Besides, since this is a conditional generation task, I'm a little confused why you only define the problem as an "inpainting" problem instead of a conditional generation problem. In other words, for this conditional generation task, you actually don't need to learn the joint distribution and just do conditional sampling. Maybe I misunderstand the available dataset or your tackled scenario, but in my mind in the learning procedure, you can also just learn a conditional model, which may make more sense.
3. By the way, for the provided PDF on my PC, the hyper references (e.g., citations) don't work correctly. Clicking on the reference will always be directed to the first page, making it difficult for me to check related work or equations. Please double-check this.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper1427/Reviewer_BXA4
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors translate diffusion models to the protein structure domain to design new protein scaffolds and motifs.

strength_and_weaknesses: The diffusion approach seems interesting and a promising way to explore sequence and structure space.

Section 5.1 - Two works are cited to justify using AlphaFold2 as the gold standard of accuracy of those structures. But for de novo scaffold and motif design, as you propose here, has this been validated? To what extent? Shouldn’t this be done first to make sure you can trust those predictions? Minimally, how accurate is it without a multiple sequence alignment?

Section 5.2 - Why PDBs 6exz and 5trv?

In Figure 3A, it looks like the vast majority of scTMs is around 0.25, which is random. If you are using this software during inference, how do you know that your scTM is > 0.5, so it is “designable? 

In Figure 4, it looks like chains are able to pass through one another in space during sampling. This does not happen for proteins in vitro or in vivo. Does your model allow for chains to pass through one another?

clarity,_quality,_novelty_and_reproducibility: For a general audience, and someone well versed in structural biology, it would be great to concretely clarify between scaffold and motif. How do they differ? From your initial paragraph in the introduction, they are described in the same sentence, and still convolved. From my understanding, neither the structure nor the motif is the biological function itself–it’s just the shape that supports the function.

Section 2.1 - “ A protein’s structure is a function of its sequence…” This is not true in the case of post-translational modifications. In the case of entities in the PDB, many modified proteins exist.

Section 5.2 - “For each topology, we chose a…” What do you mean by topology? Do you simply mean that given structure? Or do you mean Topology in the CATH nomenclature?

Figure 4 - It is difficult to understand the accuracy of the interpolations. What are you interpolating in between? Are they even valid folds between interpolations? What does interpolation look like in other methods? It is unclear what message this figure is conveying to the reader.

summary_of_the_review: While I think it is an interesting approach and great area of exploration, more work needs to be done to justify it's utility.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold