D IFFUSION PROBABILISTIC MODELING OF PROTEIN BACKBONES IN 3D FOR THE MOTIF - SCAFFOLDING PROBLEM
Anonymous authors
Paper under double-blind review
A BSTRACT
Construction of a scaffold structure that supports a desired motif , conferring protein function, shows promise for the design of vaccines and enzymes.
But a general solution to this motif-scaffolding problem remains open.
Current machine-learning techniques for scaffold design are either limited to unrealistically small scaffolds (up to length 20) or struggle to produce multiple diverse scaffolds.
We propose to learn a distribution over diverse and longer protein backbone structures via an E(3)- equivariant graph neural network.
We develop SMCDiff to efﬁciently sample scaffolds from this distribution conditioned on a given motif; our algorithm is the ﬁrst to theoretically guarantee conditional samples from a diffusion model in the largecompute limit.
We evaluate our designed backbones by how well they align with
AlphaFold2-predicted structures.
We show that our method can (1) sample scaffolds up to 80 residues and (2) achieve structurally diverse scaffolds for a ﬁxed motif.
I NTRODUCTION
A central task in protein design is creation of a stable scaffold to support a target motif . Here, motifs are structural protein fragments imparting biological function while scaffolds stabilize the motif’s structure.
Vaccines and enzymes have already been designed by solving certain instances of this motif-scaffolding problem (Procko et al., 2014; Correia et al., 2014; Jiang et al., 2008; Siegel et al., 2010).
However, successful solutions to this problem in the past have necessitated substantial expert involvement and laborious trial and error.
Machine learning (ML) offers the hope to automate, and better direct this search.
But existing ML approaches face one of two major roadblocks.
First, many methods do not build scaffolds longer than about 20 residues. For many motif sizes of interest, the resulting proteins would be smaller than the shortest commonly-studied simple protein folds (35–40 residues) (Gelman & Gruebele, 2014).
Second, while other methods may generate longer scaffolds using stochastic search techniques, they require hours of computation to generate a single plausible scaffold (Wang et al., 2022; Anishchenko et al., 2021; Tischer et al., 2020).
Moreover, when a plausible scaffold is found, it remains to be experimentally validated.
Therefore, it is desirable to return not just a single scaffold but rather a set of scaffolds exhibiting diverse sequences and structural variation to increase the likelihood of success in practice.
In the present work, we demonstrate the promise of a particular generative modeling approach within
ML for efﬁciently returning a diverse set of motif-supporting scaffolds.
Generative models have been shown to capture a distribution over diverse protein structures (Lin et al., 2021).
But it is not clear how to handle conditioning (on the motif) using these approaches.
Diffusion probabilistic models (DPMs) offer a potential alternative; not only do they provide a more straightforward path to handling conditioning, but they have also enjoyed success generating small-molecules in 3D (Hoogeboom et al., 2022).
Extending DPMs to protein structures, though, is non-trivial; since proteins are larger than small molecules, modeling proteins requires handling the sequential ordering of residues and long-range interactions.
Finally, while existing models often generate distance matrices (Anand & Huang, 2018; Lin et al., 2021), we instead focus on generating a full set of 3D coordinates, which should improve designability in practice.
Our resulting model, ProtDiff , is similar to concurrent work on E(3)-equivariant diffusion models for molecules (Hoogeboom et al., 2022), but with modiﬁcations speciﬁc to protein structure.
Moreover, we develop a novel motif-scaffolding procedure based on Sequential Monte Carlo, SMCDiff , that repurposes an unconditionally trained
We train our new protein backbone diffusion model, ProtDiff , to generate realistic protein backbone structures.
Next, we run SMCDiff , our conditional sampling algorithm, with ProtDiff to generate scaffolds (colored in red) conditioned on the motif (colored in blue).
For self-consistency evaluation, we use a pretrained ﬁxed-backbone sequence-design model ( ProteinMPNN (Dauparas et al., 2022)) to generate the scaffold sequence from a sampled backbone.
We then input the sequence to a structure prediction model, in our case AlphaFold2 (AF2) (Jumper et al., 2021), to generate the full protein structure from the generated sequence.
We compare the backbone of the predicted structure with the original backbone structure using TM-score (Xu & Zhang, 2010) and root-meansquare-distance (RMSD) for the motif.
DPM for conditional sampling.
We prove that if a DPM matches the data distribution, SMCDiff is guaranteed to provide exact conditional samples in a large-compute limit; this property contrasts with previous methods (Song et al., 2021;
Zhou et al., 2021), which we show introduce non-trivial approximation error that impedes performance.
Our ﬁnal motif-scaffolding generative framework, then, has two steps (Fig. 1): ﬁrst we train ProtDiff to learn a distribution over protein backbones, and then we use SMCDiff with ProtDiff to inpaint arbitrary motifs.
Ours is the ﬁrst machine-learning method to construct scaffolds longer than 20 residues around motifs — we build up to 80 residues scaffolds on a test case.
Beyond our progress on the motif-scaffolding problem, we provide the following technical contributions: (1) we introduce a protein-backbone generative model in 3D – with the ability to generate backbone samples that structurally agree with
AlphaFold2 predictions, and (2) we develop a novel conditional sampling algorithm for inpainting.
1. R ELATED WORK
Motif-scaffolding .
Past approaches have sought to scaffold a motif with native or prespeciﬁed protein fragments, but are limited to ﬁnding a suitable match in the Protein Data Bank (PDB) and cannot adapt the scaffold to compensate for slight structural mismatches (Cao et al., 2022; Silva et al., 2016;
Yang et al., 2021; Sesterhenn et al., 2020; Linsky et al., 2020).
More recently Wang et al.
(2022) used pre-trained protein structure prediction networks to recapitulate native scaffolds, but this method failed to generate scaffolds longer than 20 residues and can output only a single candidate scaffold rather than a diverse set.
By contrast, our goal is to sample diverse, long scaffolds.
Diffusion models for molecule generation .
Several concurrent works have extended equivariant diffusion models to molecule generation.
Anand & Achim (2022) extended diffusion models for generation of protein backbone frames and sequences conditioned on secondary structure adjacency matrices.
Similarly, Luo et al.
(2022) focused on CDR-loop generation using diffusion models conditioned on non-CDR regions of the antibody-antigen.
Our method does not require conditioning and is applicable to general proteins.
Lee & Kim (2022) approach the same problem as our work but build diffusion models over 2D distances matrices that requires post-processing to produce 3D structures through Rosetta minimization.
We demonstrate capability of diffusion models to directly model 3D coordinates of proteins.
Hoogeboom et al.
(2022) developed an equivariant diffusion model (EDM) for generating small molecules in 3D. However, because EDM does not enforce a spatial ordering of the atoms that compose small molecules, as we describe in Section 5, it does not learn a coherent chain structure as needed in proteins.
Inpainting and conditional sampling in diffusion models .
Point-Voxel Diffusion (PVD)
(Zhou et al., 2021) is a 3D diffusion model for generating shapes from the ShapeNet dataset.
Though trained to generate shapes unconditionally, PVD completes (or inpaints ) full shapes when a partial point cloud is ﬁxed during inference.
For general diffusion models, Song et al.
(2021) proposed an alternative inpainting approach and remarked that this approach produces approximate conditional samples.
However, these methods do not provide theoretical guarantees, and when we compare them to SMCDiff , we ﬁnd that their approximation error impedes performance when applied to motifscaffolding.
Saharia et al.
(2021) developed an inpainting diffusion model by training a diffusion model to denoise randomly generated masked regions while unmasked regions were unperturbed.
However, their approach requires a detailed data augmentation strategy that does not exist for proteins.
We describe additional related work on protein generative models in Appendix A.
P RELIMINARIES 2. T HE MOTIF - SCAFFOLDING PROBLEM
A protein can be represented by its amino acid sequence and backbone structure.
Let A be the set of 20 genetically-encoded amino acids.
We denote the sequence of an N -residue protein by s 2 A N and its C- ↵ backbone coordinates in 3D by x = [ x 1 , . . .
, x N ] T 2 R N, 3 .
We describe a protein as having a ﬁxed structure that is a function of its sequence, so we may write x ( s ) .
We divide the N residues into the functional motif M and the scaffold S , such that M [ S = { 1 , 2 , . . .
, N } .
The goal is to identify, given the motif structure x
M , sequences s whose structure recapitulates the motif to high precision x ( s ) M ⇡ x M .
Appendix B discusses several caveats of this simpliﬁed framing (e.g. our assumption of static structures).
2. D IFFUSION PROBABILISTIC MODELS
Our approach to the motif-scaffolding problem builds on denoising diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015).
We follow the conventions and notation set by Ho et al. (2020), which we review here.
DPMs are a class of generative models based on a reversible, discrete-time diffusion process.
The forward process starts with a sample x (0) from an unknown data distribution q , with density denoted by q ( x (0) ) , and iteratively adds noise at each step t .
By the last step, T , the distribution of x ( T ) is indistinguishable from an isotropic Gaussian: x ( T ) ⇠ N ( x ( T ) ; 0 , I ) .
Speciﬁcally, we choose a variance schedule  (1) ,  (2) , . . .
,  ( T ) , and deﬁne the transition distribution at step t as q ( x ( t ) | x ( t  1) )
=
N ( x ( t ) ;
[Equation]
DPMs approximate q with a second distribution p ✓ by learning the transition distribution of the reverse process at each t, p ✓ ( x ( t  1)
| x ( t ) ) .
We follow the conventions set by Ho et al. (2020) in our parameterization and choice of objective.
In particular, we take p ✓ ( x ( t  1) | x ( t ) ) =
N ( x ( t  1) ; µ ✓ ( x ( t ) , t ) ,  ( t ) I ) with µ ✓ ( x ( t ) , t ) = 1 p ↵ ( t )
✓ x ( t )   ( t ) p 1  ¯ ↵ ( t ) ✏ ✓ ( x ( t ) , t ) ◆ , ↵ ( t ) :
= 1   ( t ) , and ¯ ↵ ( t ) :=
Q ts =1 ↵ ( t ) .
We implement ✏ ✓ ( x ( t ) , t ) as a neural network.
For training, we marginally sample x
( t ) ⇠ q ( x ( t ) | x (0) ) from the forward process as x ( t )
=
p ¯ ↵ ( t ) x (0) + p 1  ¯ ↵ ( t ) ✏ and minimize the objective T  P
[Equation]
⇤ by stochastic optimization (Ho et al., 2020, Algorithm 1).
To generate samples from p ✓ ( x (0) ) , we simulate the reverse process.
That is, we sample noise for time T as x ( T ) ⇠ N (0 , I ) , and then for each t =
T  1 , . .
. , 0 , we simulate progressively “de-noised” samples as x ( t )
⇠ p ✓ ( x ( t ) | x ( t +1) ) .
P ROT D IFF : A DIFFUSION MODEL OF PROTEIN BACKBONES IN 3D
Implementation of diffusion probabilistic models requires choosing an architecture for the neural network ✏ ✓ ( x ( t ) , t ) introduced abstractly in Section 2.2.
In this section we describe ProtDiff , which corresponds to the choice of ✏ ✓ ( x ( t ) , t ) as a translation and rotation equivariant graph neural network tailored to modeling protein backbones. We leave architectural and input encoding details to Appendix C.
The challenge of modeling points in 3D.
The properties and functions of proteins are dictated by the relative geometry of their residues, and are invariant to the coordinate system chosen to encode them.
Recent work on neural network modeling of 3D data has found, both theoretically and empirically, that neural networks constrained to satisfy geometric invariances can provide inductive biases that improve generalization and training efﬁciency (Batzner et al., 2022).
Motivated by this observation, we parameterize ✏ ✓ by an equivariant graph neural network (EGNN) (Satorras et al., 2021), which in 3D is equivariant to transformations in the Euclidean group.
Xu et al.
(2022) proved that if ✏ ✓ is equivariant to a group then p ✓ is invariant to the same group.
Tailoring EGNN to protein backbones.
We now describe our EGNN implementation, which we tailor to protein backbones and DPMs through the choice of edge and node features.
To model every pairwise residue interaction, we represent backbones by a fully connected graph.
Each node in the graph is indexed by n = 1 , . .
. , N, and corresponds to a residue.
We associate each node with coordinates x n 2 R 3 and D features h n 2 R D .
For each pair of nodes n, n 0 we deﬁne an edge and associate it with edge features.
We construct our EGNN by stacking L equivariant graph convolutional layers (EGCL) .
Each layer takes node coordinates and features as input, and outputs updated coordinates and features with the ﬁrst layer taking initial values ( x , h ) .
We write the output of EGNN after L layers as ˆ x = EGNN [ x , h ] .
In the context of diffusion models, we predict the noise at time t with the following parameterization:
[Equation]
We now describe our choice of node and edge features.
Our choice is motivated by the linear chain structure of protein backbones; residues close in sequence are necessarily close in 3D space.
To allow this chain constraint to be learned more easily, we ﬁx an ordering of nodes in the graph to correspond to sequence order.
We include as edge features positional offsets as done in Ingraham et al. (2019), which we represent using sinusoidal positional encoding features (Vaswani et al., 2017).
For node features, we similarly use a sinusoidal encoding of sequence position as well as of the diffusion time step t following Kingma et al.
We additionally process the time encoding to be orthogonal to the positional encoding .
SMCD IFF : C ONDITIONAL SAMPLING IN DIFFUSION MODELS BY PARTICLE FILTERING
The second stage of our generative modeling approach to the motif-scaffolding problem is to sample scaffolds x (0)
S from p ✓ ( x (0) S | x (0) M ) .
Section 4.1 discusses the intractability of sampling from p ✓ ( x (0)
S | x (0) M ) exactly and the limitations of a simple approximation introduced by (Song et al., 2021).
In Section 4.2, we then frame computation of p ✓ ( x (0) S | x (0) M ) as a sequential Monte Carlo (SMC) problem (Doucet et al., 2001) and approximate it with a particle ﬁltering algorithm (Algorithm 1).
4. T HE CHALLENGE OF CONDITIONAL SAMPLING AND THE ERROR OF THE REPLACEMENT METHOD
The conditional distributions of a DPM are deﬁned implicitly through the steps of the reverse process.
We may write the conditional density explicitly as
[Equation]
However, the high-dimensional integral on the right-hand side above is intractable (both analytically and numerically) to compute.
[˘ x 12: 8 k w
[Equation]
) k p [Equation]
˘
[Equation] t  1)
M
[Equation] t
[Equation]
k ) 13: 8 k, ˜ w
[Equation] t ) k w
[-M | x ( t ) k ) 13: 8 k, ˜ w ( t ) k w ( t ) k / P Kk 
-][Equation] [Equation]
w [Equation] t
[Equation] 1: K ( ˜ w ( t ) 1: K , x ( t ) 1: K ) 15:
✓ ( x ( t  1) | ˜ x ( t ) k ) 18: end for 19: Return x (0)
S , 1: K
To overcome this intractability, we build on the work of Song et al.
(2021), who introduced a practical algorithm that generates approximate conditional samples.
This strategy is to (1) forward diffuse the conditioning variable to obtain x (1: T )
M ⇠ q ( x (1: T )
M | x (0) M ) , and then (2) for each t, sample x ( t )
[Equation] [Equation] t [Equation]
S ) .
We call this approach the replacement method (following Ho et al. and make it explicit in Appendix Algorithm 2.
However, in Proposition D.1 we show that the replacement method introduces irreducible approximation error that cannot be eliminated by making p ✓ more expressive.
Additionally, although this approximation error is not analytically tractable in general, we show in Corollary D.2 the dependence of this error on the covariance of x (0) M and x (0) S in the case that q ( x (0)
M , x (0) S ) is bivariate Gaussian.
4. C ONDITIONAL SAMPLING IS A SEQUENTIAL M ONTE C ARLO PROBLEM
We next frame approximation of q ( x (0) S | x (0) M ) as a sequential Monte Carlo problem that we may solve by particle ﬁltering.
Intuitively, particle ﬁltering addresses a limitation of the replacement method: the failure at each time t to look beyond the current step to the less-noised motif x ( t  1) M when sampling x ( t )
S ⇠ p ✓ ( x ( t ) S | x ( t +1) ) .
Our key insight is that because p ✓ ( x ( t  1) M | x ( t ) ) provides a mechanism to assess the likelihood of x ( t  1) M , we can prioritize noised scaffolds that are more consistent with the motif.
Particle ﬁltering leverages this mechanism to provide a sequence of discrete approximations to each p ✓ ( x ( t ) S | x ( t  1: T ) M ) that look ahead by this extra step.
Finally, at t =
we have an approximation to p ✓ ( x (0) S | x (0: T ) M ) .
Then, using Proposition 4.1 below, we can obtain an approximate sample from q ( x (0) S | x (0) M ) .
This framing permits the application of standard particle ﬁltering algorithms (Doucet et al., 2001).
Algorithmsummarizes an implementation of this procedure that uses residual resampling (Doucet & Johansen, 2009) to mitigate the collapse of the sequential approximations into point masses.
SMCDiff provides a tunable trade-off between computational cost and statistical accuracy through the choice of the number of particles K .
In our next proposition we make this trade-off explicit.
Proposition 4.1.
Suppose that p ✓ exactly matches the forward diffusion process such that for every x ( t +1) , p ✓ ( x ( t ) | x ( t +1) )
=
q ( x ( t ) | x ( t +1) ) and consider any motif x (0) M .
Let x S ,K be a particle chosen at random from the output of Algorithm 1 with K particles.
Then x S ,K converges in distribution to q ( x (0)
S | x (0) M ) as K goes to inﬁnity.
The signiﬁcance of Proposition 4.1 is that it guarantees Algorithm 1 can provide arbitrarily accurate conditional samples provided an accurate diffusion model and large enough compute budget (determined by the number of particles).
To our knowledge, SMCDiff is the ﬁrst algorithm for asymptotically exact conditionally sampling from unconditional DPMs.
Our proof of the proposition, which we leave to Appendix D, is obtained from an application of standard asymptotics for particle ﬁltering (Chopin & Papaspiliopoulos, 2020, Proposition 11.4).
E XPERIMENTS
We empirically demonstrate the ability of our method to scaffold motifs and sample protein backbone structures.
We describe our procedure for evaluating backbone designs in Section 5.1.
We demonstrate the promise of our method for the motif-scaffolding problem in Section 5.2.
And we investigate our method’s strengths and weaknesses via experiments in unconditional sampling in Section 5.3.
We train a single instance of ProtDiff and use it across all of our experiments.
For simplicity, we limited our training data to single chain proteins taken from PDB that are no longer thanresidues.
See Appendix F for training details.
Code for ProtDiff and SMCDiff is included in the supplementary material and will be made available through github pending publication.
Baselines.
As mentioned in Section 1.1, Wang et al.
(2022) is the only prior machine learning work to address the motif-scaffolding problem.
We do not compare against this as a baseline because no stable implementation was available at the time of writing.
The most closely related method for unconditional sampling with available software is trDesign (Anishchenko et al., 2021), but this method does not allow speciﬁcation of a motif.
The ML method most similar to ProtDiff is the concurrently developed equivariant diffusion model (EDM) proposed by Hoogeboom et al.
Like ProtDiff , EDM uses a densely connected EGNN architecture but without sequence-distance edge features.
Consequently, it does not impose any sequence order, and therefore does not yield a way to relate generated coordinates to a backbone chain.
5. In silico EVALUATION OF DESIGNED BACKBONES
While experimental validation via X-ray crystallography remains the gold standard for evaluating computationally designed proteins, recent work (Wang et al., 2022; Lin et al., 2021) has proposed to leverage highly accurate protein structure prediction neural networks as an in silico proxy for true structure.
More speciﬁcally, Wang et al.
jointly design protein sequence and structure, and validate by comparing the design and AlphaFold2 (AF2) (Jumper et al., 2021) predicted structures.
Here, our goal is to assess the quality of scaffolds generated independent of a speciﬁc sequence, so we treat ﬁxed backbone sequence design as a downstream step as in Lin et al.
Our evaluation with AF2 is as follows.
For each generated scaffold we use a C- ↵ only version of
ProteinMPNN (Dauparas et al., 2022) with a temperature of 0.1 to sample 8 amino acid sequences likely to fold to the same backbone structure.
We then run AF2 with the released CASP14 1 weights and 15 recycling iterations.
We do not include a multiple sequence alignment as an input to AF2.
Our choice of utilizing ProteinMPNN and AF2 (without MSAs) is motivated by their empirical success in various de novo protein design tasks and the ability to recapitulate native proteins (Dauparas et al., 2022; Bennett et al., 2022).
To assess unconditionally sampled scaffolds, we then evaluate the agreement of our backbone sample with the AF2 predicted structures using the maximum TMscore (Zhang & Skolnick, 2005) across all generated sequences which we refer to as scTM , for self-consistency TM-score.
To assess whether prospective scaffolds generated support a motif, we compute the root mean squared distances (RMSD) of the desired and predicted motif coordinates after alignment and refer this metric as the motif RMSD .
Appendix Algorithm 4 outlines the exact steps.
Because a TM-score > 0.5 indicates that two structures have the same fold (Zhang & Skolnick, 2005), we say that a backbone is designable if scTM > 0.5.
The ability for AF2 to reproduce the same backbone from an independently designed sequence is evidence a sequence can be found for the starting structure.
To verify this is a reasonable cutoff, we analyzed scTM over our training set and found 87% to be designable
5. M OTIF - SCAFFOLDING VIA CONDITIONAL SAMPLING
We evaluated our motif-scaffolding approach (combining SMCDiff and ProtDiff ) on motifs extracted from existing proteins in the PDB and found that our approach can generate long and diverse scaffolds that support these motifs.
We chose to ﬁrst evaluate on motifs extracted from proteins present in the training set because we knew that at least one stabilizing scaffold exists.
We considered 2 examples taken from the PDB with IDs 6exz and 5trv, which are 69 and 118 residues long, respectively.
We chose these examples due to their high secondary structure composition while being representative of the shortest and longest lengths seen during training.
For each structure, we chose a 15–25 residue helical segment as the motif (see Appendix H for details).
The remainder of each protein is one possible supporting scaffold.
We sought to assess if we could recover this and other scaffolds with the same size and motif placement.
Based on prior work (Wang et al., 2022) we expected that building larger scaffolds around a motif would be more challenging than building smaller scaffolds.
To assess this length dependence we expanded the segment of used as the motif when running SMCDiff by including additional residues on each side.
In each case, though, we compute the motif RMSD over the minimal motif.
In
Figure 2B we present motif-scaffolding performance and its dependence on scaffold size for 5trv, the longer of the two test proteins.
For the 5trv test case, the lower quartile of the motif RMSD for SMCDiff is below 1Å for scaffolds up to 80 residues.
Since 1Å is atomic-level resolution, we conclude that our approach can succeed in this length range.
Figure 2A provides a visualization of our method’s capacity to generate long and diverse scaffolds.
The ﬁgure depicts two dissimilar scaffolds of lengths 34 and 54 produced by SMCDiff withparticles.
Both scaffolds are designable and agree with AF2 ( scTM > 0.5).
Diversity is particularly evident in the different orderings of secondary structures.
Figure 2B compares SMCDiff to two naive inpainting methods, fixed and replacement .
In fixed , the motif is ﬁxed for every timestep t, and the reverse diffusion is applied only to the scaffold (as done by Zhou et al. ; replacement is the method described in Section 4.
In contrast to
SMCDiff , these baselines fail to generate a successful scaffolds longer than 50 residues on 5trv, as determined by the location of their lower quartiles.
We next applied these three inpainting methods to harder targets in order to measure generalization to out-of-distribution and more difﬁcult motifs comprising of dis-contiguous regions and loops.
We consider a motif obtained from the respiratory syncytial virus (RSV) protein and calcium binding EF-hand motif, both of which are not in the training dataset.
RSV is known to be difﬁcult due to its composition of helical, loop, and sheet segments, while EF-hand is a dis-contiguous loop motif found in a calcium binding protein.
More details about both motifs can be found in Wang et al. ; there the authors report the only known successful scaffold of these motifs but they attain it with a computationally intensive hallucination approach.
We found that our method failed to generate scaffolds predicted to recapitulate the motif (Appendix H); however, SMCDiff provided smaller median motif RMSDs than the other two inpainting methods.
Compute cost.
The computation of SMCDiff with 64 particles is approximately 2 minutes per independent sample, while alternative methods fixed and replacement can produce 64 independent samples in the same time.
By contrast, the hallucination approach of Wang et al. (2022) involves running a Markov chain for thousands of steps, and has runtime on the order of hours for a single sample (Anishchenko et al., 2021).
5. U NCONDITIONAL SAMPLING
We next investigate the origins of the diversity seen in Figure 2 by analyzing the diversity and designability of ProtDiff samples without conditioning on a motif.
We ﬁrst check that ProtDiff produces designable backbones.
To do this, we generated 10 backbone samples for each length between 50 and 128 and then calculated scTM for each sample.
In Fig.
3A, we ﬁnd that 11.8% of samples have scTM > 0.5.
However, the majority of backbones do not pass this threshold.
We also observe designability has strong dependence on length since we expect that longer proteins are harder to model in 3D and design sequences for.
We separated the lengths below 128 residues into two categories and refer to them as short (50–70) and long (70–128).
Our results in
Figure 3A indicate 17% of designs in the short category are designable vs. 9% in the long category.
In Figure 13, we present a structural clustering of these designable backbones; we ﬁnd that these backbones exhibit diverse topologes.
We next sought to evaluate the ability of ProtDiff to generalize beyond the training set and produce novel backbones.
In Figure 3B each point represents a backbone sample from ProtDiff .
The horizontal coordinate of a point is the scTM , and the vertical coordinate is the minimum TM-score across the training set.
We found a strong positive correlation between scTM and this minimum
TM-score, indicating that many of the most designable backbones generated by ProtDiff were a result of training set memorization.
However, if the model were only memorizing the training set, we would see TM-scores consistently near 1.0; the range of scores in Figure 3B indicate this is not the case – and the model is introducing a degree of variability.
Figure 3C gives an example of backbone with scTM > 0.5 that appears to be novel.
Its closest match in the PDB has TM-score = 0.54.
Fig.
3B illustrates a limitation of our method: many of our sampled backbones are not designable.
One contributing factor is that ProtDiff does not handle chirality.
Hence ProtDiff generates backbones with the wrong handedness, which cannot be realized by any sequence.
Fig.
3B shows that 45% of all backbone samples had at least one incorrect, left-handed helix.
Of these, most have scTM < 0.5.
We describe calculating left-handed helices in Appendix G.
Fig.
4 illustrates an interpolation between two samples, showing how ProtDiff ’s outputs change as a function of the noise used to generate them.
To generate these interpolations, we pick two backbone samples that result in different folds.
For independent samples generated with noise ✏ (0: T ) and ˜ ✏ (0: T ) we interpolate with noise set to p ↵✏ (0: T ) + p 1  ↵ ˜ ✏ (0: T ) for ↵ between 0 and 1.
The depicted values of ↵ are chosen to highlight transition points with full interpolations included in Appendix H.3.
A future direction is to exploit the latent structure of ProtDiff to control backbone topology.
D ISCUSSION
The motif-scaffolding problem has applications ranging from medicine to material science (King et al., 2012), but remains unsolved for many functional motifs.
We have created the ﬁrst generative modeling approach to motif-scaffolding by developing ProtDiff , a diffusion probabilistic model of protein backbones, and SMCDiff , a procedure for generating scaffolds conditioned on a motif.
Although our experiments were limited to a small set of proteins, our results demonstrate that our procedure is the ﬁrst capable of generating diverse scaffolds longer than 20 residues with computation time reliably on the order of minutes or less.
Our work demonstrates the potential of machine learning methods to be applied in realistic protein design settings.
General conditional sampling .
SMCDiff is applicable to generic DPMs and is not limited to only proteins and motif-scaffolding.
While we do not make claims of SMCDiff outperforming state-of-the-art conditional diffusion models on other tasks such as image generation, we demonstrate a clear advantage of SMCDiff over the replacement method on a toy task of inpainting MNIST images in Appendix I. Extending SMCDiff outside of motif-scaffolding is outside the scope of the present work, but the advantages of a single model for both unconditional and conditional generation warrants additional research.
Modeling limitations .
Our present results do not indicate our procedure can generalize to motifs that are not present in the training set.
We believe improvements in protein modeling could provide better inductive biases for generalization.
ProtDiff , based on EGNN, is reﬂection equivariant since it only sees pairwise distances between 3D C- ↵ coordinates.
Additionally, ProtDiff does not explicitly model primary sequence or side-chains.
Hoogeboom et al.
(2022) demonstrate the beneﬁts of modeling sequence information in small molecules; joint modeling sequence and structure in a single model could improve the designability of protein scaffolds and backbones as well.
Data limitations .
We remarked our training set is small due to ﬁltering based on length and oligometry (using only monomeric proteins).
Scaling up to longer proteins opens up thousands more examples from the PDB, but in preliminary experiments has proven challenging.
Lastly, further development and comparison of methods for motif scaffolding will beneﬁt from standard evaluation benchmarks.
Developing a benchmark proved to be difﬁcult since motifs are not labeled in protein databases.
It will be important to gather motifs of biological importance in order to guide ML method development towards real-world applications.
Because no such benchmarks exist, developing them is a valuable direction for future work.