D IFFUSION PROBABILISTIC MODELING OF PROTEIN BACKBONES IN 3D FOR THE MOTIF - SCAFFOLDING PROBLEM
Anonymous authors
Paper under double-blind review
A BSTRACT
Construction of a scaffold structure that supports a desired motif , conferring protein function, shows promise for the design of vaccines and enzymes.
But a general solution to this motif-scaffolding problem remains open.
Current machine-learning techniques for scaffold design are either limited to unrealistically small scaffolds (up to length 20) or struggle to produce multiple diverse scaffolds.
We propose to learn a distribution over diverse and longer protein backbone structures via an E(3)- equivariant graph neural network.
We develop SMCDiff to efﬁciently sample scaffolds from this distribution conditioned on a given motif; our algorithm is the ﬁrst to theoretically guarantee conditional samples from a diffusion model in the largecompute limit.
We evaluate our designed backbones by how well they align with
AlphaFold2-predicted structures.
We show that our method can (1) sample scaffolds up to 80 residues and (2) achieve structurally diverse scaffolds for a ﬁxed motif
I NTRODUCTION
A central task in protein design is creation of a stable scaffold to support a target motif – that is, a particular structural protein fragment conferring biological function.
Vaccines and enzymes have already been designed by solving certain instances of this motif-scaffolding problem (Procko et al., 2014; Correia et al., 2014; Jiang et al., 2008; Siegel et al., 2010).
However, successful solutions to this problem in the past have necessitated substantial expert involvement and laborious trial and error.
Machine learning (ML) offers the hope to automate, and better direct this search.
But existing ML approaches face one of two major roadblocks.
First, these methods do not build scaffolds longer than about 20 residues; for many motif sizes of interest, the resulting proteins would be smaller than the shortest commonly-studied simple protein folds (35–40 residues) (Gelman & Gruebele, 2014).
Second, these methods require hours of computation to generate a single plausible scaffold (Wang et al., 2021; Anishchenko et al., 2021; Tischer et al., 2020).
Moreover, when a plausible scaffold is found, it remains to be experimentally validated.
Therefore, it is desirable to return not just a single scaffold but rather a set of scaffolds exhibiting diverse sequences and structural variation to increase the likelihood of success in practice.
In the present work, we demonstrate the promise of a particular generative modeling approach within
ML for efﬁciently returning a diverse set of motif-supporting scaffolds.
Generative models have been shown to capture a distribution over diverse protein structures (Lin et al., 2021).
But it is not clear how to handle conditioning (on the motif) using these approaches.
Diffusion probabilistic models (DPMs) offer a potential alternative; not only do they provide a more straightforward path to handling conditioning, but they have also enjoyed success generating small-molecules in 3D
Hoogeboom et al.
Extending DPMs to protein structures, though, is non-trivial; since proteins are larger than small molecules, modeling proteins requires handling the sequential ordering of residues and long-range interactions.
Finally, while existing models often generate distance matrices (Anand & Huang, 2018; Lin et al., 2021), we instead focus on generating a full set of 3D coordinates, which should improve designability in practice.
Our resulting model, ProtDiff , is similar to concurrent work on E(3)-equivariant diffusion models for molecules (Hoogeboom et al., 2022), but with modiﬁcations speciﬁc to protein structure.
Moreover, we develop a novel motif-scaffolding procedure based on Sequential Monte Carlo, SMCDiff , that repurposes an unconditionally trained
DPM for conditional sampling.
In our case, we condition on the motif structure, a task analogous to inpainting (Saharia et al., 2021).
We prove SMCDiff is guaranteed to provide exact conditional samples in the large-compute limit, in contrast to previous methods (Song et al., 2021;
Zhou et al., 2021), which we show introduce non-trivial approximation error that impedes performance.
Our ﬁnal motif-scaffolding generative framework, then, has two steps (Fig. 1): ﬁrst we train ProtDiff to learn a distribution over protein backbones, and then we use SMCDiff with ProtDiff to inpaint arbitrary motifs.
Ours is the ﬁrst machine-learning method to construct scaffolds longer than 20 residues around motifs — we build up to 80 residues scaffolds on a test case.
Beyond our progress on the motif-scaffolding problem, we provide the following technical contributions: (1) we introduce a protein-backbone generative model in 3D – with the ability to generate backbone samples that structurally agree with
AlphaFold2 Jumper et al.
(2021) predictions, and (2) we develop a novel conditional sampling algorithm for inpainting.
1. R ELATED WORK
Motif-scaffolding .
Past approaches have sought to scaffold a motif with native or prespeciﬁed protein fragments, but are limited to ﬁnding a suitable match in the Protein Data Bank (PDB) and cannot adapt the scaffold to compensate for slight structural mismatches (Cao et al., 2022; Silva et al., 2016;
Yang et al., 2021; Sesterhenn et al., 2020; Linsky et al., 2020).
More recently Wang et al.
(2021) used pre-trained protein structure prediction networks to recapitulate native scaffolds, but failed to generate scaffolds longer than 20 residues and can output only a single candidate scaffold rather than a diverse set.
By contrast, our goal is to sample diverse, long scaffolds.
Diffusion models for molecule generation .
Several concurrent works have extend equivariant diffusion models to molecule generation.
Anand & Achim (2022) extended diffusion models for generation of protein backbone frames and sequences conditioned on secondary structure adjacency matrices.
Similarly, Luo et al.
(2022) focused on CDR-loop generation using diffusion models conditioned on non-CDR regions of the antibody-antigen.
Our method does not require conditioning and is applicable to general proteins.
Lee & Kim (2022) approach the same problem as our work but build diffusion models over 2D distances matrices that requires post-processing to produce 3D structures through Rosetta minimization.
We demonstrate capability of diffusion models to directly model 3D coordinates of proteins.
Hoogeboom et al.
(2022) developed an equivariant diffusion model (EDM) for generating small molecules in 3D. However, because EDM does not enforce a spatial ordering of the atoms that compose small molecules, as we describe in Section 5, it does not learn a coherent chain structure as needed in proteins.
Inpainting and conditional sampling in diffusion models .
Point-Voxel Diffusion (PVD) Zhou et al.
(2021) is a 3D diffusion model for generating shapes from the ShapeNet dataset.
Though trained to generate shapes unconditionally, PVD completes (or inpaints ) full shapes when a partial point cloud is ﬁxed during inference.
For general diffusion models, Song et al.
(2021) proposed an alternative inpainting approach and remarked that this approach produces approximate conditional samples.
However, these methods do not provide theoretical guarantees, and when we compare them to SMCDiff , we ﬁnd that their approximation error impedes performance when applied to motifscaffolding.
Saharia et al.
(2021) developed an inpainting diffusion model by training a diffusion model to denoise randomly generated masked regions while unmasked regions were unperturbed.
However, their approach requires a detailed data augmentation strategy that does not exist for proteins.
We describe additional related work on protein generative models in Appendix A.
P RELIMINARIES 2. T HE MOTIF - SCAFFOLDING PROBLEM
A protein can be represented by its amino acid sequence and backbone structure.
Let A be the set of 20 genetically-encoded amino acids.
We denote the sequence of an N -residue protein by s 2 A N and its C- ↵ backbone coordinates in 3D by x = [ x 1 , . . .
, x N ] T 2 R N, 3 .
A protein’s structure is a function of its sequence, so we may write x ( s ) .
We divide the N residues into the functional motif
M and the scaffold S , such that M [ S = { 1 , 2 , . . .
, N } .
The goal is to identify, given the motif structure x
M , sequences s whose structure recapitulates the motif to high precision x ( s ) M ⇡ x M .
Appendix B discusses several caveats of this simpliﬁed framing.
2. D IFFUSION PROBABILISTIC MODELS
Our approach to the motif-scaffolding problem builds on denoising diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015).
We follow the conventions and notation set by (Ho et al., 2020), which we review here.
DPMs are a class of generative models based on a reversible, discrete-time diffusion process.
The forward process starts with a sample x (0) from an unknown data distribution q , with density denoted by q ( x (0) ) , and iteratively adds noise at each step t .
By the last step, T , the distribution of x ( T ) is indistinguishable from an isotropic Gaussian: x ( T ) ⇠ N ( x ( T ) ; 0 , I ) .
Speciﬁcally, we choose a variance schedule  (1) ,  (2) , . . .
,  ( T ) , and deﬁne the transition distribution at step t as q ( x ( t ) | x ( t  1) )
=
N ( x ( t ) ;
[Equation]
DPMs approximate q with a second distribution p ✓ by learning the transition distribution of the reverse process at each t, p ✓ ( x ( t  1)
| x ( t ) ) .
We follow the conventions set by (Ho et al., 2020) in our parameterization and choice of objective.
In particular, we take p ✓ ( x ( t  1) | x ( t ) ) =
N ( x ( t  1) ; µ ✓ ( x ( t ) , t ) ,  ( t ) I ) with µ ✓ ( x ( t ) , t ) =
[Equation]
t )
[Equation]
x
[Equation]
t
[Equation]
( t
[Equation]
p
[Equation]
x
[Equation] [Equation]
) ◆ , ↵ ( t ) :
= 1   ( t ) , and ¯ ↵ ( t ) :=
Q t s =1 ↵ ( t ) .
We implement ✏ ✓ ( x ( t ) , t ) as a neural network.
For training, we marginally sample x
( t ) ⇠ q ( x ( t ) | x (0) ) from the forward process as x ( t )
=
p ¯ ↵ ( t ) x (0) + p 1  ¯ ↵ ( t ) ✏ and minimize the objective T
[Equation]
⇤ by stochastic optimization (Ho et al., 2020, Algorithm 1).
To generate samples from p ✓ ( x (0) ) , we simulate the reverse process.
That is, we sample noise for time T as x ( T ) ⇠ N (0 , I ) , and then for each t =
T  1 , . .
. , 0 , we simulate progressively “de-noised” samples as x ( t )
⇠ p ✓ ( x ( t ) | x ( t +1) ) .
P ROT D IFF : A DIFFUSION MODEL OF PROTEIN BACKBONES IN 3D
Implementation of diffusion probabilistic models requires choosing an architecture for the neural network ✏ ✓ ( x ( t ) , t ) introduced abstractly in Section 2.2.
In this section we describe ProtDiff , which corresponds to the choice of ✏ ✓ ( x ( t ) , t ) as a translation and rotation equivariant graph neural network tailored to modeling protein backbones.
The challenge of modeling points 3D.
The properties and functions of proteins are dictated by the relative geometry of their residues, and are invariant to the coordinate system chosen to encode them.
Recent work on neural network modeling of 3D data has found, both theoretically and empirically, that neural networks constrained to satisfy geometric invariances can provide inductive biases that improve generalization and training efﬁciency Batzner et al.
Motivated by this observation, we parameterize ✏ ✓ by an equivariant graph neural network (EGNN) (Satorras et al., 2021), which in 3D is equivariant to transformations in the Euclidean group.
Xu et al.
(2022) proved that if ✏ ✓ is equivariant to a group then p ✓ is invariant to the same group.
Tailoring EGNN to protein backbones.
We now describe our EGNN implementation, which we tailor to protein backbones and DPMs through the choice of edge and node features.
To model every pairwise residue interaction, we represent backbones by a fully connected graph.
Each node in the graph is indexed by n = 1 , . .
. , N, and corresponds to a residue.
We associate each node with coordinates x n 2 R 3 and D features h n 2 R D .
For each pair of nodes n, n 0 we deﬁne an edge and associate it with edge features a nn 0 2 R D .
We construct our EGNN by stacking L equivariant graph convolutional layers (EGCL).
Each layer takes node coordinates and features as input, and outputs updated coordinates and features.
We take the input to the ﬁrst layer to be the initial coordinates and features ( x 0 , h 0 ) =
( x , h ) .
Each layer l = 1 , . . .
, L deﬁnes an update as ( x l , h l ) =
EGCL [ x l  1 , h l  1 ]
where for each node n x ln = x l  1 n
[Equation]
x ( h l  1 n , h l  1 n 0 , d nn 0 , a nn 0 ) and
h ln =
h
( h l  1 n , m n ) , for
[Equation]
nn
[Equation]
=
[Equation]
l
[Equation]
n
[Equation]
l
[Equation]
[Equation]
, h
[Equation]
[Equation]
n 0 , d nn 0 , a nn 0 ) , and d nn 0 = k x l  1 n  x l  1 n 0 k 22 .
e ,  h , and  x are fully connected neural networks, and  is a small positive constant included for numerical stability.
We write the output of EGNN after L layers as ˆ x = EGNN [ x 0 , h 0 ] .
In the context of diffusion models, we predict the noise with the following parameterization:
[Equation]
We perform a scaling operation that transforms the input protein coordinates to be in nanometers rather than Angstroms by dividing by 10.
Because typical proteins are on the order of 2–10 nanometers in diameter, this scaling brings the backbones to a spatial scale more similar to an isotropic Gaussian with unit variance.
We now describe our choice of node and edge features.
Our choice is motivated by the linear chain structure of protein backbones; residues close in sequence are necessarily close in 3D space.
To allow this chain constraint to be learned more easily, we ﬁx an ordering of nodes in the graph to correspond to sequence order.
We include as edge features positional offsets, which we represent using sinusoidal positional encoding features (Vaswani et al., 2017) as
[Equation]
( n  n 0 , 1) .
' ( n  n 0 , D )  , where ' ( x, k ) =
⇢ sin  x · ⇡ /N 2 · k/D  , k mod 2 =cos  x · ⇡ /N 2 · ( k  1) /D
, k mod 2 = 1 .
For node features, we similarly use a sinusoidal encoding of sequence position as well as of the diffusion time step t (following (Kingma et al., 2021)) as
[Equation] [Equation]
, where R is a D ⇥ D orthogonal matrix chosen uniformly at random.
Intuitively, applying R transforms the time encoding to be orthogonal to the positional encoding.
SMCD IFF : C ONDITIONAL SAMPLING IN DIFFUSION MODELS BY PARTICLE FILTERING
The second stage of our generative modeling approach to the motif-scaffolding problem is to sample scaffolds x (0)
S from p ✓ ( x (0) S | x (0) M ) .
Section 4.1 discusses the intractability of sampling from p ✓ ( x (0)
S | x (0) M ) exactly and the limitations of a simple approximation introduced by (Song et al., 2021).
In Section 4.2, we then frame computation of p ✓ ( x (0) S | x (0) M ) as a sequential Monte Carlo (SMC) problem Doucet et al.
and approximate it with a particle ﬁltering algorithm (Algorithm 1).
4. T HE CHALLENGE OF CONDITIONAL SAMPLING AND THE ERROR OF THE REPLACEMENT METHOD
The conditional distributions of a DPM are deﬁned implicitly through the steps of the reverse process.
We may write the conditional density explicitly as
[Equation]
However, the high-dimensional integral on the right-hand side above is intractable (both analytically and numerically) to compute.
1: Input: x (0) M (motif)
[Equation]
K (# particles) 2: //
Forward diffuse motif 3: ˘ x (1: T )
M ⇠ q ( x (1: T )
M | x (0) M ) 4: 5: // Reverse diffuse particles 6: 8 k, x ( T ) k i.i.d.
⇠ p ✓ ( x ( T ) ) 7: for t =
T, . .
, 1 do 8: //
Replace motif 9: 8 k, x ( t ) k
[˘ x ( t )
M , x ( t ) S ,k ] 10: 11: // Re-weight based on ˘ x ( t  1)
M 12: 8 k, w ( t )
k p ✓ (˘ x ( t  1)
M | x ( t ) k ) 13: 8 k, ˜ w ( t ) k w ( t ) k /
P
Kk
=1 w ( t ) k14:
˜ x ( t ) 1: K ⇠ Resample ( ˜ w ( t ) 1: K , x ( t ) 1: K ) 15: 16: //
Propose next step 17: 8 k, x ( t  1) k indep.
⇠ p ✓ ( x ( t  1) | ˜ x ( t ) k ) 18: end for 19: Return x (0)
S , 1: K
To overcome this intractability, we build on the work of Song et al.
(2021), who introduced a practical algorithm that generates approximate conditional samples.
This strategy is to (1) forward diffuse the conditioning variable to obtain x (1: T )
M ⇠ q ( x (1: T )
M | x (0) M ) , and then (2) for each t, sample x ( t )
S ⇠ p ✓ ( x ( t ) S | x ( t +1) M , x ( t +1)
S ) .
We call this approach the replacement method (following (Ho et al., 2022)) and make it explicit in Appendix Algorithm 2.
However, in Proposition C.1 we show that the replacement method introduces irreducible approximation error that cannot be eliminated by making p ✓ more expressive.
Additionally, although this approximation error is not analytically tractable in general, we show in Corollary C.2 the dependence of this error on the covariance of x (0) M and x (0) S in the case that q ( x (0)
M , x (0) S ) is bivariate Gaussian.
4. C ONDITIONAL SAMPLING
IS A
SEQUENTIAL M ONTE C ARLO PROBLEM
We next frame approximation of q ( x (0) S | x (0) M ) as a sequential Monte Carlo problem that we may solve by particle ﬁltering.
Intuitively, particle ﬁltering addresses a limitation of the replacement method: the failure at each time t to look beyond the current step to the less-noised motif x ( t  1) M when sampling x ( t )
S ⇠ p ✓ ( x ( t ) S | x ( t +1) ) .
Our key insight is that because p ✓ ( x ( t  1) M | x ( t ) ) provides a mechanism to assess the likelihood of x ( t  1) M , we can prioritize noised scaffolds that are more consistent with the motif.
Particle ﬁltering leverages this mechanism to provide a sequence of discrete approximations to each p ✓ ( x ( t ) S | x ( t  1: T ) M ) that look ahead by this extra step.
Finally, at t =
we have an approximation to p ✓ ( x (0) S | x (0: T ) M ) .
Then, using Proposition 4.1 below, we can obtain an approximate sample from q ( x (0) S | x (0) M ) .
This framing permits the application of standard particle ﬁltering algorithms (Doucet et al., 2001).
Algorithmsummarizes an implementation of this procedure that uses residual resampling (Doucet & Johansen,
2009) to mitigate the collapse of the sequential approximations into point masses.
SMCDiff provides a tunable trade-off between computational cost and statistical accuracy through the choice of the number of particles K .
In our next proposition we make this trade-off explicit.
Proposition 4.1.
Suppose that p ✓ exactly matches the forward diffusion process such that for every x ( t +1) , p ✓ ( x ( t ) | x ( t +1) )
=
q ( x ( t ) | x ( t +1) ) and consider any motif x (0) M .
Let x S ,K be a particle chosen at random from the output of Algorithm 1 with K particles.
Then x S ,K converges in distribution to q ( x (0)
S | x (0) M ) as K goes to inﬁnity.
The signiﬁcance of Proposition 4.1 is that it guarantees Algorithm 1 can provide arbitrarily accurate conditional samples provided a large enough compute budget (determined by the number of particles).
To our knowledge, SMCDiff is the ﬁrst algorithm for conditionally sampling from DPMs that can provide arbitrary accuracy.
Our proof of the proposition, which we leave to Appendix C, is obtained from an application of standard asymptotics for particle ﬁltering (Chopin & Papaspiliopoulos, 2020,
Proposition 11.4).
E XPERIMENTS
We empirically demonstrate the ability of our method to scaffold motifs and sample protein backbone structures.
We describe our procedure for evaluating backbone designs in Section 5.1.
We demonstrate the promise of our method for the motif-scaffolding problem in Section 5.2.
And we investigate our method’s strengths and weaknesses via experiments in unconditional sampling in Section 5.3.
We train a single instance of ProtDiff and use it across all of our experiments.
For simplicity, we limited our training data to single chain proteins taken from PDB that are no longer thanresidues.
See Appendix E for training details.
Code for ProtDiff and SMCDiff is included in the supplementary material and will be made available through github pending publication.
Baselines.
As mentioned in Section 1.1, Wang et al.
(2021) is the only prior machine learning work to address the motif-scaffolding problem.
We do not compare against this baseline because no stable implementation was available at the time of writing.
The most closely related method for unconditional sampling with available software is trDesign (Anishchenko et al., 2021), but this method does not allow speciﬁcation of a motif.
The ML method most similar to ProtDiff is the concurrently developed equivariant diffusion model (EDM) proposed by Hoogeboom et al.
Like ProtDiff , EDM uses a densely connected EGNN architecture but without sequence-distance edge features.
Consequently, it does not impose any sequence order, and therefore does not yield a way to relate generated coordinates to a backbone chain.
5. In silico EVALUATION OF DESIGNED BACKBONES
While experimental validation via X-ray crystallography remains the gold standard for evaluating computationally designed proteins, recent work (Wang et al., 2021; Lin et al., 2021) has proposed to leverage highly accurate protein structure prediction neural networks as an in silico proxy for true structure.
More speciﬁcally, Wang et al.
jointly design protein sequence and structure, and validate by comparing the design and AlphaFold2 (AF2) (Jumper et al., 2021) predicted structures.
Here, our goal is to assess the quality of scaffolds generated independent of a speciﬁc sequence, so we treat ﬁxed backbone sequence design as a downstream step as in (Lin et al., 2021).
Our evaluation with AF2 is as follows.
For each generated scaffold we use a C- ↵ only version of
ProteinMPNN (Dauparas et al., 2022) with a temperature of 0.1 to sample 8 amino acid sequences likely to fold to the same backbone structure.
We then run AF2 with the released CASP14 1 weights and 15 recycling iterations.
We do not include a multiple sequence alignment as an input to AF2.
To assess unconditionally sampled scaffolds, we then evaluate the agreement of our backbone sample with the AF2 predicted structures using the maximum TM-score (Zhang & Skolnick, 2005) across all generated sequences which we refer to as scTM , for self-consistency TM-score.
To assess whether prospective scaffolds generated support a motif, we compute the root mean squared distances (RMSD) of the desired and predicted motif coordinates after alignment and refer this metric as the motif RMSD .
Appendix Algorithm 4 outlines the exact steps.
Because a TM-score > 0.5 indicates that two structures have the same fold Zhang & Skolnick (2005), we say that a backbone is designable if scTM > 0.5.
The ability for AF2 to reproduce the same backbone from an independently designed sequence is evidence a sequence can be found for the starting structure.
5. M OTIF - SCAFFOLDING VIA CONDITIONAL SAMPLING
We evaluated our motif-scaffolding approach (combining SMCDiff and ProtDiff ) on motifs extracted from existing proteins in the PDB and found that our approach can generate long and diverse scaffolds that support these motifs.
We chose to ﬁrst evaluate on motifs extracted from proteins present in the training set because we knew that at least one stabilizing scaffold exists.
We considered 2 examples with different topologies taken from PDB with ID 6exz and 5trv, which are 69 andresidues long, respectively.
For each topology, we chose a 15–25 residue segment as the motif (see
Appendix G for details).
The remainder of each protein is one possible supporting scaffold.
We sought to assess if we could recover this and other scaffolds with the same size and motif placement.
Based on prior work (Wang et al., 2021), we expected that building larger scaffolds around a motif would be more challenging than building smaller scaffolds.
To assess this length dependence, we expanded the segment of used as the motif when running SMCDiff by including additional residues on each side.
In each case, though, we compute the motif RMSD over the minimal motif.
In
Figure 2B, we present motif-scaffolding performance and its dependence on scaffold size for 5trv, the longer of the two test proteins.
For the 5trv test case, the lower quartile of the motif RMSD for
SMCDiff is below 1 Å for scaffolds up to 80 residues.
Since 1 Å is atomic-level resolution, we conclude that our approach can succeed in this length range.
Figure 2A provides a visualization of our method’s capacity to generate long and diverse scaffolds.
The ﬁgure depicts two dissimilar scaffolds of lengths 34 and 54 produced by SMCDiff withparticles.
Both scaffolds are designable and agree with AF2 ( scTM > 0.5).
Diversity is particularly evident in the different orderings of secondary structures.
Figure 2B compares SMCDiff to two naive inpainting methods, fixed and replacement .
In fixed , the motif is ﬁxed for every timestep t, and the reverse diffusion is applied only to the scaffold (as in (Zhou et al., 2021)); replacement is the method described in Section 4.
In contrast to
SMCDiff , these baselines fail to generate a successful scaffolds longer than 50 residues on 5trv, as determined by the location of their lower quartiles.
We next applied these three inpainting methods to a harder target.
We consider a motif obtained from the respiratory syncytial virus (RSV) protein, which is not in the training dataset.
We found that our method failed to generate scaffolds predicted to recapitulate the motif (Appendix G).
This motif has been successfully scaffolded by the computationally intensive hallucination approach of (Wang et al., 2021).
Our results indicate that our method does not improve uniformly upon this previous approach.
Compute cost.
The computation of SMCDiff with 64 particles is approximately 2 minutes per independent sample, while alternative methods fixed and replacement can produce 64 independent samples in the same time.
By contrast, the hallucination approach of (Wang et al., 2021) involves running a Markov chain for thousands of steps, and has runtime on the order of hours for a single sample (Anishchenko et al., 2021).
5. U NCONDITIONAL SAMPLING
We next investigate the origins of the diversity seen in Figure 2 by analyzing the diversity and designability of ProtDiff samples without conditioning on a motif.
We ﬁrst check that ProtDiff produces designable backbones.
To do this, we generated 10 backbone samples for each length between 50 and 128 and then calculated scTM for each sample.
In Fig.
3A, we ﬁnd that 11.8% of samples have scTM > 0.5.
However, the majority of backbones do not pass this threshold.
We also observe designability has strong dependence on length since we expect that longer proteins are harder to model in 3D and design sequences for.
We separated the lengths below 128 residues into two categories and refer to them as short (50–70) and long (70–128).
Our results in
Figure 3A indicate 17% of designs in the short category are designable vs. 9% in the long category.
In Figure 12, we present a structural clustering of these designable backbones; we ﬁnd that these backbones exhibit diverse topologes.
We next sought to evaluate the ability of ProtDiff to generalize beyond the training set and produce novel backbones.
In Figure 3B each point represents a backbone sample from ProtDiff .
The horizontal coordinate of a point is the scTM , and the vertical coordinate is the minimum TM-score across the training set.
We found a strong positive correlation between scTM and this minimum
TM-score, indicating that many of the most designable backbones generated by ProtDiff were a result of training set memorization.
However, if the model were only memorizing the training set, we would see TM-scores consistently near 1.0; the range of scores in Figure 3B indicate this is not the case – and the model is introducing a degree of variability.
Figure 3C gives an example of backbone with scTM > 0.5 that appears to be novel.
Its closest match in the PDB has TM-score = 0.54.
Fig.
3B illustrates a limitation of our method: many of our sampled backbones are not designable.
One contributing factor is that ProtDiff does not handle chirality.
Hence ProtDiff generates backbones with the wrong handedness, which cannot be realized by any sequence.
Fig.
3B shows that 45% of all backbone samples had at least one incorrect, left-handed helix.
Of these, most have scTM < 0.5.
We describe calculating left-handed helices in Appendix F.
Fig.
4 illustrates an interpolation between two samples, showing how ProtDiff ’s outputs change as a function of the noise used to generate them.
To generate these interpolations, we pick two backbone samples that result in different folds.
For independent samples generated with noise ✏ (0: T ) and ˜ ✏ (0: T ) we interpolate with noise set to p ↵✏ (0: T ) + p 1  ↵ ˜ ✏ (0: T ) for ↵ between 0 and 1.
The depicted values of ↵ are chosen to highlight transition points with full interpolations included in Appendix G.3.
A future direction is to exploit the latent structure of ProtDiff to control backbone topology.
D ISCUSSION
The motif-scaffolding problem has applications ranging from medicine to material science (King et al., 2012), but remains unsolved for many functional motifs.
We have created the ﬁrst generative modeling approach to motif-scaffolding by developing ProtDiff , a diffusion probabilistic model of protein backbones, and SMCDiff , a procedure for generating scaffolds conditioned on a motif.
Although our experiments were limited to a small set of proteins, our results demonstrate that our procedure is the ﬁrst capable of generating diverse scaffolds longer than 20 residues with computation time reliably on the order of minutes or less.
Our work demonstrates the potential of machine learning methods to be applied in realistic protein design settings.
Modeling limitations .
Our present results do not indicate our procedure can generalize to motifs that are not present in the training set.
We believe improvements in protein modeling could provide better inductive biases for generalization.
For example, ProtDiff sees only 3D backbone coordinates and is blind to the the orientation of backbone residues As a consequence, ProtDiff is reﬂection symmetric and generates left-handed helices (Appendix D), which do not stably occur in natural proteins.
Additionally, ProtDiff does not explicitly model primary sequence or side-chains.
Hoogeboom et al.
(2022) demonstrate the beneﬁts of modeling sequence information in small molecules, jointly modeling sequence and structure in a single model could lead to improved designability of protein scaffolds and backbones as well.
Data limitations .
We remarked our training set is small due to ﬁltering based on length and oligometry (using only monomeric proteins).
Scaling up to longer proteins opens up thousands more examples from the PDB, but in preliminary experiments has proven challenging.
Lastly, further development and comparison of methods for motif scaffolding will beneﬁt from standard evaluation benchmarks.
Because no such benchmarks exist, developing them is a valuable direction for future work.