D EEP SHARPENING OF TOPOLOGICAL FEATURES FOR DE NOVO PROTEIN DESIGN
Bruno E
Correia bruno.correia epfl ch
A BSTRACT
Computational de novo protein design allows the exploration of uncharted areas of the protein structure and sequence spaces.
Classical approaches to de novo protein design involve an iterative process where the desired protein shape is outlined, then sampled for structural backbones and designed with low energy amino acid sequences.
Despite numerous successes, inaccuracies within energy functions and sampling methods often lead to physically unrealistic protein backbones yielding sequences that fail to fold experimentally.
Recently, deep neural networks have successfully been used to design novel protein folds from scratch by iteratively predicting a structure and optimizing the sequence until a target protein structure is reached.
These methods work well under circumstances where distributions of physically realistic target protein backbones can be readily defined, but lack the ability to de novo design loosely specified protein shapes.
In fact, a major challenge for de novo protein design is to generate “designable” protein structures for defined folds, including native and artificial (“dark matter”)
folds that can then be used to find low energetic sequences in a generic manner.
Here, we automate the task of creating designable backbones using a variational autoencoder framework, termed G ENESIS , to denoise sketches of protein topological lattice models by sharpening their 2D representations in distance and angle feature maps.
In conjunction with the trRosetta design framework, large pools of diverse sequences for different protein folds were generated for the maps.
We found that
the G ENESIS -
trDesign
framework generates native-like feature maps for known and dark matter
protein
folds
Ultimately, the G ENESIS framework addresses the protein backbone designability problem and could
contribute to
the de novo design of structurally defined
artificial
proteins
that
can be tailored for novel functionalities.
I NTRODUCTION
Evolution is a slow and gradual process that has only sampled a very small fraction of the possible protein amino acid (AA) sequence space (Grant et al., 2004; Kolodny et al., 2013).
In order to explore new sequences that fold into well-defined three-dimensional (3D) conformations outside the natural repertoire and are amenable to tailored functionalities, de novo protein design strategies have been developed (Huang et al., 2016a; Marcos & Silva, 2018).
Established de novo protein design methods employ an iterative process where (1) the protein shape is outlined and corresponding backbones are sampled, and (2) low energy AA sequences are fitted onto the generated backbones.
Despite numerous successes (Thomson et al., 2014; Huang et al., 2016b; Marcos et al., 2017; Dou et al., 2018), the stochastic nature of sampling methods and inaccuracies within current energy functions frequently drive the design simulations towards incorrect solutions, hence a significant number of design trajectories are needed (2,000–20,000 trajectories) in order to sufficiently probe the sequence and conformational landscape and to select potential low energy solutions (Rocklin et al., 2017; Chevalier et al., 2017; Bonet et al., 2018; Sesterhenn et al., 2020; Yang et al., 2021).
In fact, many design failures arise from frustrated backbones that are a priori non-“designable”.
Designable backbones have optimal secondary structure configurations with favored tertiary structure symmetries such that they are physically realizable with the 20 natural AAs (Li et al., 1996; England &
Shakhnovich, 2003;
Wingreen et al., 2004; Grigoryan & Degrado, 2011).
Quantifying designability is challenging as it includes properties that are difficult to measure, such as fold specificity (Govindarajan & Goldstein, 1996;
Wingreen et al., 2004), or native-like structural arrangements (Simons et al., 1997).
It has been observed that highly designable backbones can accommodate a large variety of energetically favorable sequences (Govindarajan & Goldstein, 1996;
Zhang et al., 2014; Helling et al., 2001).
To craft designable backbones, empirical rules have been derived from analysis of protein databases and simulations (Koga et al., 2012), and, together with small structural protein backbone fragment (3-mers and 9-mers) assembly protocols (Rohl et al., 2004), have led to the design of “ideal” protein folds.
These rules are based on loop lengths that embed the packing of local tertiary motifs such as β / β -, β / α -, and α / β -units to secondary structure elements (SSEs).
Since, the design rules have been steadily updated, for instance, loops can be structurally defined to bridge non-local motifs (Lin et al., 2015; Marcos et al., 2018), cavities can be created by inducing strong curvatures into β -strands through controlling resisters shifts and βbulges (Huang et al., 2016b; Marcos et al., 2017), and strategically placed residues relieving strain from the backbone allow the design of β -barrels (Dou et al., 2018; Vorobieva et al., 2021).
Recent advances within deep neural networks (DNNs) combined with the availability of large-scale protein structure data in the protein data bank (PDB) (Berman et al., 2000) have enabled highly accurate structure prediction from sequence (Yang et al., 2020;
Baek et al., 2021; Jumper et al., 2021)
(Fig.
1A).
Interestingly, the trained structure prediction DNNs can be “reversed” for the protein design task.
A good example is the transform-restrained Rosetta (trRosetta) neural network that was used to hallucinate novel proteins by using a specific loss that maximizes the contrast between random (background) and native distance predictions (Anishchenko et al., 2021).
TrRosetta can also be employed for fixed backbone design via backpropagating gradients from the target structure to the sequence, which has the effect of implicitly optimizing over the full sequence and structure landscape (Norn et al., 2021)
(Fig.
1B).
In the latter case, the method searches for the lowest-energy sequence while maximizing the probability of the target structure relative to all other conformations.
Encouragingly, the trRosetta design framework is able to design new sequences for a target structure within minutes on modern graphical processing units (GPUs), enabling multi-state and high-throughput sampling of the design space.
Inspired by these recent advances, this work puts forth the hypothesis that trRosetta can also facilitate tailored de novo design, where the shape and secondary structure element composition is controlled (Fig. 1C).
To achieve this, we implemented a framework that uses a simple string description of a protein fold (termed “Form”) and auto-generates realistic designed proteins.
The framework first creates a 3D representation of the Form termed “Sketch”.
We trained a variational autoencoder (VAE) termed G ENESIS to encode the inter-residue distances and orientations of a Sketch to a latent representation, sampled and then decoded close-native distance and orientation probabilities from this representation ready for the trRosetta sequence design task.
Our approach circumvents the need to create designable 3D backbones and is, unlike conventional de novo design methods, not directly based on energy functions.
This allows the de novo design process to be fast and efficient biasing the search towards productive sequence spaces.
P ROTEIN FOLD SKETCHING
We define the overall shape of a protein through a string specifying the SSE types, lengths, and relative positions on a lattice, termed “Form” (Taylor et al., 2008).
In a Form, each level or layer of the lattice can be populated by an arbitrary number of SSEs.
The layers are equally spaced from each other by 8 ˚A for β - β layers and 10–11 ˚A for α - α or helix- β layers (Chothia & Finkelstein, 1990).
A Form can be expanded into a 3D representation that we call a “Sketch”.
A Sketch is a rough 3D approximation of a native protein structure albeit lacking loops, native-like irregularities within SSEs, and AA side chains.
To address these limitations, we employ a DNN to automatically learn to decipher important structural features and incorporate native-like patterns into the Sketches.
The DNN takes the form of a
VAE that is trained to transform a large dataset of Sketches into their respective native structures.
The dataset was built by generating different sets of “Sketches” and mapping them to their native counterparts (Fig.
2A) (Appendix A.3).
The Sketches have small idealized SSEs (5 residues for β -strands and 9 residues for α -helices), no sequence information, and dummy backbone residues along the shortest path between end- and starting points of the SSEs representing the loops.
The loops were modeled in this way because we do not have information about their potential conformations.
The sets encompass many 2- and 3-layer fully β -, fully α - and mixed α / β topologies and capture a large scope of possible protein folds (Appendix A.1).
The Sketches do not resemble their native counterparts having a median template modeling (TM)-score of below 0.5 indicating that they are not classified as the same protein fold (Fig. 2B).
Importantly, although Sketches can fit onto multiple native structures the majority only map to 1 or 2 conformations (Fig.
2C).
Since not all protein domains can be formulated as a Form (e.g., β -barrels), we augment our data set by adding corrupted backbone structures, where the loops are replaced by dummy residues as done on the
Sketches (Sup.
Fig.
5A).
The corrupted backbones add architectural and structural diversity to our data set by retaining tertiary motif dispositions and secondary structure irregularities, respectively (Sup.
Fig.
5B).
We split our data set into a series of training and test sets with different structural properties based on the Structural Classification of Proteins — extended (SCOPe) scheme (Andreeva et al., 2020).
We then optimize G ENESIS for a training set consisting of a particular type of structure and test the validity of its predictions on proteins with increasingly different structural properties (Appendix A.2).
Our evaluation procedure quantitatively assesses the extent to which our framework generalizes beyond the distribution induced by a given training set.
We argue that any method that facilitates de novo design should generalize to unknown subsets (“dark matter”) of the protein space.
R ESULTS 3. N EURAL BACKBONE REFINEMENT
We developed a convolutional variational autoencoder (VAE) that operates on inter-residue distance and orientations rather than atomistic coordinates (Fig. 3A)
(see Appendix A.4).
Importantly, distances and orientations are invariant with respect to translation and rotation which ensures stable and predictable performance in the presence of transformations of the data input under the special Euclidean group.
Our VAE is conditioned on the real-valued distances and orientations of the
Sketches, and from the latent conditional distribution predicts distance- and orientation probabilities of native-like conformations.
We train the VAE in a supervised manner by minimizing the 1 st Wasserstein distance between the true feature maps and the distribution predicted by the VAE.
In contrast to the previously utilized cross-entropy loss (Senior et al., 2020; Yang et al., 2020), the Wasserstein distance enables weighting individual errors between the distributions, i.e., penalizing large differences between the true and predicted distributions more than small differences (see Appendix A.5 for details).
We follow a standard pre-train—fine-tune regimen (Fig.
3A): (1) We pre-train the VAE on the corrupted backbones with a learning rate set to 1e-3 over 300 epochs.
During this phase, the VAE is conditioned on real backbones with loops that have been corrupted with noise.
(2) We fine-tune the VAE forepochs on the Sketches.
The pre-training slightly improves the performance on the test set when compared to directly training the VAE on the Sketches (Sup.
Fig. 6).
We coupled the fine-tuned VAE (called G ENESIS ) with the trRosetta framework.
We use the trRosetta design method to optimize sequences for our generated distance- and orientation probabilities.
We subsequently used the generated sequences and constraints from trRosetta to minimize the energy with gradient descent and generate 3D models using PyRosetta (see Appendix A.7).
In summary, the G ENESIS -trRosetta de novo design framework uses a Form to build a Sketch that is then refined by G ENESIS , designed through trRosetta and finally assembled and minimized with a full atomistic energy function in PyRosetta.
Our ablation studies showed the importance of both G ENESIS and trRosetta modules.
First, we removed the trRosetta design module by gathering structural restraints directly from the G ENESIS distance- and orientation probabilities and using a poly-Valine AA sequence.
We see a low performance with the median TM-score (Zhang & Skolnick, 2005) below 0.5 and the median root-meansquared-deviations (RMSDs) around 4 ˚A between the predicted 3D model and the native structure on the training and test set examples across all major classes (Fig.
3B).
Second, we removed G ENESIS resulting in a framework where trRosetta is challenged to directly design sequences for a given
Sketch.
On the training examples, we obtained a TM-score median around 0.6 and an RMSD median around 2 ˚A for the 3-helical bundle architectures, while for the fully- β and mixed α / β architectures the results were rather modest with a median TM-score around 0.4 and the median RMSD around 3.
˚A. The few selected test examples follow the same trend as the train examples: G ENESIS alone is not sufficient to build native-like poly-Valine backbones, and simply using trRosetta to design sequences from Sketches resulted in a poor performance with sequences and constraints that do not recapitulate the target fold described by the Sketch.
Thus, our experiments support that, while G ENESIS cannot solve the backbone design problem by itself, its predicted features can guide sequence generating engines (trRosetta) towards the sequences that are more likely to adopt the specified folds.
On the other hand, a naively assembled Sketch lacks native-like features are required by trRosetta to use and design fold-specific sequences .
We assess the performance of different variations of the G ENESIS pipeline.
Using the basic framework Sketch → G ENESIS → trRosetta → PyRosetta, we achieved a TM-score of 0.8 and a median
RMSD of 2 ˚A for fully α -helical proteins, a median TM-score of 0.55 and median RMSD 3.5 ˚A for fully- β proteins, and a median TM-score of 0.5 with a median RMSD 4 ˚A for mixed α / β proteins.
We saw an improvement when adding a simple structural relaxation step that favors SSE pairing and packing after the gradient descent minimization with median TM-scores of approximately 0.8, 0.6, 0.55 and RMSDs of 2 ˚A, 3 ˚A, and 3.5 ˚A for α -, β -, and mixed α / β -proteins from the training set, respectively.
We also tested the pipeline using the trRosetta hybrid-design protocol, where, instead of optimizing for a single sequence, the algorithm optimizes for multiple sequences from which a position-specific scoring matrix (PSSM) is generated and used to guide the sequence design task.
The results were comparable to the standard pipeline in terms of TM-scores and RMSDs (Sup.
Fig.
In order to evaluate the generalization capability of G ENESIS , we trained and tested G ENESIS on the series of data splits given by SCOPe (see Appendix A.2).
The SCOPe database hierarchically classifies proteins based on structural similarities: The top level (Class) divides proteins into major classes:
fully- α , fully- β and mixed α / β .
The Fold-level arranges the structures according to
SSE disposition and connectivity.
The Superfamily and Family-levels further classify the structures according fine-grained structural and functional features.
At the Family-, Superfamily- and Fold-level test sets, α -helical proteins reach TM-scores and
RMSDs are around 0.6 and 3 ˚A, respectively, whereas for fully- β and mixed α / β proteins, a degradation in performance is observed (Sup.
Fig.
8B).
At the Superfamily-level test set, 2 out of 25 test proteins cross the critical 0.5 TM-score threshold for fully- β structures (Sup.
Fig.
8B).
For mixed α / β proteins, around 3 test proteins are predicted with a TM-score above 0.5 for both Superfamilyand Fold-level test sets (Sup.
Fig.
8B).
These results indicate that G ENESIS shows some ability for generalization across families, while proteins with different connectivities and structural features are more challenging to (re-)generate successfully.
3. C ONDITIONED SAMPLING AND DESIGN OF NATIVE PROTEIN TOPOLOGIES
To showcase the G ENESIS -trRosetta de novo design framework, we conditionally sample 5 different topologies.
We sample a 2-layer mixed α / β Ubiquitin-like fold, where 4 strands are packed against a helix, and a 3-layer mixed α / β Rossmann fold with a central 4 stranded β -sheet and 2 exterior packing helices on both sides.
We additionally challenge the framework by generating 2 different 2-layer β -sandwiches, an Immunoglobulin (Ig)-like fold and a Jelly-roll fold.
Finally, we design sequences that adopt the Top7 fold (Kuhlman et al., 2003), a novel fold not observed in the natural repertoire and representing a challenging generalization test for our method.
As we do not have prior knowledge about SSEs and loop lengths, we first sampled 20–30 combinations and generated a small set of sequences for these.
We then used AlphaFold (AF) Jumper et al.
(2021) to predict a structure for the initial set of sequences and realigned the AF onto the G ENESIS models and selected the combinations achieving a TM-score above 0.5.
Using the top combinations, we designed up to 10,000 sequences and predict AF models for each of them.
We assessed the quality of the designed sequences and models by comparing the TM-score between the G ENESIS (Gen) and AF model (TM-score(Gen,AF)) and the AFs’ median confidence score (median plDDT) (Fig.
4A).
We saw that with a stringent threshold of TM-score(Gen,AF) > 0.6 and median plDDT >around 10% or more of the designed sequences and models were selected as a success (red area in
Fig.
4A).
These results indicate that the G ENESIS -trRosetta framework is able to successfully yield backbone conformations that can harbor sequences with strongly encoded native structural features.
3. P ROBING THE “ DARK MATTER ” OF PROTEIN FOLDS
The ultimate goal of de novo protein design methods is to generate protein folds non-existent in nature.
We asked the question if our framework based on DNNs and trained on natural derived structure data is capable of (besides the Top7 fold) generalizing outside the distribution of natural folds e.g., if our framework is able to generate out-of-distribution.
To this end, we sought to sample protein folds not included in the training set, nor observed in nature.
Previously, Taylor and colleagues (Taylor et al., 2009) computationally analyzed possibly unexplored regions of the 3-layer mixed α / β fold space through C α -traces that obey constraints of natural protein structures, such as handedness of connection and loop-crossing.
We further reduced the set by discarding C α -traces that have mixed secondary structure types on the same layer, disembodied SSEs (unpacked) or nearly crossing loops.
We selected three distinct folds to design with the G ENESIS -trRosetta method.
We used the G ENESIS -trRosetta framework to sample sequences using as input different Forms varying in the loop and SSE lengths and selected the top models (by TM-score between the G ENESIS model and the C α -trace) to up-sample sequences.
All sequences were then fed into AF and the predictions aligned to the initial G ENESIS model.
By assessing the sequence-to-structure qualities of the designed models via the TM-score(Gen,AF) and the AFs’ median plDDT (Fig.
4B,C), we observe that the G ENESIS -trRosetta framework is capable of resolving blurry feature maps towards native-like distributions even for novel folds (Fig.
4B,C).
D ISCUSSION
We show that a specialized VAE termed G ENESIS is able to encode representations of idealized protein folds and decode native-like conformations.
By basing ourselves on distance- and orientation representations, we are able to alleviate the need of generating designable protein backbones in 3D space with fold-specific restraints and energy functions, and thereby also bypassing the need for designable backbones.
We couple G ENESIS to the trRosetta design engine to generate multiple sequences for the sampled distance- and orientation representations for a set of known and novel folds.
Our results demonstrate that the G ENESIS -trRosetta framework is capable of designing new proteins adopting known folds and novel folds non-existent in nature.
By changing secondary structure and loop lengths the overall size can be adjusted and different conformations sampled.
The generalization capability of G ENESIS is significant and can guide the trRosetta-Rosetta hybrid design method to sequences with strong fold signatures.
Using AlphaFold as an orthogonal test shows that many of these sequences adopt the intended target shape.
Additionally, our framework is considerably fast, within minutes to generate a sequence and a 3D model for a given target protein shape even on a CPU.
This demonstrates the usage of DNNs can leverage the automated generation of proteins normally only accessible through large-scale simulations.
Our work opens exciting new horizons for de novo protein design where control over the shape is desired.
For example, our method could be harvested to generate custom protein backbones such that they fit onto non-canonically structured protein interfaces.
Often nanomaterials exhibit highly regular patterns, and could therefore be engaged by secondary structures that are placed respecting the regularity constraints.
Another example where our method could be used is for the design of larger molecular assemblies that are constructed from smaller protein domains.
Often, the overall shape of the assembly is controlled by the shape of the individual subunits.
Hence, we expect that the versatility and speed of the G ENESIS -trRosetta method together with other potential deep neural network tools for protein design and engineering to explore the protein universe should be broadly useful. A CKNOWLEDGMENTS
B.E.C. is a grantee from the European Research Council (Starting grant - 716058), the Swiss National Science Foundation, and the Biltema Foundation.
Parts of the computational simulations were performed at the CSCS - Swiss National Supercomputing Centre through a grant obtained by
B.E.C.. Z.H. is supported by a grant from the National Center of Competence in Research in Chemical Biology. MB is supported in part by the ERC Consolidator Grant No. 724228 (LEMAN). JS is supported by the UKRI CDT in AI for Healthcare http://ai4health.io (Grant No. P/S023283/1). A A PPENDIX A. D ATASET GENERATION
We created two distinct datasets from the scope (v2.07 stable)
Fox et al.
(2014) domains of medium sizes (40–128 residues).
The pre-training data set was created by corrupting existing protein structures by removing the loops based on the DSSP (hydrogen bond estimation algorithm)
Kabsch & Sander (1983) assignments.
We remodel the loops as done in a Sketch, e.g. we add dummy residues (Nitrogen (N), Carbon (C), Carbon (C α ), Oxygen (O) backbone atoms with randomized torsion angles) along the shortest path between the two endpoints C α atoms of the consecutive SSEs.
We add as many dummy residues as in the native structure, hence the corrupted structure has the same length as its native counterpart.
This procedure leaves the native SSE dispositions that may incorporate important native structural features for the pre-training.
In total, we created 40,726 pairs.
We developed a program that creates small fold Sketches obeying simple topological rules such as non-crossing loops and loop distance restraints from the architecture types: EE EEE, EEE EEE, H EEE, H EEEE, H EEE H, HH EE, HH EEE, HH EE H, HHH, HHH EE (where “ ” represents a layer separation and E: β -strand and H: α -helix).
We searched the SCOPe domains for partial structural matches within 3 ˚A RMSD using MASTER Zhou & Grigoryan (2015; 2020) for each of the generated Sketches.
Importantly, a Sketch can partially match onto a native domain.
We crop the overlapping regions of the native domain at the first and the last residue of the matching Sketch.
SSEs within the cropped domain that do not map to SSEs in the Sketch are assigned as loops.
Furthermore, we remove domains larger than 128 residues and identical matches for the mapping to the same Sketch.
This resulted in a total of 35,435 Sketch - native domain pairs.
A. D ATA SPLITS
Within SCOPe, protein structures are hierarchically classified into groups where the “Class” groups proteins based on secondary structure content and organization (fully- α , fully- β , mixed- α/β ), “Fold” divides them based on SSE disposition and connectivity, “Superfamily” is based on structural features and “Family” contains the structures with similar sequences.
We pick protein families that represent compact structures with small loops for our family test set.
The test set includes the SCOPe families b.1.22.1, b.11.1.6, b.69.2.3, b.70.2.1, b.82.1.22, b.114.1.1, a.7.2.0, a.7.2.1, a.7.8.2, a.7.12.1, a.8.11.1, a.24.10.3, a.24.13.1, a.60.9.0, a.160.1.2, c.2.1.7, c.25.1.2, c.118.1.0, c.93.1.0, c.56.5.6, d.110.4.3, e.51.1.1, c.97.1.5, d.17.1.5, d.58.3.2, d.58.10.0, d.58.23.1, d.92.1.13, d.230.1.1, d.240.1.0.
We generate higher-level test sets (Fold and Superfamily) by removing all corresponding groups from the picked structures in the Family test set, e.g. for the Family b.1.22.1 the Superfamily is b.1.22 and the Fold is b.1..
Importantly, identical structures and Sketches in the training set were removed in order to avoid any biases during testing.
A. D ATA ENCODING
The coordinates of the Sketches and their native counterparts are encoded into a total of four 2D distance- and orientation feature maps as done by trRosetta.
Briefly, the first feature map is allagainst-all C β distances.
The second feature map is the dihedral “ ω ” that measures the rotation along the virtual axis of two connecting C β residues.
The distances and ω angles are symmetric, e.g. measuring from residue i to residue j will give the same result as measuring from residue j to residue i.
The third and fourth feature map are the “ θ ” dihedrals and the “ ϕ ” angles specifying the direction of C β of residue i with respect to residue j. Both, θ and ϕ are asymmetric metrics.
Together the four feature maps fully define a protein backbone in 3D space.
While we use real valued feature maps as input to G ENESIS , we bin the true feature maps according to the trRosetta scheme.
The distances from 2 to 20 ˚A are binned into 36 equally spaced segments (0.5 ˚A each) and a 37 th bin to indicate that pairs are not in contact.
The dihedral ( ω , θ ) and angular ( ϕ ) features are binned into 15° segments yielding 24, 24, and 12 with an additional bin indicating no contact, respectively.
Therefore, we have encoded the true feature maps into tensors of shape 128x128x1x37 for the distances, 128x128x1x25 for the dihedrals and 128x128x1x13 for the angles.
Thus, at each “pixel” (each residue pair) we have an additional dimension that can be seen as a Dirac distribution with a score of one for the bin with the distance and zero everywhere else.
A. G ENESIS ARCHITECTURE
The VAE includes an encoder, a decoder, and a loss function.
The input Sketchs’ feature maps (realvalued) of shapes 128x128x4 are processed by the encoder that is a sequence of four convolutional blocks.
A single block includes a 2D convolution, an instance norm and ELU activation followed by a 40% dropout.
From the compressed data representation, two multilayer perceptrons (MLPs) are used to predict predict means and covariances vectors (of sizes 128).
Using the reparametrization trick, a latent variable z from a normally distributed p ( z | x ) is sampled.
The decoder q ( y | z ) passes z through three blocks of 2D transposed convolution, instance norm, ELU activation and 40% dropout to create a decompressed representation.
The final layer of the decoder branches into four different output heads.
Each head is a convolutional block with a final softmax activation over each pixel yielding distance outputs of shape 128x128x1x37, two dihedral outputs of sizes 128x128x1x25 and an angular output of shape 128x128x1x13.
A. L OSS FUNCTION
Our loss function is composed of five individual losses (four reconstruction losses, and a loss on the latent space).
We use the Wasserstein distance (for details see Panaretos & Zemel (2019)) as reconstruction loss.
Let us define x P and y Q and the corresponding densities as p and q , respectively.
We assume that ( x, y ) ∈ R d .
Additionally, let us denote J ( P, Q ) all joint distributions J for ( x, y ) that have marginals P and Q .
Then the general Wasserstein distance can be written as
[Equation]
In the discrete case, when P and Q are distributions ( x 1 , . . . , x n ) and ( y 1 , . . . , y n ) the formulation becomes
[Equation]
In the case of 1D discrete distributions ( p = 1) , the 1-Wasserstein ( W 1 ) distance is also called Earth mover’s distance (EMD) and is efficiently computable.
The main advantage of the 1-Wasserstein distance compared to other measures such as the binary cross-entropy and the Kullback-Leibler (KL) divergence is that it takes into account the metric space.
This means that larger deviations from the predicted to the true distributions are more penalized while small errors are less penalized.
We define the reconstruction loss as the sum over the 1-Wasserstein distances between the predicted distributions ( ˆ D ) and the true distributions ( D ) of each pixel normalized by the length of the protein ( N AA ) .
Each pixel is defined as ( i, j ) where i = 1 , . .
. , n w and j = 1 , . .
. , n h with n w being the width and n h the height.
[Equation]
Note that the true distribution is modeled as a Dirac distribution supported by the true values, whereas the predicted distribution ( ˆ D ) is parametrized by the VAE decoder.
We additionally use the Kullback-Leibler ( KL ) divergence on the latent space normalized by the length of the protein to penalize latent vectors not following a Normal distribution KLD = 1 N AA KL ( p ( z | x )
∥ p ( z )) , with p ( z ) Normal (0 , 1) .
Thus the final loss is defined as
[Equation]
A. T RAINING REGIMEN
We use a batch size of 64 Sketches, the Adam optimizer with a starting learning rate of 0.001 for the pre-training and fine-tuning.
We reduce the learning rate with a step of 0.97 at each epoch during pre-training and every second epoch during fine-tuning.
A. M ODELING AND DESIGN
The trRosetta design framework is utilized to design a set of 1,000 sequences matching G ENESIS refined maps.
A position-specific scoring matrix (PSSM) is generated from the library of sequences and used within the PyRosetta protocol.
In a first stage, the PyRosetta protocol first generates a coarse-grained model using gradient descent with the optimized restraints and a single sequence from trRosetta design.
In a second stage we remove the restraints assuming that the generated coarse grain model has adopted the target shape.
We further optimize the coarse grain structure with a full atom protocol.
We use the Rosetta FastDesign with layer and PSSM sequence constraints during the design task and topological secondary structure energy bonuses during the relaxation task.
In this way, the full atom protocol improves the quality of the final sequence and structure model.
A. S UPPLEMENTARY FIGURES