R OTAMER D ENSITY E STIMATORS ARE U NSUPERVISED L EARNERS OF THE E FFECT OF M UTATIONS ON P ROTEIN -P ROTEIN I NTERACTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Protein-protein interactions play a fundamental role in a broad range of biological processes.
Predicting the effect of amino acid mutations on binding is crucial to protein engineering.
Traditional biophysical and statistical methods have dominated the area for years, but they depend heavily on expert prior and face the trade-off between efficiency and accuracy.
Recent success in deep learning for proteins has made data-driven approaches more appealing than ever.
Nevertheless, the major challenge is the scarcity of experimental mutational data annotated with the change in binding affinity.
In this work, we demonstrate that mutational effects on binding can be predicted by the change in conformational flexibility of the protein-protein interface.
We propose a flow-based generative model to estimate the probability distribution of conformation (named Rotamer Density Estimator,
RDE) and use entropy as the measure of flexibility.
The model is trained solely with protein structures and does not require the supervision of the experimental values of changes in binding affinities.
Further, the unsupervised representations extracted by the model can be used for prediction even more accurately using simple downstream neural networks.
The proposed method outperforms empirical energy functions and other machine learning-based approaches.
I NTRODUCTION
Proteins rarely act alone but usually interact with other proteins to orchestrate biological processes (Alberts & Miake-Lye, 1992; Kastritis & Bonvin, 2013).
For example, antibodies, a kind of immune system protein, recognize pathogens by binding to proteins on their surface and elicit immune responses by interacting with the receptor protein of immune cells
(Lu et al., 2018).
Since proteinprotein interactions determine a large number of biological functions, developing methods to modulate protein-protein interactions is critical.
A typical way to modulate protein-protein interactions is to mutate amino acids on the interface — some mutations improve the strength of binding while others weaken and even disrupt the interaction (Gram et al., 1992; Barderas et al., 2008).
Biologists choose either to increase or decrease the binding strength depending on specific goals.
For example, if one would like to enhance the effect of a neutralizing antibody against a virus, it is usually necessary to increase the binding strength between the antibody and the viral protein.
However, as the combinatorial space of amino acid mutations is large, it is not always feasible or affordable to conduct wet-lab assays to test all the viable mutations.
Therefore, computational approaches are needed to guide the identification of desirable mutations via predicting the mutational effect on binding strength measured by the change in binding free energy ( ∆∆ G ).
Traditional computational approaches are mainly based on biophysics and statistics (Schymkowitz et al., 2005;
Park et al., 2016; Alford et al., 2017).
Though having dominated the area for years, their limitations are non-negligible.
In general, biophysics-based methods face the trade-off between efficiency and accuracy as they rely on sampling from energy functions.
Statistical methods are efficient but their capacity is limited by the descriptors considered in the model.
Both biophysics and statistics-based methods rely heavily on human knowledge.
Thus, they can hardly benefit from the fast-growing of available protein structures.
These limitations mark that predicting the effect of mutation on protein binding remains an open problem.
Recently, deep learning has gained tremendous success in modeling proteins, making data-driven methods more appealing than ever (Rives et al., 2019; Jumper et al., 2021).
Nevertheless, challenges exist for developing deep learning-based models to predict mutational effects on protein-protein binding.
The major challenge is the scarcity of experimental data — only a few thousands of protein mutations annotated with the change in binding affinity are publicly available (Geng et al., 2019b).
This hinders supervised learning as the insufficiency of training data tends to cause over-fitting.
Another difficulty is the absence of the structure of mutated protein-protein complexes.
Mutating amino acids on a protein complex leads to changes on sidechain conformations (rotamers) (Najmanovich et al., 2000; Gaudreault et al., 2012).
They account for the change in binding free energy but we do not have the knowledge of how exactly the conformation changes upon mutation.
[Figure]
In this work, we take inspiration from the observation that finds its root in thermodynamics: Protein-protein binding generally leads to entropy loss on the interface and more entropy loss implies higher binding affinity 1 (Brady & Sharp, 1997;
Cole & Warwicker, 2002; Kastritis & Bonvin, 2013).
Intuitively speaking, amino acids on the interface become less flexible (lower entropy) after they get contact with other proteins due to the geometrical and physical restraints imposed by the binding partner (Figure 1).
Higher binding affinity implies better complementarity between the two parts, thus higher rigidity.
Therefore, we seek to predict the change in binding free energy via estimating the change in conformational flexibility .
Specifically, our method consists of three main parts.
The first part is a conditional generative model built upon normalizing flows on torus for estimating the density of amino acid sidechain conformations (rotamers) given the structural environment of the amino acid, named Rotamer Density Estimator (RDE).
We choose normalizing flows because it allows exact likelihood computation, which is indispensable for the second part of the method — entropy estimation.
We estimate the entropy of the rotamer distribution predicted by the model and use it as the measurement of conformational flexibility.
Finally, we use the entropy of protein-protein interfaces at different states to predict the change in binding free energy ( ∆∆ G ).
In addition to directly using the entropy to predict ∆∆ G , we also use simple neural networks to extract ∆∆ G from the unsupervised representations produced by the RDE.
Our method provides a solution to the aforementioned challenges.
Training the rotamer density estimator requires only protein structures.
Thus, it is an unsupervised learner of the effect of mutations on binding and it alleviates the difficulty rising from the scarcity of annotated mutation data.
In addition, our method does not require the structure of the mutated protein as input.
Rather, it treats mutated structures as latent variables and the rotamer density estimator is an approximator of the latent distribution.
Our method outperforms empirical energy functions and machine learning models for ∆∆ G prediction .
In addition, as a generative model for rotamers, the RDE predicts sidechain conformations accurately .
R ELATED W ORK 2. M UTATIONAL E FFECT P REDICTION FOR P ROTEIN -P ROTEIN B INDING
Traditional approaches to predicting the effect of mutation on protein binding can be roughly divided into two classes: biophysical methods and statistical methods.
Biophysical methods focus on modeling inter-atomic interactions, e.g. hydrogen bonding, electrostatic forces, etc., with mechanical and statistical energy functions.
These methods use their underlying energy function to sample conformations of the mutated protein complex and predict the change in binding free energy upon mutations (Schymkowitz et al., 2005;
Park et al., 2016; Alford et al., 2017; Steinbrecher et al., 2017).
Statistical methods are based on feature engineering.
Descriptors that summarize the geometric, physical, evolutionary, and motif properties of proteins are used to fit the statistical model that predicts the effect of mutations on binding free energy (Geng et al., 2019a; Zhang et al., 2020).
Traditional methods generally face the trade-off between speed and accuracy due to the time-consuming sampling process.
Sophisticated energy functions and feature engineering underlying these methods depend heavily on human knowledge, thus limiting their pace to improve with the fast-growing of available protein structures.
Recently, deep learning-based approaches to predicting mutational effects on protein binding have emerged.
We group them into three categories: end-to-end models, pre-training-based models, and unsupervised models.
End-to-end models take both mutant and wild-type (not mutated) protein structures along with other features as input and directly predict the difference in binding free energy.
End-to-end models achieve satisfactory correlation between prediction and ground truth on the whole SKEMPI benchmark consisting of many different structures (Shan et al., 2022), but the perstructure correlation, which is more relevant to practical use, is still low.
In attempt to settle the issue of data scarcity, another line of work proposes to pre-train a feature extraction network and then use regression models to predict the effect of mutations based on the learned features (Liu et al., 2021;
Yang et al., 2022).
Nevertheless, these pre-training strategies do not capture well the foundation of protein-protein interactions.
More recently, the success of protein language models in has drawn interest in adopting the mask-predict (BERT) paradigm to protein 3D structures (Wang et al., 2018;
Shroff et al., 2020;
Jing et al., 2020; Yang et al., 2022; Zhang et al., 2022; Hsu et al., 2022).
These methods partially mask amino acid types on a given protein backbone and use neural networks to recover the masked information.
It has been reported that the difference in the probability of amino acid types before and after mutation show correlation to the change in binding free energy
(Yang et al., 2022).
Hence, they can serve as unsupervised predictors of the mutational effects on binding.
2. M UTATIONAL E FFECT P REDICTION FOR S
Methods for predicting mutational effects for single proteins are either structure-based or sequencebased (evolution-based).
Structure-based approaches can also be divided into biophysical methods, statistical methods, and deep learning-based methods.
They aim at predicting thermal stability or fitness of the protein rather than binding free energy between proteins (Schymkowitz et al., 2005;
Park et al., 2016; Alford et al., 2017).
The mutational effects of single proteins can also be predicted using only sequences via mining its evolutionary history.
Traditionally, this is done by performing statistics on multiple sequence alignments (MSAs), which are constructed by searching from largescale sequence databases (Hopf et al., 2017; Riesselman et al., 2018;
Rao et al., 2021; Luo et al., 2021; Frazer et al., 2021; Notin et al., 2022).
Recent studies show that protein language models (PLMs) trained on large sequence databases are capable of evaluating mutations without MSAs (Meier et al., 2021).
We would like to highlight that, sequence-based (evolution-based) methods for single proteins are not suitable for protein-protein interactions due to the lacking of evolutionary information in most cases — protein-protein interactions involve two or more chains.
The chains might belong to different species (e.g. host and virus).
Sometimes inter-chain co-evolution simply does not happen (e.g. immune response where effective antibodies are produced to induce rapid clearance of pathogens, leaving no time for the pathogen to evolve).
These make it infeasible to predict mutational effects via mining sequence databases with existing powerful tools such as multiple sequence alignments, protein language models, etc.
Hence, currently, effective ways for predicting mutational effects on binding are based on structures rather than sequences alone.
M ETHOD 3. O VERVIEW AND P RELIMINARIES
Overview Our method consists of three parts.
The core component is the rotamer density estimator (RDE), a conditional normalizing flow that models the probability density of sidechain conformations (rotamers) given the amino acid type and environments (Section 3.2).
Next is the algorithm for estimating the entropy of the distribution parameterized by the normalizing flow (Section 3.3).
Finally, we show how to use the entropy of the mutated and wild-type protein-protein interfaces at both bound and unbound states to estimate the change in binding free energy ( ∆∆ G ) upon mutation, and how to use neural networks to predict ∆∆ G more accurately using the unsupervised representations from the RDE (Section 3.4).
A protein-protein complex is a multi-chain protein structure that can be divided into two groups.
Each group contains at least one protein chain and each chain consists of multiple amino acids.
For a protein complex containing n amino acids, we number them from 1 to n .
The two groups of the complex can be represented by two disjoint sets of indices A, B ⊂ { 1 . . .
n }
An amino acid is characterised by its type, position, orientation, and sidechain conformation.
We denote the type, position, and orientation of the i -th ( i ∈ { 1 . . .
n } ) amino acid as a i ∈ { 1 . . .
20 } , p i ∈ R 3 , and O i ∈ SO (3) respectively.
The sidechain conformation of the amino acid is called rotamer .
As the conformational degree of freedom of the sidechain is defined by rotatable bonds, a rotamer can be sufficiently parameterized by torsional angles w.r.t.
the rotatable bonds.
The number of torsional angles vary between 0 to 4 depending on the amino acid type.
For an amino acid with d torsional angles, we denote the k -th ( k ∈ { 1 . . .
4 } ) torsional angle by χ ( k ) i ∈ [0 , 2 π ) .
Collectively, all the torsional angles are denoted by a vector χ i =
( χ ( k ) i )
dk
=1 .
Using the language of geometry, an angle can represented by a point on the unit circle S 1 .
A vector consisting of d angular values reside on the product of d unit circle, known as the d -dimensional torus T D =
( S 1 ) D .
In this work, the first goal is to model the conditional probability density of rotamers, given the type, position, orientation, and prior rotamer of itself and other amino acids: p ( χ i | { a j , p j , O j , ˜ χ j } nj =1 ) .
The prior rotamers ˜ χ
j are inaccurate or unknown in many cases.
For example, if we mutate some amino acids in the protein complex, the rotamers of the mutated amino acids are unknown, and the rotamers of amino acids nearby the mutated ones are inaccurate because they are affected by the mutation.
The probability density is defined over the d -dimensional torus T D =
( S 1 ) D , and we show below our proposed flow-based architecture to model the density.
3. R OTAMER D ENSITY E STIMATOR
The rotamer density estimator (RDE) is designed to model the conditional distribution of the rotamer of the i -th amino acid given the information of itself and other amino acids: p ( χ i | { a j , p j , O j , ˜ χ j } nj =1 ) .
In this section, we first introduce the encoder network that produces hidden representations for each amino acid by taking into account its environment { a i , p i , O i , ˜ χ i } ni =1 .
Next, we formulate the conditional normalizing flow on S 1 for rotamers with only 1 torsional angle.
Based on the flow on S 1 , we further formulate the flows on d -dimensional torus T D ( D > 1 ) for rotamers with more than 1 torsional angle.
Encoder
Network The encoder network starts with two multi-layer perceptrons (MLPs) that generate embeddings for each single amino acid and each pair of amino acids respectively.
The MLP for single amino acid encodes the amino acid type, backbone dihedral angles, and local atom coordinates of each amino acid into a vector e i ( i = 1 . . .
n ).
The other MLP for amino acid pairs mainly encodes the distance and relative position between two amino acids and we denote a pair embedding vector as z ij ( i, j = 1 . . .
n ).
Next, we use a self-attention-based network invariant to rotation and translation (Jumper et al., 2021) to transform the single embeddings { e i } and pair embeddings { z ij } into hidden representations { h i } .
The hidden representation h
i is aimed at capturing the informa- tion of both the i -th amino acid itself and its structural environments.
It serves as an encoding of the condition { a j , p j , O j , ˜ χ j } nj =1 for the probability density w.r.t.
χ i .
Conditional Flow on S
We model the distribution of rotamer with 1 torsional angle with conditional normalizing flows on S 1 (Rezende et al., 2020).
A normalizing flow is basically a bijective function.
We choose to parameterize S 1 by an angle θ ∈ [0 , 2 π ] and construct a bijective function on [0 , 2 π ] which is equivalent to a bijective on S 1 .
A common method for constructing a bijective function is ensuring the strict monotonicity.
We adopt this method and construct a strictly monotonically increasing function on [0 , 2 π ] , denoted by f : [0 , 2 π ] → [0 , 2 π ] .
Due to the periodicity of angular values, 0 and 2 π are the same point.
In order to preserve the continuity at the two ends, we have to additionally ensure that i. f (0) = 0 , ii.
f (2 π ) = 2 π , and iii.
f ′ (0) =
f ′ (2 π ) .
We resort to the rational quadratic spline flow (Durkan et al., 2019;
Rezende et al., 2020) to construct a function that satisfies all the requirements.
Specifically, the spline-based bijective function is a piece-wise function that contains K pieces delimited by K + 1 knots, and each piece takes the following form:
[Equation]
where s k =
y k
+1 − y k x k +1 − x k , and ξ ( x ) =
[Equation]
k ( x ∈ [ x [Equation] x
[Equation]
+1 ]) .
[Equation]
The spline is parameterized by the coordinates and derivatives of the K + 1 knots denoted by x k , y k and δ k ( k = 1 . . .
K + 1 ).
To fulfill the requirement of monotonicity, continuity, and periodicity, [-we impose the following constraints on the knots: i. 
-]N [Table] F
N A [Table] E
[Table]
… … (A) (B)
[Table]
[Equation]
[Figure]
[Equation]
{+[Figure] [Equation] [Table] +Hbmt5+o0iyWD2aS0EDgoWQRI9hYqdUjI9b3+6WyV/XmQOvEz0kZcjT6pZ/eICapoNIQjrXu+l5iggwrwwinU7eXappgMsZD2rVUYkF1kM2vnaKKVQYoipV90qC5+rcjw0LriQhtpcBmpFe9mfif101NdBNkTCapoZIsFkUpRyZGs6+jAVOUGD6xBBPF7K2IjLDCxNiAXLeCSKpNLFA+b3lxKKauzclfTWWdtC6r/lW1dl8r1708sSKcwTlcgA/XUIc7aEATCDzCM7zCm/PivDsfzueitODkPaewBOfrF6h/nDg=</latexit>  we impose the following constraints on the knots: i. 
+}
= x 1 < x 2 < . . . < x K +1 = 2 π , ii.
0 =
y 1 < y 2 < . .
<
y K +
= 2 π , iii.
δ k > 0( k = 1 . . .
K +1) , and iv.
δ 1 =
δ K +1 .
The parameters are produced by transforming the hidden representation h i of the amino acid using neural networks, such that the probability distribution defined by the bijective is conditional on the amino acid and its environment.
Therefore, we may also denote the spline by f ( x | h i ) .
We choose the uniform distribution on [0 , 2 π ] as the base distribution, and denote it by p z ( z ) = 12 π ( z ∈ [0 , 2 π ] ).
We let f be the mapping from the target distribution to the base distribution.
According to the change-of-variable formula, the target rotamer density of the i -th amino acid is:
[Equation]
The derivative f ′ can computed analytically according to Eq.1.
The model is illustrated in Figure 2A.
In practice, we stack multiple bijectives to enable more complex transform.
The derivative of the composite can be computed efficiently by applying the chain rule.
At inference time, the inverse mapping f − 1 ( y ) can be computed efficiently (Rezende et al., 2020).
To find the solution of f − 1 ( y ) , the first step is to locate the unique bin that contains y .
Assuming y belongs to the k -th bin, finding its corresponding x amounts to finding the root of the quadratic equation f k ( x | x k,k +1 , y k,k +1 , δ k,k +1 )
=
y in [ x k , x k +1 ] , whose solution is closed-form.
Conditional Flow on T D
Rotamers with d torsional angles can be viewed as points on the ddimensional torus.
The D -dimensional torus is the product of D circles S 1 , i.e. T D = S 1 ×· · ·× S 1 .
Therefore, we can model the distribution on T D by modeling the joint distribution of D variables on S 1 .
In specific, we adopt the coupling layer technique to model the joint distribution (Dinh et al., 2016).
On each coupling layer, we update one dimension using the bijective for S 1 , keeping the other D − 1 dimensions fixed and using them along with the hidden representation of the amino acid as the condition to parameterize the bijective (Figure 2B):
[Equation]
where d is the dimension to update.
The coupling layer g d preserves invertibility and has closedform inversion.
The determinant of its Jacobian is equal to the derivative of f ( x j | x \ j , h i ) .
We stack multiple spline-based coupling layers to ensure that all the dimensions are updated at least once:
[Equation]
We choose the uniform distribution as the base distribution, denoted by p z
( z ) =
( 12 π ) D and let g map the target distribution to the base distribution.
The log-likelihood is computed by the changeof-variable rule:
[Equation]
Training Objective To train the rotamer density estimator, we minimize the negative loglikelihood of native (ground truth) rotamers:
[Equation]
where S = { a j , p j , O j , ˜ χ j } Nj =1 is a protein structure sampled from the dataset.
Since the model is designed for mutation analysis, the prior rotamers should emulate the rotamers after some amino acids are mutated.
Thus, we mask the rotamers of a random portion of amino acids and add noise to the rotamers of the neighbors of the masked ones.
The negative log-likelihood training objective function is evaluated on the masked amino acids and a random part of the perturbed amino acids.
3. R OTAMER E NTROPY E STIMATION
The rotamer density estimator models the distribution of possible rotamers of an amino acid.
Therefore, the entropy of the rotamer distribution is a natural measurement of conformational flexibility (Brady & Sharp, 1997):
[Equation]
[Equation]
We use the stochastic method to estimate the entropy.
First, we sample a set of rotamers from the distribution using the inversion of the flows (Eq.1, 4).
Then, we compute the negative log probability of the samples and take the average as an estimation of the entropy.
These two steps can be efficiently done thanks to the capability of computing the exact likelihood of normalizing flows.
For amino acids without rotatable sidechains (alanine, glycine, and proline), we consider their conformational entropy constant.
However, we can still estimate the entropy of amino acids nearby them to evaluate their impact on conformational flexibility.
3. M UTATIONAL E FFECT ( ∆∆ G ) P REDICTION
Assume that we have a protein-protein complex W LR
= { a i , p i , O i , ˜ χ i } consisting of N amino acids.
M of them belong to the first group L = { 1 , . . .
, M } , and the remaining N − M belong to the second group R =
{ M +1 , . . .
, N } .
In the unbound state, we denote the two separated structures by
W L and W R respectively.
We call group L ligand as it contains mutations and call group R receptor .
Consider that we mutate m amino acids in the group L , numbered by { 1 , . . .
, m } ⊂ L .
The mutated structure in the bound state is denoted by M LR =
{ ˜ a i , p i , O i , ∅ } mi =1 ∪ { a i , p i , O i , ˜ χ i } unmutated , where ˜ a i is the mutated amino acid type and ∅ indicates that the rotamers of the mutated amino acids are unknown.
The two groups in the unbound state are denoted by M L and M R respectively.
Note that M R = W R if no mutations are in the receptor group.
Linear Predictor
The entropy loss upon binding of a protein complex S ( S = W , M ) is defined by the difference of entropy between the bound state and the unbound state (Kastritis & Bonvin, 2013).
We approximate the entropy loss ∆ H S by linear model using our estimated entropy:
[Equation]
where w S J > 0 ( S = W , M ; J = L, R ) is the coefficient that controls the contribution of entropy to the binding free energy, and E ( · ) is the entropy bias for 20 different amino acid types.
According to the thermodynamic definition, the core component in binding ∆∆ G is the difference in entropy loss between the wild-type and mutated structure.
Therefore, we estimate ∆∆ G by:
[Equation]
Note that when there are no mutations in the receptor group ( M R = W R ), H unbnd M R and H unbnd W R cancel out each other and should have no contribution to ∆∆ G .
On some occasions where mutations are [Table] on both groups, we consider the set of only mutated amino acids as the ligand group and all the remaining as the receptor group.
To calibrate the parameters in Eq.9, we use the block coordinate descent method, which alternates between optimizing the coefficients ( w S J , b ) and the entropy biases ( E ) using the mean squared error (MSE) loss.
Neural Network Predictor
The hidden representation h i of each amino acid for parameterizing the normalizing flows contains sufficient information about the rotamer probability density.
To extract binding information from the representations in a more flexible way, we also try neural networks.
Specifically, we use MLPs to transform the representation h i and apply max-pooling to obtain a global representation of the structure.
Then, we subtract the representation of the wild-type structure from the representation of the mutant and feed it to another MLP to predict ∆∆ G .
To enforce anti-symmetry, we swap the wild-type and the mutant to predict − ∆∆ G and take (∆∆ G − ( − ∆∆ G )) / 2 as the final prediction.
The network is trained with the MSE loss.
During training, gradients are not back-propagated through h
i so the rotamer density estimator is frozen, as our goal is to exploit the unsupervised representations learned by the RDE.
3. M ODEL T RAINING
The datasets for training the rotamer density estimator are derived from PDB-REDO (Joosten et al., 2014), a database containing refined X-ray structures in PDB.
Structures with resolution worse than 3.5 ˚A are removed.
All the protein chains are clustered by 50% sequence identity.
This process leads to 38,413 chain clusters, and they are randomly divided into the training set, the validation set, and the test set by 95%/0.5%/4.5%.
At training time, the data loader randomly selects a cluster and then randomly chooses a chain from the cluster.
During training, the structure is cropped into a patch containing 128 amino acids.
The patch is constructed by first choosing a seed amino acid, and then finding its 127 nearest neighbors according to C-beta distances.
To emulate mutations, the rotamers of 10% of amino acids in the patch are masked.
Noise is added to the rotamers of amino acids whose
C-beta distance to the closest masked amino acid is less than 8 ˚A.
The SKEMPI2 database (Jankauskait˙e et al., 2019) is used to train the models for ∆∆ G prediction described in Section 3.4.
The dataset is split into 3 folds by structure, each containing unique protein complexes not appearing in other folds.
Two folds are used for training and validation, and the rest is used for testing.
This yields 3 different sets of parameters and ensures that every data point in SKEMPI2 is tested on once.
R ESULTS 4. P REDICTION OF THE E FFECT OF M UTATIONS ON B INDING
Baselines We compare our two ∆∆ G predictors RDE-Linear and RDE-Network with various categories of baseline methods.
The first category is traditional empirical energy functions, including Rosetta Cartesian ddG (Park et al., 2016; Alford et al., 2017; Leman et al., 2020) and FoldX (Delgado et al., 2019).
The second class is sequence/evolution-based methods represented by ESM-1v
[Figure]
(Meier et al., 2021), PSSM (position-specific scoring matrix), MSA Transformer (Rao et al., 2021), and Tranception (Notin et al., 2022).
The third category is end-to-end learning models, including DDGPred (Shan et al., 2022) and a model that shares the same encoder architecture with RDE but uses an MLP to directly predict ∆∆ G
( End-to-End ).
The fourth class is unsupervised/semisupervised learning methods including ESM-IF Hsu et al. and Masked Inverse Folding (MIF)
(Yang et al., 2022).
Similar to our RDE-Network, this class of methods pre-train a network on structures and uses the pretrained representations to predict ∆∆ G s.
The baseline MIF network also uses the same encoder architecture as RDE for comparison.
There are two variants for ∆∆ G prediction: MIF- ∆ logit uses the difference in log-probability of amino acid types to predict ∆∆ G , and MIF-Network predicts ∆∆ G from the learned representations using the network identical to the prediction network of RDE-Network.
Finally, as our method is based on conformational flexibility, we train a network to predict the B-factor of amino acids and use predicted B-factors in place of entropy to predict ∆∆ G .
Metrics Five metrics are used to evaluate the overall accuracy of ∆∆ G prediction: Pearson correlation coefficient, Spearman ’s rank correlation coefficient, minimized RMSE (root mean squared error), minimized MAE (mean absolute error), and AUROC (area under the receiver operating characteristic).
To calculate AUROC, we classify mutations according to the sign of ∆∆ G .
In practical applications, we often care about the correlation for a specific protein complex.
Therefore, we group mutations by structure, discard groups with less than 10 mutations and calculate correlations for each structure separately.
This leads to two other metrics: average per-structure Pearson correlation coefficient and average per-structure Spearman correlation coefficient.
Results According to Table 1, our RDE-Network outperforms all the baselines.
Notably, RDENetwork improves per-structure correlations by a large margin, which implies that it is significantly more reliable for practical applications.
The advantage of RDE-Network over MIF-Network shows that representations obtained by fitting rotamer densities are more effective than those from masked inverse folding because protein binding is driven by atomic interactions which RDE captures well by modeling the conformation of sidechain atoms.
RDE-Linear achieves performance comparable to Rosetta and outperforms some unsupervised representation learning baselines.
Though it does not outperform most of the baselines over the whole
SKEMPI2 dataset, we find that its performance is much better when considering only single-point mutations (Table 6 in the appendix).
The reason might be that simple linear models cannot capture well the non-linear relationship dominating multi-point mutations.
Anyway, RDE-Linear shows that using simple statistics of the estimated rotamer density alone can predict ∆∆ G , laying the foundation for the more accurate RDE-Network.
Sequence-based models do not predict ∆∆ G accurately for protein-protein binding in accordance with the discussion in Section 2.2.
Figure 3 plots the distribution of per-complex correlation coefficients.
Please refer to Section B of the appendix for more results and discussion.
4. O PTIMIZATION OF H UMAN A NTIBODIES A GAINST SARS-C O V-
Shan et al.
(2022) identifies 5 single-point mutations on a human antibody against SARS-CoV-2 that enhance neutralization (effectiveness).
There are 494 possible single-point mutations on the heavy chain CDR region of the antibody in total.
We use the most competitive methods benchmarked in
Section 4.1 to predict ∆∆ G s for all the mutations and rank them in ascending order (lowest ∆∆ G in the top).
A predictor is considered more effective if it ranks more favorable mutations in the top place.
Table 2 shows that RDE-Network and DDGPred successfully identify three mutations (Ranking ≤ 10%), and RDE-Network ranks them higher.
[Table] [Table]
4. A NALYSIS OF THE E NTROPY E STIMATED BY THE R OTAMER D ENSITY E STIMATOR
Statistical Significance To show that there is a statistically significant relationship between the entropy estimated by the RDE and the experimental ∆∆ G values, we conduct linear regression analysis using the RDE-Linear model defined in Eq.9.
The linear model contains 7 coefficients and 1 bias: w bound W L , w bound W R , w unbnd W L , w bound M L , w bound M R , w unbnd M L , ( w unbnd M R − w unbnd W R ) , and b .
Here, w unbnd M R and w unbnd W R are merged because the receptor is not mutated, so the entropy estimated by the model are identical.
For short, we denote the merged coefficient by w unbnd R .
We perform linear regression on the
SKEMPI2 dataset and present the regression coefficients, bias, and P-values in Table 3.
According to the statistics, all the entropy terms except the entropy of unbound receptor H unbnd R (coefficient w unbnd R ) have significant relationship with experimental ∆∆ G s.
The coefficients of the significant terms are all positive and roughly close to each other.
H unbnd R has no contribution because the receptor alone does not involve in the mutation.
These agree with the thermodynamic definition of the change in binding free energy:
[Equation]
where G M R and G W R cancel each other as the receptor is unmutated.
Therefore, it is evidenced that our model captures well the thermodynamics underlying protein-protein interactions.
Correlation Between Estimated Entropies and B-factors B-factor is an experimental measurement of conformational flexibility.
We collect the average b-factor of sidechain atoms of amino acids in the test split of the PDB-REDO dataset.
Simultaneously, we use the RDE to estimate the conformational entropy for each amino acid in the testset.
The average Pearson correlation coefficient between these two quantities is 0.4637 and the average Spearman coefficient is 0.4282.
Detailed results are presented in Table 10 in the appendix.
In summary, this supports that the entropy estimated by the RDE is correlated to experimentally determined conformational flexibility.
4. P REDICTION OF S IDECHAIN C ONFORMATIONS
[Table]
The rotamer density estimator (RDE) is a generative model for sidechain structures.
It can be used to predict sidechain conformations by sampling from the estimated distribution.
We use the RDE to sample sidechain torsional angles (rotamers) for structures with sidechains removed in our test split of PDB-REDO.
For each amino acid, 10 rotamers are sampled independently, and the one with the highest probability is selected as the final prediction.
We compare RDE with two baseline methods Rosetta (fixbb) (Leman et al., 2020) and SCWRL4 (Krivov et al., 2009).
Table 4 shows that RDE outperforms the baselines on all four torsional angles in terms of angular errors.
Detailed per-amino-acid accuracy is presented in Table 11 in the appendix.
C ONCLUSIONS AND L IMITATIONS
In this work, we propose the rotamer density estimator (RDE) that estimates the distribution of rotamers.
We find the entropy of the estimated distributions and the unsupervised representations produced by the RDE enable more accurate prediction of binding ∆∆ G .
One of the major limitations of this work is that it cannot directly model backbone flexibility.
Neither do all the considered baselines.
Therefore, an important non-trivial future direction is extending the proposed model to backbone-flexible cases.
A I MPLEMENTATION D ETAILS A. N ETWORK A RCHITECTURE
The encoder of the rotamer density estimator is a stack of attention layers invariant to the rotation and translation (Jumper et al., 2021) of the input protein structure.
Let h ℓi denote the embedding of the i -th amino acid output by the previous attention layer.
The logit of attention weights is between residue i and residue j is defined as:
[Equation]
where Q , K , G , and H are MLP networks that transform the embeddings into queries, keys, pairwise bias, and distance bias, respectively.
The attention weight is computed by taking the softmax on the j dimension: w ij = softmax j ( a ij ) .
In practice, we use multiple attention heads.
Each attention head has different attention weights.
The vector to update the representation of residue i from residue j is computed by:
[Equation]
Finally, the sum of { v ij } j weighted by { w ij } j is used to update the representation of residue j using residual connection and layer normalization, similar to the standard transformer architecture. The rotamer density estimator has 6 encoder layers.
Node features and pairwise features havechannels and 64 channels respectively.
The normalizing flow has 8 blocks and each spline hasknots, dividing [0 , π ] into 64 bins.
A. T RAINING
The rotamer density estimator is trained using the Adam optimizer for 200K iterations.
The learning rate is 0.0001.
The learning rate decays by 0.8 if the validation loss does not decrease in the last 5 validation steps (the model is validated every 1000 iteration), until the learning rate reaches 0.000001.
The batch size is 64.
It takes 8h56m in total on a single A100 GPU.
To emulate mutations, the rotamers of 10% of amino acids are masked.
Noise is added to the rotamers of amino acids whose C-beta distance to the closest masked amino acid is less than 8.0 ˚A.
The noise added to χ angles consists of two components.
The first component is Gaussian noise centered at 0 wrapped into [ − π, π ] .
It standard deviation is dependent on the C-beta distance: σ ij = − 116 β ij + 1 , where i is the index of the nearest masked amino of the j -th amino acid that the noise is added to, and β ij is the C-beta distance between them.
The second component is random flipping (adding π to the angle).
Every χ angle in the 8 ˚A neighborhood has 25% chance of being flipped.
Our noise model is totally empirical.
There are other ways to perturb rotamers, for example, using rotamer libraries (Dunbrack Jr & Karplus, 1993; Bower et al., 1997; Dunbrack Jr, 2002; Shapovalov & Dunbrack Jr, 2011).
We leave the problem of finding an optimal noise model that emulates mutations in future work.
A. B ASELINES
Baselines that require training and calibration using the SKEMPI2 dataset (DDGPred, End-to-End,
B-factor, MIF- ∆ logit, MIF-Network) are trained independently using the 3 different splits of the dataset as described in Section 3.5.
This is to ensure that every data point in the SKEMPI2 dataset is tested on once.
Below are descriptions of the implementation of baseline methods.
Rosetta (Alford et al., 2017; Leman et al., 2020) The version we used is 2021.16, and the scoring function is ref2015 cart .
Every protein structures in the SKEMPI2 dataset are first pre-processed using the relax application.
The mutant structure is built by cartesian ddg .
The binding free energies of both wild-type and mutant structures are predicted by interface energy (dG separated/dSASAx100).
Finally, the binding ∆∆ G is calculated by substracting the binding energy of the wild-type structure from the binding energy of the mutant.
FoldX (Delgado et al., 2019) Structures are first relaxed by the RepairPDB command.
Mutant structures are built with the BuildModel command based on the repaired structure.
The change in binding free energy ∆∆ G is calculated by subtracting the wild-type energy from the mutant energy. ESM-1v (Meier et al., 2021) We use the implementation provided in the ESM open-source repository.
Protein language models can only predict the effect of mutations for single protein sequences.
Therefore, we ignore cases where mutations are on multiple sequences.
We extract the sequence of the mutated protein chain from the SEQRES entry of the PDB file.
We use masked-marginals mode to score both wild-type and mutant sequences and use their difference as the estimation of ∆∆ G .
PSSM We construct MSAs from the Uniref90 database (Suzek et al., 2007) for chains with mutation annotations in the SKEMPI dataset.
We use Jackhmmer (Johnson et al., 2010) version 3.3.1 following the setting in Meier et al. The MSAs are filtered using HHfilter (Steinegger et al., 2019) with coverage 75 and sequence identity 90. This HHfilter parameter is reported to have the best performance for MSA Transformer according to Meier et al. We calculate position-specific scoring matrices (PSSM) and use the change in probability as the prediction of ∆∆ G .
MSA Transformer (Rao et al., 2021)
We use the implementation provided in the ESM open-source repository.
We input the MSAs constructed during the evaluation of PSSM to MSA Transformer.
We used masked-marginals mode to score both wild-type and mutant sequences and use their difference as the prediction of ∆∆ G .
Tranception (Notin et al., 2022)
We use the implementation provided in the Tranception opensource repository.
We predict mutation effects using the large model checkpoint.
Previously built
MSAs (not filtered by HHfilter) are used for inference-time retrieval.
DDGPred (Shan et al., 2022)
We use the implementation accompanying the paper by Shan et al.
Since this model requires predicted sidechain structures of the mutant, we use mutant structures packed during our evaluation of Rosetta to train the model and run prediction.
End-to-End The end-to-end model shares the same encoder architecture as the rotamer density estimator.
The difference is that in the RDE normalizing flows follow the encoder to model rotamer distributions, but in the end-to-end model, the embeddings are directly fed to an MLP to predict ∆∆ G .
B-factor This model predicts per-atom b-factors for proteins.
It has the same encoder architecture as the RDE.
Following the encoder is an MLP that predicts a vector for each amino acid where each dimension is the predicted b-factor of different atoms in the amino acid.
The amino acid-level b-factor is calculated by averaging the atom-level b-factors.
The predicted b-factors are used as a measurement of conformational flexibility.
They are used to predict ∆∆ G using the linear model same as RDE-Linear defined in Eq.9. ESM-IF (Hsu et al., 2022)
ESM-IF can score protein sequences using the log-likelihood.
The scoring function implementation is provided in the ESM repository.
We enable the --multichain backbone flag to let the model see the whole protein-protein complex.
We subtract the log-likelihood of the wild-type from the mutant to predict ∆∆ G .
MIF Architecture
The masked inverse folding (MIF) network uses the same encoder architecture as RDE.
Following the encoder is a per-amino-acid 20-category classifier that predicts the type of masked amino acids.
We use the same PDB-REDO train-test split to train the model.
At training time, we randomly crop a patch consisting of 128 residues, and randomly mask 10% amino acids.
The model learns to recover the type of masked amino acids with the standard cross entropy loss.
MIF- ∆ logit To score mutations, we first mask the type of mutated amino acids.
Then, we use the log probability of the amino acid type as the score.
Analogously, we have the score of the wildtype bound ligand, wild-type bound receptor, wild-type unbound ligand, unbound receptor, mutated bound ligand, mutated bound receptor, and mutated unbound ligand.
Therefore, we use the linear model identical to RDE-Linear (Eq.9) to predict ∆∆ G from the scores.
MIF-Network This is similar to RDE-Network.
The difference is that we use the pre-trained encoder of MIF rather than the encoder of RDE.
We also freeze the MIF encoder as we aim to utilize the unsupervised representations.
A. S OURCE C ODE
The code and data will be made available once this paper is made public.
B A DDITIONAL R ESULTS B. F URTHER A NALYSIS ON THE P ERFORMANCE OF S EQUENCE -B ASED M ETHODS
As discussed in Section 2.2, sequence-based (evolution-based) methods are not suitable for proteinprotein interactions due to missing co-evolutionary information between two proteins. This is supported by the results in Table 1, which show that sequence-based methods do not accurately predict ∆∆ G . Below we analyze two classes of PPIs to provide a better understanding of the performance of sequence-based methods on PPIs.
Antibody-antigen binding is a typical class of PPI that lacks co-evolutionary information.
The first reason is that the binding interface of antibodies (complementarity determining region) is highly variable (Weitzner et al., 2015). There is no evolutionary history in the region. Therefore, it is infeasible to mine the preference of mutations from sequence databases. Another reason is that antigens do not evolve to increase binding to antibodies.
The evolution of antigens is typically independent of specific antibodies.
Hence, in most cases, sequence databases can provide little information about mutational effects on antibody-antigen binding.
We select antibody-antigen complexes from the
SKEMPI datasets and evaluate per-structure Spearman correlation coefficients of the prediction by
MSA Transformer and RDE-Network on these antibody-antigen complexes.
The average Spearman score of MSA Transformer is 0.0744 and the average score of RDE-Network is 0.4284.
Further, we plot the results in Figure 4, where x -axis
and y -axis are per-structure Spearman coefficients of MSA
Transformer and RDE-Network respectively.
Orange crosses represent antibody-antigen complexes and blue dots represent other complexes.
The results show that under circumstances where we have little co-evolutionary information such as antibody-antigen binding, structure-based methods represented by our RDE perform better than evolution-based methods represented by MSA Transformer.
When the proteins in a complex come from the same organism, evolution-based methods are more likely to be effective.
Proteins in such complex usually function as a whole in the organism, so they evolve together.
Mutations that enhance the complexation might be more favorable and this preference might be reflected in evolutionary history.
We inspect 10 complexes on which MSA
Transformer performs the best in terms of Spearman coefficients, and find that 9 of them consist of proteins from a single organism (Table 5).
However, when evaluating MSA Transform on all the single-organism complexes, its Spearman score is 0.1651, which is still not high though better than antibody-antigen cases.
The reason might be that even if the proteins in a complex come from the same organism, the member protein might also bind to other proteins to be functional.
In this case, it is more difficult to predict its binding in a specific complex according to its general evolutionary history.
[Figure]
N e t w o r k
( S p e a r m a n )
[Table]
[Table]
[Table]
[Table]
End-to-End 0.
B-factor 0. ESM-IF 0. MIF- ∆ logit 0. MIF-Net.
RDE-Linear 0.
RDE-Net.
[Table] [Table] [Table] [Figure]
C ORRELATION WITH B-F ACTOR
[Table] [Table] [Table]