Non-identiﬁability and the Blessings of Misspeciﬁcation in Models of Molecular Fitness
Abstract
Understanding the consequences of mutation for molecular ﬁtness and function is a fundamental problem in biology.
Recently, generative probabilistic models have emerged as a powerful tool for estimating ﬁtness from evolutionary sequence data, with accuracy sufﬁcient to predict both laboratory measurements of function and disease risk in humans, and to design novel functional proteins.
Existing techniques rest on an assumed relationship between density estimation and ﬁtness estimation, a relationship that we interrogate in this article.
We prove that ﬁtness is not identiﬁable from observational sequence data alone, placing fundamental limits on our ability to disentangle ﬁtness landscapes from phylogenetic history.
We show on real datasets that perfect density estimation in the limit of inﬁnite data would, with high conﬁdence, result in poor ﬁtness estimation; current models perform accurate ﬁtness estimation because of, not despite, misspeciﬁcation.
Our results challenge the conventional wisdom that bigger models trained on bigger datasets will inevitably lead to better ﬁtness estimation, and suggest novel estimation strategies going forward.
Introduction
The past decades have witnessed a tremendous increase in the scale of genome sequence data available from across life.
Recently, methods for estimating molecular ﬁtness using generative sequence models have seen widespread success at translating this evolutionary data into predictions of the functional consequences of mutation.
Such models have been shown to accurately predict the outcomes of experimental assays of protein function [23, 44, 36], and have been applied to infer 3D structures of RNA and protein [35, 56] and to design novel proteins [50, 47, 33].
The models have also been used to predict whether human mutations are pathogenic, directly informing the diagnosis of genetic disease [19].
In this paper, we investigate how and why generative sequence models ﬁt to evolutionary sequence data are successful at estimating molecular ﬁtness, and how they might be improved and generalized going forward.
Existing approaches to ﬁtness estimation with generative sequence models rest on an assumed relationship between density estimation and ﬁtness estimation.
Given a dataset of sequences X 1 , . . .
, X N , assumed to be drawn i.i.d.
from some underlying distribution p 0 , ﬁtness models proceed by (1)
[Table]
+HOv3HaZqGtBy4czrmXe+/xY0alsqxvY2l5ZXVtPbeR39za3tk19/ZbMkoEJk0csUg4PpKE0ZA0FVWMOLEgiPuMtP3h9cRv3xMhaRQ21CgmLkf9kAYUI6Ulzzx0PBt2JeWwfpc2itXSuPhwCh2vWvLMglW2poCLxM5IAWSoe+ZXtxfhhJNQYYak7NhWrNwUCUUxI+N8N5EkRniI+qSjaYg4kW46fWEMT7TSg0EkdIUKTtXfEyniUo64rzs5UgM5703E/7xOooJLN6VhnCgS4tmiIGFQRXCSB+xRQbBiI00QFlTfCvEACYSVTi2vQ7DnX14krUrZPitXbs8Ltassjhw4AsegCGxwAWrgBtRBE2DwCJ7BK3gznowX4934mLUuGdnMAfgD4/MHkY+U9w==</latexit>
C D ﬁtting a probabilistic model q θ to X 1: N and (2) using the inferred density log q ˆ θ ( x ) ≈ log p 0 ( x ) as an estimate of the ﬁtness f ( x ) of a sequence x ; this estimate in turn is used to predict other covariates such as whether the mutated sequence is pathogenic [23, 44, 19].
Innovation in ﬁtness models has come out of a trend of building increasingly ﬂexible models ﬁt to increasing amounts of data: simple models that treat each column of a sequence alignment independently were improved by energy-based models that accounted for epistasis [23], which in turn were improved by deep variational autoencoders [44], which in turn were improved by deep autoregressive alignment-free models [50, 33, 36].
Naively, one might assume that these improvements have come from obtaining better and better estimates of the data distribution p 0 , and improvements will continue with bigger models and bigger datasets.
In this article, we argue that this presumption is incorrect.
Technical summary
First, we show that that the true data distribution p 0 may not reﬂect ﬁtness, and argue instead that we should be focused on estimating another distribution that does, p ∞ (the “stationary distribution”, to be deﬁned below).
In particular, we demonstrate that phylogenetic effects – i.e. the history of how current sequences evolved over time – can “distort” the observed data, leading to a situation where p 0 (cid:54) = p ∞ (Sec. 2).
Second, we show in this situation that p ∞ and ﬁtness f are non-identiﬁable: even with inﬁnite data, there always exists some alternative ﬁtness function ˜ f
that explains the same data just as well as f .
This sets fundamental limits on what we can learn about ﬁtness from evolutionary data (Sec. 3).
Third, although exact estimation of p ∞ is impossible, we show that it is still possible to get closer to p ∞ than p 0 , that is, to ﬁnd a better estimator of ﬁtness than the true data density p 0 .
This can be done by ﬁtting to data a parametric generative sequence model M = { q θ : θ ∈ Θ } that is (approximately) well-speciﬁed with respect to p ∞ (i.e. p ∞ ∈ M ) but misspeciﬁed with respect to the data distribution p 0 (i.e. p 0 / ∈ M ), thus illustrating how misspeciﬁcation can be a blessing rather than a curse (Sec. 4).
Fourth, we construct a hypothesis test to determine whether the blessings of misspeciﬁcation occur on real data, for existing ﬁtness estimation models; our test uses a recently developed Bayesian nonparametric sequence model to construct a credible set for p
(Sec. 6).
Fifth, we apply our test to over 100 separate sequence datasets and ﬁtness estimation tasks, to conclude that existing ﬁtness estimation models systematically outperform the true data distribution p 0 at estimating ﬁtness (Sec. 7).
The takeaway is that better ﬁtness estimation (i.e. better p ∞ estimation) will not come from better density estimation (i.e. better p 0 estimation); bigger models and bigger datasets are not enough.
Instead, better ﬁtness estimation can come from developing models that describe p ∞ better but the data density p 0 worse .
Models of Fitness and Phylogeny
In this section we illustrate how p 0 may not accurately reﬂect the true ﬁtness landscape, by analyzing a generative model of sequence evolution that takes into account not only ﬁtness but also phylogeny.
Our model is general: it allows for arbitrarily complex epistatic ﬁtness landscapes, and recovers standard phylogenetic and ﬁtness models as special cases.
Our concerns about the effects of phylogeny on ﬁtness estimation are motivated by the widespread use – and trust – of phylogenetic models for evolutionary sequence data (phylogenetic models are far more widely applied than ﬁtness models) [21, 12, 17, 18].
Note, however, that phylogeny is only one reason why p 0 may not not reﬂect the true ﬁtness landscape, and that our later results on the beneﬁts of misspeciﬁcation in ﬁtness models (Sec. 4) do not depend on the speciﬁc cause of mismatches between p 0 and the ﬁtness landscape.
Joint ﬁtness and phylogeny models We deﬁne “joint ﬁtness and phylogeny models (JFPMs)” using two elements: a description of how individual species (or populations or individuals) change over time, which depends on ﬁtness f , and a description of the species’ relationship to one another, a phylogeny H .
To describe the dynamics of individual species, let P τ ( x, x 0 ) denote the probability of sequence x 0 evolving into sequence x after time τ ; in particular, P τ ( x, x 0 ) is assumed to be the transition probability of an irreducible continuous-time Markov chain deﬁned over sequence space X .
For example, under neutral evolution (i.e. without selection based on ﬁtness), P τ ( x, x 0 ) may follow a Jukes-Cantor model [18].
With selection, for simple population genetics models (e.g. Moran or
Wright processes), Sella and Hirsh [49] demonstrate under general conditions that for any x 0 ,
[Equation]
where f ( x ) is the log ﬁtness of the sequence x and β > 0 is a constant (Appx. B).
The implication of Eqn. 1 is that the stationary distribution of the evolutionary dynamics follows a Boltzmann distribution, with energy proportional to the log ﬁtness of the sequence.
Estimating p ∞ is of interest because it provides a direct estimate of log ﬁtness, up to a linear transform, since f ( x )
= β − 1 (log p ∞ ( x ) + log Z ) .
(N.b. in the remainder of the paper, when we say “estimate ﬁtness” we mean, implicitly, “estimate log ﬁtness up to a linear transform”.)
The sequences we observe, however, do not necessarily come from the stationary distribution.
Instead, they are correlated with one another according to their evolutionary history.
This is described by a phylogeny H =
( V, E, T ) consisting of a directed and rooted full binary tree with edges E and nodes V , along with time labels for the nodes, T : V → R + (Fig.
1A).
Each node v is associated with a sequence X v , drawn as X v ∼ P ∆ t ( x, X v 0 ) , where X v 0 is the sequence of the parent node, v is the child node, and ∆ t = T ( v 0 ) − T ( v ) is the length of the edge between them (Fig. 1B).
The root sequence is drawn from p ∞ .
The observed datapoints X 1 , . . .
, X N correspond to the leaf nodes.
In general we will assume all leaves are observed at effectively the same time, the present day T = 0 .
Special cases Standard probabilistic phylogenetic models ignore ﬁtness and assume
Assumption 2.1 (Pure phylogeny models (PMs)) .
Constant ﬁtness: f ( x ) = C .
Example models that ﬁt this form include most of those used in BEAST [14], MrBayes [25],
RaxML [51], etc.
Standard probabilistic ﬁtness models, on the other hand, ignore phylogenetic history and assume that the stationary distribution has been reached,
Assumption 2.2 (Pure ﬁtness models (FMs)) .
Let τ i be the distance in time between observed sequence X i and its parent node.
Take τ i → ∞ for all i , which implies that
[Equation]
The key implication of this assumption is that density estimation and ﬁtness estimation are linked: the data follows X 1 , . . .
, X N ∼ iid p 0 = p ∞ , and so if we can estimate p 0 we can estimate the ﬁtness.
Example models include EVMutation [23], DeepSequence [44], EVE [19], etc.
Note although
Assumptions 2.1 and 2.2 do not conﬂict directly, conclusions made based on them conﬂict in practice:
PMs typically infer ﬁnite and different lengths for branches (i.e. τ i < ∞ ), while FMs typically infer differences in ﬁtness (i.e. f ( x ) (cid:54)
= C ), even when applied to the same dataset.
1D Example
If Asm.
2.2 does not hold, then there is no reason for the distribution of observed sequences X 1 , X 2 , . . .
to follow p ∞ .
We illustrate this with an example, the most widely used JFPM that does not use Assumptions 2.1 or 2.2: the Ornstein-Uhlenbeck tree (OUT) model [18, 8].
In this model, X is continuous, i.e. X ∈ R , and evolves on a quadratic ﬁtness landscape of the form f ( x ) ∝ ( x − µ ) 2 + C according to the dynamics P [Equation] ( x,
[Equation]
The stationary distribution p ∞ is Normal( µ, σ 2 ) .
One can show (Appx. C.1)
that for any H ,
Proposition 2.3 (OUT observations) .
The distribution of observed genotypes X 1: N is drawn from a multivariate normal distribution with mean µ(cid:126) 1 N and covariance Σ where
[Equation]
and t ij ( H ) is the total time of the shortest path between leaves i and j along the phylogeny H .
We drew samples from the OUT with a Kingman coalescent prior on H ([3] Def. 2.1) and plotted their density (Fig. 1C).
Even as N → ∞ , the distribution of samples does not follow p ∞ .
Moreover, rerunning the process with a new sample from the prior yields a very different distribution (Fig. 1D).
Non-identiﬁability
In this section we investigate whether, given inﬁnite sequence data, it is possible to infer ﬁtness f without Asm. 2.2, and conversely, whether it is possible to infer phylogeny H without Asm. 2.1.
That is, we are interested in whether ﬁtness and phylogeny are identiﬁable in general JFPMs.
We conclude they are not: given inﬁnite data generated with any f and H , there exists some alternative ˜ f and ˜ H , where ˜ H satisﬁes Asm. 2.2, that explains the data equally well.
Naively, this result may be surprising: in FMs, each sequence is drawn independently, i.e.
X i ⊥⊥ X j | H , f , while in JFPMs and PMs there is (in general) correlation between sequences, i.e.
X i (cid:54)⊥⊥ X j | H , f .
One might then hope that examining correlations between sequences would enable us to infer whether Asm. 2.2 holds.
However, we can show that these correlations are uninformative due to a symmetry in JFPMs, exchangeability.
Assumption 3.1 (Exchangeability) .
Let m ( X 1 , X 2 , . . . )
denote the marginal probability of an inﬁnite set of sequences X 1 , X 2 , . .
integrating over all phylogenies, i.e. m ( X 1 , X 2 , . . . )
= (cid:82) p ( X 1 , X 2 , . . .
| H ) p ( H ) d H .
Then, for any permutation π of the integers,
[Equation]
Exchangeability says that if we had observed the sequences in a different order, it would not change their probability.
Exchangeability is a ubiquitous assumption in machine learning and statistics models, and its application depends primarily on the information available in a dataset: it is a sensible assumption whenever the ordering of the datapoints provides no useful information.
In typical datasets used for ﬁtness estimation, sequences are separated by millions of years of evolution, and are thus all effectively observed at the same time: the present day, T = 0 . In other words, there is no a priori way of ordering the sequences in the dataset, and so we must assume exchangeability. Standard priors on phylogenetic trees, such as the Kingman coalescent, are explicitly constructed to enforce exchangeability [3, 14].
Exchangeability implies that ﬁtness and phylogeny are not identiﬁable.
Even if X 1 , X 2 , . . . are generated from a JFPM with a ﬁnite branch length phylogeny H , we can describe the same data just as well using an FM model with an inﬁnite branch phylogeny ˜ H :
Theorem 3.2 (Non-identiﬁability) .
Assume X 1 , X 2 , . .
satisfy Assumption 3.1.
Then with probability 1 there exists some function ˜ f such that
[Equation]
Proof.
Applying de Finetti’s Theorem ([29], Thm. 11.10), a.s. there exists a random measure G such that for i ∈ { 1 , 2 , ... } , X i iid ∼ G .
Let p G ( x ) be the pmf of G .
(We assume x is a ﬁnite discrete sequence; we can also use continuous x assuming the pdf p G ( x ) exists.)
Set ˜ f
( x ) =
[ p G ( x )] 1 /β .
This result says that the observed sequences from an exchangeable JFPM, X 1 , X 2 , . .
. , are precisely i.i.d.
samples from some p 0 .
Although in the standard tree representation X i (cid:54)⊥⊥ X j | H , f , there must be some alternative description of the same process where X i ⊥⊥ X j | ˜ H , ˜ f .
Fitness and phylogeny are thus non-identiﬁable: data generated from a JFPM with ﬁtness f and phylogeny H can be described just as well using ˜ f and ˜ H , and vice versa.
We emphasize that this non-identiﬁability result is highly general, and does not depend on the speciﬁc choice of evolutionary dynamics P τ , only on the assumption of exchangeability.
The biological intuition behind Thm.
3.2 is that if two sequences are similar to each other and distant from a third, they may be similar either because they are closely related (i.e. the distance τ to the most recent common ancestor is small) or because they are in a local maximum of the ﬁtness landscape.
Without further assumptions, we cannot tell the difference between these two explanations.
The machine learning intuition is that evolution, as described by a JFPM, is in effect a Markov chain
Monte Carlo process whose stationary distribution gives the ﬁtness.
However, the samples we observe may not be fully independent: each pair of samples was initialized from the same point (the most recent common ancestor), and the burn-in since that point may not be sufﬁciently long.
Without independent samples, our estimate of the stationary distribution will be biased.
Fitness inference as hyperparameter inference While general, Thm. 3.2 is not constructive, and does not tell us what the distribution p 0 actually is, or how exactly it differs from p ∞ . Thm. 3.2 also [Table] [Figure]
[Table] [Figure] [Table] +x+NqjW3Lo7B1olXkFqUKA1qH71h5KkMRWGcKy177mJCTKsDCOczir9VNMEkwkeUd9SgWOqg2weeYbOrDJEkVT2CYPm6u+NDMdaT+PQTuYR9bKXi/95fmqi6yBjIkkNFWTxUZRyZCTK70dDpigxfGoJJorZrIiMscLE2JYqtgRv+eRV0mnUvYt64+Gy1rwp6ijDCZzCOXhwBU24gxa0gYCEZ3iFN8c4L86787EYLTnFzjH8gfP5A4PYkWc=</latexit>
[Figure] leaves unclear how much we need to know to learn the ﬁtness landscape: could we infer ﬁtness f if we knew the parametric form of p ∞ , i.e. if we had some model M and knew that p ∞ ∈ M ?
What if we also knew the underlying phylogeny H ?
In the long branch limit (Asm. 2.2), ﬁtness is identiﬁable if H is known; if M is also known, learning ﬁtness is a matter of inferring model parameters.
In the limit where all the branch lengths in the phylogeny are zero, the distribution of observations from a JFPM reduces to X 1 ∼ p ∞ and X {+
+} = X
= X
= . . . .
Here ﬁtness is non-identiﬁable even if H and
M are known; learning ﬁtness is a matter of learning from a single sample.
In the realistic intermediate branch length case, if H and M are known, we will show that learning ﬁtness is essentially a matter of hyperparameter rather than parameter inference.
We demonstrate this last claim in the context of the OUT example, by approximating the OUT model as a Gaussian process latent variable model (GPLVM). We ﬁnd that ﬁtness only appears as a hyperparameter of the derived GP, not as a parameter.
The GPLVM has latent variables Z 1 , Z 2 , . . .
that lie on the hyperbolic plane H , and uses the Gaussian process kernel k ( · , · ) =
exp( − d ( · , · )) , where d ( · , · ) is a distance metric over H .
Let W 1 ( · , · ) be the Wasserstein metric for distributions over inﬁnite matrices, i.e. over R ∞×∞ , using the sup norm on matrices.
Theorem 3.3 (GPLVM approximation of OUT) .
Assume a prior over phylogenies H that is exchangeable in its leaves and where the minimum time between any pair of nodes is greater than η > 0 with probability 1.
Deﬁne the leaf distance matrix ν ij = log( 12 t ij ( H )) .
For any (cid:15) > 0 , there exists a.s. a GPLVM of the form,
[Equation]
where G is a random measure over H , such that W 1 ( p ( ν ) , p (˜ ν )) < (cid:15) ,
where ˜ ν
ij = log( d ( Z i , Z j )) .
If W 1 ( p ( ν ) , p (˜ ν )) = 0 , the OUT and GPLVM produce identical distributions over X 1 , X 2 , . . .
a.e..
The proof is in Appx.
C.2, and uses the embedding of Sarkar [48].
This result says that, by embedding phylogenies H in a metric space, we can approximate an OUT arbitrarily well with a GPLVM; as the
Wasserstein bound gets smaller, the distribution of covariance matrices of the two models get closer.
In the GPLVM, the observations are conditionally independent, X i ⊥⊥ X j | s, G , in line with Thm. 3.2.
The phylogeny H enters the GPLVM only through the latent space embedding Z 1 , Z 2 , . . . .
Learning phylogeny, given the ﬁtness landscape, is thus essentially a matter of inferring latent variables [44, 13].
The ﬁtness landscape enters the GPLVM only through the prior on the Gaussian process (i.e. through µ and σ ).
Inferring ﬁtness given phylogeny is thus essentially a matter of inferring hyperparameters.
This is both good and bad news for ﬁtness inference.
On the one hand, hyperparameters are often learned in practice, and doing so can yield substantially better predictions, so we should be able learn something about µ and σ given data ([58], Chap. 5).
On the other hand, hyperparameters are in general (though not always) non-identiﬁable, and therefore so is ﬁtness [34].
Ho and Ané [22] describe non-identiﬁability conditions for the OUT in particular.
We conclude that even when H and
M are known, ﬁtness inference in JFPMs is fundamentally challenging.
Beneﬁts of misspeciﬁcation
We have demonstrated that a plausible biological mechanism – namely, phylogenetic effects – can produce a data distribution p 0 that does not reﬂect ﬁtness, and can make exact inference of ﬁtness impossible even given inﬁnite data.
Nonetheless, the practical success of ﬁtness estimation methods suggest it is possible to at least approximate ﬁtness landscapes from observational sequence data.
Recall that existing methods proceed by ﬁtting a probabilistic model q θ ∈ M = { q θ : θ ∈ Θ } to data X 1: N , typically via maximum likelihood estimation or approximate Bayesian inference, and then using the predicted log density log q ˆ θ ( x ) as an estimate of the ﬁtness of a sequence x .
Why is this approach empirically successful?
In this section we consider two hypotheses, either of which may hold true in theory.
In Secs.
6-7 we develop and apply tests to evaluate them on real data.
Note that our results in this and the following sections are independent of the speciﬁc evolutionary mechanisms that generate a mismatch between p 0 and p ∞ , i.e. they are not speciﬁc to phylogenetic effects or JFPMs, nor do they even depend on the existence of a stable reproductive ﬁtness function f over evolutionary time.
We can, in fact, redeﬁne p ∞ to be an arbitrary “target distribution”, with log p ∞ proportional to any chosen measure of molecular ﬁtness or function (such as enzyme activity, ﬂuorescence, etc.).
For the sake of concrete illustration, however, we will continue to focus on JFPMs as our primary example of why the data distribution, p 0 , may not equal the target distribution we want to estimate, p ∞ .
We consider two hypotheses for the empirical success of existing ﬁtness estimation methods.
Hypothesis #1 (informal).
Fitness estimation methods succeed by ﬁnding q ˆ θ ≈ p 0 , since for all practical purposes on real data, p 0 = p ∞ .
This hypothesis would make sense, in JFPMs, if Asm.
2.2 held, i.e. branch lengths were long enough in real datasets for P
τ i ( x, x 0 ) to be close to its stationary distribution.
Under this hypothesis, better density estimators have been, and will continue to be, better ﬁtness estimators.
We should focus on developing models M that are well-speciﬁed with respect to the data, i.e. p 0 ∈ M (Fig. 2A).
Hypothesis #2 (informal).
Fitness estimation methods succeed by using models M that are misspeciﬁed with respect to p 0 , i.e. p 0 / ∈ M .
The inferred model q ˆ θ is then closer to p ∞ than p 0 is.
To show this second hypothesis is plausible, we prove that it is guaranteed to hold under general conditions.
We study the projection of p 0 onto M via the Kullback-Leibler (KL) divergence, q θ ∗ =
argmin q θ ∈M KL ( p 0 (cid:107) q θ ) .
The KL projection is relevant because maximum likelihood estimation minimizes the approximate KL divergence between the data and the model, and the posterior in Bayesian inference asymptotically concentrates around the maximum likelihood estimator [38].
We thus expect the ﬁt model q ˆ θ to be close to q θ ∗ , and get closer with N .
Assume that M is “log-convex”, meaning that for any θ, θ (cid:48) ∈ Θ and 0 < r < 1 , there exists some θ (cid:48)(cid:48) such that
[Equation]
; examples of log-convex models include the Potts model, as well as all other exponential family models.
Let TV ( p (cid:107) q ) be the total variation distance between p and q , and let (cid:107) g (cid:107) ∞
=
sup x | g ( x ) | be the uniform (sup) norm of g .
Theorem 4.1 (Beneﬁts of misspeciﬁcation) .
Assume that the model M is log-convex and that q θ ∗ exists and is unique.
If the model is “less misspeciﬁed” with respect to the stationary distribution p ∞ than with respect to the data distribution p 0 , in the sense that
[Equation]
(6) then,
[Equation]
However, if the model is well-speciﬁed with respect to the data distribution, i.e. p 0 ∈ M , we have,
[Equation]
Sec.
C.3 contains the proof, and explains how Thm.
4.1 can be extended with more general conditions (we also emphasize again that the proof does not make any assumption that p 0 and p ∞ follow a
JFPM).
Thm.
4.1 says that if the model is less misspeciﬁed with respect to the target distribution than with respect to the data distribution, then projecting the data distribution onto the model will yield a distribution q θ ∗ closer to the target distribution.
In other words, the best models for ﬁtness estimation are those at a “sweet spot” of complexity, ﬂexible enough to capture p ∞ but not so ﬂexible as to capture p 0 .
To understand the biological intuition behind this result, consider a situation where two neutral mutations with no effect on ﬁtness occur successively at different sites (Fig.
2C).
Due to phylogenetic correlation, there is no observed sequence x ∗ in which the second mutation is present but not the ﬁrst, so an accurate density estimator will ﬁnd p 0 ( x ∗ ) ≈ 0 .
However, if we can guess correctly that the ﬁtness landscape is independent across sites, then ﬁtting a site-wise independent model M will imply the mutation is allowed, q θ ∗ ( x ∗ ) > 0 , correctly inferring p ∞ ( x ∗ ) > 0 .
Under Hypothesis 2, progress in the ﬁeld of ﬁtness estimation has not come from building better density estimators (Hypothesis 1), but rather from an iterative process of (1) hypothesizing, based partly on biophysical knowledge, models that are approximately well-speciﬁed with respect to p ∞ but poorly speciﬁed with respect to the data distribution p 0 , and then (2) comparing their density estimates against experimental ﬁtness measurements.
We will show that on real data, Hypothesiscan often be rejected in favor of Hypothesis 2.
Related Work
Efforts to account for the effects of phylogeny in ﬁtness estimation have a long history [32].
Practical generative sequence models that explicitly account for both epistatic ﬁtness landscapes and phylogeny have long been sought, but stymied primarily by computational challenges [28, 46].
In their place, a variety of non-generative (and often heuristic) methods for correcting for phylogeny have been proposed, including data reweighting schemes [35, 46], data segmentation schemes [10], postinference parameter adjustments [16], covariance matrix denoising methods [42], simulation based statistical testing [45], and more.
In this article, we show that deconvolving ﬁtness and phylogeny is not just computationally hard, but also in general statistically impossible: ﬁtness and phylogeny are non-identiﬁable.
We further show that use of a misspeciﬁed parametric model can on its own (without further corrections) partially adjust for phylogenetic effects.
Our results also intersect with the literature on robust statistics: we can think of the observed data distribution p 0 as a “distorted” version of the true distribution of interest p ∞ .
However, in typical robust inference frameworks (e.g. Huber’s epsilon contamination model), the observed distribution differs from the true distribution by the addition of outliers [24, 52].
Our setup is, in some sense, the opposite: inliers are deleted, as phylogenetic correlations can result in an effective support of p 0 that is smaller than that of p ∞ (Fig.
1CD).
Diagnostic Method
In this section, we develop diagnostic methods to discriminate between Hypothesis 1 and [-Hypothesis
-]Hypothesis 2 (Sec. 4) based on observational sequence data and experimental ﬁtness measurements, and validate these diagnostics in simulation.
Recall that under Hypothesis 2, the estimate q ˆ θ from a parametric ﬁtness model is a better estimate of ﬁtness than the true data density p 0 , while under Hypothesis 1, p 0 is better.
Discriminating these two hypotheses on real data is nontrivial because we do not have access to p 0 .
Ideally, then, a diagnostic test would evaluate the probability that the true density poutperforms q ˆ θ at predicting ﬁtness, taking into account uncertainty in what p 0 could actually be, given the data.
To accomplish this, we compute a posterior over p
using a Bayesian nonparametric sequence model.
In particular, we apply the Bayesian embedded autoregressive (BEAR) model, which can be scaled to terabytes of data and satisﬁes posterior consistency ([2], Thm. 35):
Theorem 6.1 (Summary of BEAR posterior consistency) .
Assume p 0 is subexponential, i.e. for some t > 0 , E X ∼ p
[exp( t | X | )] < ∞ , where | X | is the length of sequence X .
Assume the conditions on the prior detailed in Amin, Weinstein and Marks [2].
If X 1 , X 2 , . . .
∼ p 0 i.i.d., then for M >sufﬁciently large and (cid:15) ∈ (0 , 1 / 2) sufﬁciently small,
[Equation]
in probability, where B ( p, r ) is a Hellinger ball of radius r centered at p , and Π BEAR ( ·| X 1: N ) is the BEAR posterior.
Crucially, this result implies that the BEAR posterior will converge to effectively any value of p 0 , no matter what p 0 is (unlike a parametric model’s posterior).
Moreover, BEAR quantiﬁes uncertainty in its estimates, giving the range of possible values of p 0 that are consistent with the evidence.
We construct our diagnostic test by comparing the ﬁtness estimation performance of q ˆ θ to the range of possible performances of p 0 estimated by BEAR.
Let S f ( p ) be a scalar score evaluating how accurately a density p predicts ﬁtness f .
In practice, S f will be based on experimental and clinical measurements of quantities directly related to ﬁtness.
Diagnostic test (Test Hypothesis 1 vs. Hypothesis 2.)
Hypothesis 1 H 1 :
[Equation]
Hypothesis
H 2 : [Equation] f (
[Equation]
) > [Equation] f ( p 0 ) .
Accept Hypothesis 2 at signiﬁcance level α >
if
[Equation]
[Figure]
Accept Hypothesis signiﬁcance level α if
[Equation]
So long as S f ( p ) is a well-behaved function of p (in particular, so long as S f is continuous in a neighborhood of p 0 with respect to the topology of convergence in total variation), Thm.
6.1 implies that this diagnostic test will be asymptotically consistent, in the sense that it converges to the correct hypothesis in probability.
Simulations We next evaluate the performance of our diagnostic test on simulated data.
We considered two scenarios, the ﬁrst in which Hypothesis 1 holds and the second in which Hypothesis 2 holds.
In both, we let M be a site-wise independent (SWI) model, in which each position of the sequence is drawn independently, i e. X l ∼ Categorical( v l ) for l ∈ { 1 , . . .
, | X |} .
The parameter v l is in the simplex ∆
B , where B + 1 is the alphabet size.
(Further details in Appx.
D.)
In Scenario 1, the true data are generated according to a Potts model and p 0 = p ∞ .
In this scenario, the SWI model is misspeciﬁed, and misspeciﬁcation is bad : using a more ﬂexible model will produce an asymptotically more accurate estimate of p ∞ .
We ﬁnd that our diagnostic test asymptotically correctly accepts
Hypothesis 1, in line with Thm.
6.1 (Figs.
3A and 7A).
In Scenario 2, the true data are generated according to a JFPM with ﬁnite branch lengths, and p ∞ ∈ M while p 0 / ∈ M .
The mutational dynamics P τ follow the Sella and Hirsh [49] process.
The phylogeny
H is drawn from a Kingman coalescent.
In this scenario, the SWI model is again misspeciﬁed, but misspeciﬁcation is good : while the nonparametric BEAR model can achieve better density estimates than the SWI model (Fig.
3C), the SWI model outperforms BEAR at ﬁtness estimation (Figs.
3D and 8).
We ﬁnd that our diagnostic test correctly accepts Hypothesis 2 (Figs.
3B and 7B).
A possible point of concern is that the test is poorly calibrated from a frequentist perspective, and in the low N regime accepts Hypothesis 2 in Scenario 1 more than {+
+} α % of the time when the data is resampled from p 0 (Fig. 9A).
This behavior is common in nonparametric Bayesian tests, and not necessarily a problem: the test is still valid from a purely Bayesian perspective.
Nevertheless, on real data we will check that we are close to the large N regime by (1) checking that the BEAR posterior predictive is at least as close to p 0 as q ˆ θ is (as measured by perplexity on held out data;
Figs.
3C and 9B) and (2) examining the plot of the BEAR posterior over S f ( p ) as a function of N (as in Fig.
3AB), to check that it has converged.
Empirical Results
We now evaluate whether existing ﬁtness estimation methods outperform the true data density p 0 , i.e. whether we can reject Hypothesis 1 in favor of Hypothesis 2 on real data.
Tasks We consider two key prediction tasks where ﬁtness models are applied in practice.
The ﬁrst task is to predict whether variants of a protein are functional, according to an experimental assay of protein function; the metric S f ( · ) is the Spearman correlation between p ( x ) and the assay result [23].
There are typically ∼ 1000s of measurements per assay.
The second task is to predict whether a variant of a protein observed in humans causes disease, according to clinical annotations; the metric
S f ( · ) is the area under the ROC curve when p ( x ) is used to predict whether or not a variant is [Figure] pathogenic [19].
There are typically only a handful of labels for each gene.
For the ﬁrst task, we considered 37 different assays across 32 different protein families, and for the second task, 97 genes across 87 protein families; for each protein family, we assembled datasets of evolutionarily related sequences, following previous work.
Note that across the 37 assays and 97 genes, the data used for
S f comes from different experiments and different clinical evidence, often collected by different laboratories or doctors.
Thus, our overall conclusions should be robust to the choice of S f .
Models We considered three existing ﬁtness estimation models: a site-wise independent model (SWI), a Bayesian variational autoencoder (EVE [19], which is similar to DeepSequence [44]), and a deep autoregressive model (Wavenet) [50].
Note that SWI and EVE, unlike Wavenet, require aligned sequences as training data.
Details in Appx.
E.
Results Applied to the ﬁrst prediction task, our diagnostic test accepts Hypothesis 2 at signiﬁcance level α = 0 .
025 in 35/37 assays (95%) for SWI, 33/37 assays (89%) for EVE, and 36/37 assays (97%) for Wavenet (Fig. 4A).
Applied to the second prediction task, our diagnostic test accepts Hypothesis 2 at signiﬁcance level α = 0 .
025 in 31/97 genes (32%) for SWI and 46/97 genes (47%) for EVE (Fig.
4B).
Thus, ﬁtness estimation models are capable of outperforming the true data distribution p 0 .
We found evidence for Hypothesis 1 in only a handful of examples: on the ﬁrst task, Hypothesis 1 was accepted at signiﬁcance level α = 0 .
025 in 0/37 assays for SWI, 3/37 assays (8%) for EVE, and 0/37 assays for Wavenet, while on the second task, Hypothesis 1 was accepted for 5/97 genes (5%) for SWI and 4/97 genes (4%) for EVE.
We conﬁrmed that the diagnostic test was in the large N regime:
BEAR outperformed Wavenet at density estimation, providing better predictive performance on 27/37 assays (73%) and similar performance on the remaining 10 assays (Fig. 10).
(Note that we cannot do this comparison for SWI or EVE since they are alignment-based [57].)
Example plots of the BEAR posterior’s convergence with N on the ﬁrst prediction task showed convergence to values of S f well below that for parametric ﬁtness estimation models (Figs.
4C and 11-12).
Overall, we conclude that there is strong evidence that existing ﬁtness estimation methods reliably outperform the true data distribution p 0 across a range of datasets and tasks.
To study the tradeoffs between density estimation and ﬁtness estimation in more depth, we smoothly and nonparametrically relaxed a parametric autoregressive (AR) model (Appx. E.4).
We embedded the AR model (a convolutional neural network) into a BEAR model, and ﬁt the BEAR model with empirical Bayes.
We found evidence that the AR model was misspeciﬁed on every dataset, following the methodology of Amin, Weinstein and Marks [2]: the optimal h selected by empirical Bayes was on the order of 1 − 10 in each dataset.
Now, in the limit as the hyperparameter h → 0 , the
BEAR model collapses to its embedded AR model; so by scanning h from low to high values we can interpolate between the parametric and nonparametric regime.
We ﬁnd a smooth tradeoff between
S f ( p ) and the likelihood of the data under the BEAR model, with higher h corresponding to better density estimation but worse ﬁtness estimation (Fig.
4EF and 13).
This relationship held across many datasets: the diagnostic test, evaluated against the AR model (the h → 0 limit), accepts Hypothesisin 28/37 assays (76%), but Hypothesis 1 in only 6/37 (16%) (Fig. 14).
These results conﬁrm that making a model well-speciﬁed (relaxing from a parametric to a nonparametric model) can bring improved density estimation at the cost of worse ﬁtness estimation.
Discussion
In this article, we have argued that better density estimation does not necessarily lead to better ﬁtness estimation.
Further, we estimate with high probability that existing ﬁtness estimation methods systematically outperform the true training data density.
Although existing methods rely on ﬂexible, high-parameter deep neural network models, they can nonetheless be misspeciﬁed; but this misspeciﬁcation acts as a blessing, rather than a curse for ﬁtness estimation.
Successful models are at a sweet spot of complexity, ﬂexible enough to capture the target ﬁtness distribution well but not so ﬂexible as to match the data distribution itself.
We have focused on state-of-the-art ﬁtness estimation methods which are trained on data from individual protein families [44, 50, 19].
Recently, large-scale generative sequence models (“protein language models”) trained on more diverse datasets (containing proteins from many different families) have show ﬁtness estimation performance comparable to, and in some settings surpassing, single family models [36, 40, 41].
Although applying our diagnostic test to these datasets requires further work, there is no reason to expect the same limitations of density estimation do not hold for such models. Indeed, following a preprint of this paper, Nijkamp et al. [40] presented evidence of the beneﬁts of misspeciﬁcation in a protein language model: past a certain number of parameters, density estimation improved while ﬁtness estimation deteriorated. See Appx.
G for further discussion.
One future direction is to further explore models M that are less ﬂexible than existing models and worse at density estimation, since they can increase the gap between KL ( q θ ∗ (cid:107) p ∞ ) and KL ( p 0 (cid:107) p ∞ )[-
-]
(Thm. 4.1).
There may also be opportunity to improve model geometry: while exponential family models are guaranteed to be log-convex (and thus can satisfy Thm. 4.1), we have no such guarantee for variational autoencoders or other neural network methods.
Meanwhile, uncertainty quantiﬁcation is crucial for applications such as those in clinical genetics, but challenging in misspeciﬁed models [53, 39, 26].
Alternatively, it might be useful to abandon the strategy of using misspeciﬁed models for ﬁtness estimation altogether, and instead construct JFPM models where the ﬁtness landscape is represented explicitly as a latent variable.
Recent progress on amortized variational inference for phylogenetic models is promising for building ﬂexible and scalable JFPMs [55].
However, handling non-identiﬁability is challenging, and may require new assumptions and/or new methods of sensitivity analysis to infer the full set of ﬁtness landscapes consistent with the data [9].
Although this article has focused on technological applications of ﬁtness models in solving prediction problems, ﬁtness models also have implications for our fundamental understanding of evolution.
Pure phylogeny models and pure ﬁtness models present very different pictures of the past history of life: in PMs, similarities and differences among genetic sequences are determined primarily by history and ancestry (Asm. 2.1), while in FMs they are primarily determined by functional constraints (Asm. 2.2).
PMs and FMs also present very different implications for the future of life: in PMs, the diversity of sequences seen in nature will likely expand dramatically going forward, while in FMs, the landscape of functional sequences has already been well-explored.
Our results emphasize that where and to what extent each model offers an accurate picture of reality remains an open question. Acknowledgements
We thank members of the Marks lab, Pascal Notin, Ali Madani, John Ingraham and the anonymous reviewers for insights and suggestions.
E.N.W.’s work was supported by the Fannie and John Hertz
Foundation.
D.S.M. is supported by the Chan Zuckerberg Initiative.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See e.g. Sec.
6 on calibration and Appx.
G on evaluating very large models.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Appx.
A (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Secs. 2 - 6 and Appx.
C. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appx.
C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material for code and instructions, and see Appx.
E for data.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appx.
E (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Fig.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appx.
E.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Appx.
E. (b) Did you mention the license of the assets?
[Yes] See Appx.
E (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Appx.
E (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] See Appx.
E 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Societal Impact Statement
Fitness estimation methods have wide potential for positive impact, including in understanding and diagnosing disease [19] and designing novel proteins that can accelerate research or treat disease [50].
However, like most powerful technologies, ﬁtness estimation methods also have potential for negative impact.
Improved ability to diagnose genetic disease could enable discrimination on the basis of genetic mutations, for instance insurance companies could refuse coverage or charge high premiums to patients with pathogenic variants.
These risks can be mitigated via appropriate laws and regulations, such as the Genetic Information Nondiscrimination Act of 2008 in the US.
Synthetic biology and molecular design are dual use technologies, and novel proteins could be designed that cause harm.
For a further discussion of dual use concerns for machine learning-based molecular design, and recommendations for mitigation, see Urbina et al.
B Evolutionary dynamics models
Application of the Sella and Hirsh [49] model (Eqn. 1) in JFPMs rests on a number of assumptions; we brieﬂy the most relevant here.
When applying Eqn. 1 to amino acid sequences, as is typical for ﬁtness estimation models, we ignore biases that come from the genetic code, which can modify the steady state probability of amino acids (in the absence of ﬁtness effects) away from a uniform distribution.
This is justiﬁed practically by the small effect sizes: if at steady state an amino acid has probability 1 / 64 instead of 1 / 20 , the total difference in log probability is log(1 / 20) − log(1 / 64)
≈ 1 , which is small compared to (for instance) the log probability differences relevant for disease risk prediction with ﬁtness models, which are ≈ 10 ([19], Extended data Fig. 3).
Moreover, this bias only contributes an overall shift in amino acid probabilities, independent of position, and so does not change our main theoretical results.
We ignore biases caused by asymmetric mutation rates for analogous reasons (though note they are often included in PMs in practice) [49].
The constant β depends on the effective population size, as well as the underlying population genetics model (Moran or Wright) and organismal ploidy ([49], Table 1).
Following standard practice, we treat β as ﬁxed for simplicity, though in reality it may vary over time and across lineages.
Taking into account these possible changes clearly would not contradict our main theoretical result, that ﬁtness and phylogeny are non-identiﬁable.
C Proofs C.1 Proof of Proposition 2.
N.b.
this result is known in the literature (see e.g. [22], Eqn. 1) but we are unaware of a proof, so we provide one here for completeness.
Proof.
For notational convenience, we will work with a standardized OUT, with µ = 0 and σ = 1 .
The ﬁnal result can be obtained by translating and scaling the distribution of leaves.
The transition distribution from point x (cid:48) at time t (cid:48) to point X at time t under the Ornstein-Uhlenbeck (OU) process is
[Equation]
This distribution can be reparameterized in location-scale form as
[Equation]
As t → ∞ we reach the stationary distribution Normal(0 , 1) .
Let b ∈ { 1 , ..., B } index the branches of the tree, let λ b be the length of branch b , and let j ∈ { 1 , ..., N } index the leaves (observed species or sequences); see Fig.
We have assumed that the most recent common ancestor of the observed sequences was sampled from p ∞ ; this can be represented by adding a single branch length (indexed b = 1 ) to the root with length λ 1 = ∞ .
Let (cid:15) b be the noise describing the OU diffusion over each branch.
Let ξ j,b be the total time from leaf j to the nearest vertex on branch b , so long as branch b is on the path from leaf j to the root; otherwise, set ξ j,b = ∞ .
For instance, in the diagram in Figure 5, we have ξ 1 , 4 = 0 , ξ 1 , {+
+} = λ 4 , ξ 1 ,
= λ
+ λ 2 , and ξ 1 , 5 = ξ 1 , 6 = ξ 1 , 7 = ξ 1 , 3 = ∞ .
We can now write the leaf position as
[Equation]
[Equation]
Deﬁne the matrix
[Equation]
such that X j = (cid:80) b M j,b (cid:15) b .
We can now describe the complete leaf distribution as
[Equation]
where I B is the B -dimensional identity matrix.
Thus, according to the location-scale representation of the multivariate normal,
[Equation]
[Equation]
We can simplify the covariance matrix Σ := MM (cid:62) .
First
[Equation]
Before introducing the notation required to derive the general result, it’s helpful to get a sense of how the derivation works; in the example tree (Figure 5),
[Equation]
The sum over b telescopes, leaving only the initial term, which corresponds to the total time between leaf node 1 and leaf node 2 .
To construct the general result, deﬁne ˜
b j,j (cid:48) as the branch whose later node is the most recent common ancestor of leaves j and j (cid:48) .
In the example in Figure 6, ˜ b 2 , 4 = 4 .
Let R be an ordered list of branches from ˜ b
j,j (cid:48) to b = 1 , the earliest branch.
In the example in
Figure 6, R = [4 , 2 , 1] .
Finally, let t jj (cid:48) be the length of the shortest path from leaf j to leaf j (cid:48) , the time from the most recent common ancestor to j plus the time to j (cid:48) .
In the example in Figure 6, t 2 , 4 =
λ
+ λ 6 + λ 8 .
We now have
[Equation]
Breaking down the telescoping sum, and using the fact that the ﬁnal element of R is t 1 = ∞ ,
[Equation]
So we have the simple result that the covariance matrix depends just on the divergence times between leaves,
[Equation]
Translating the distribution Eqn.
14 by µ and scaling by σ yields the result.
C.2 Proof of Theorem 3.
Before proving the result, we brieﬂy clarify a deﬁnition in the statement of the theorem:
Deﬁnition C.
(Exchangeable in leaves) .
Let H be a tree with countably inﬁnite leaves and let H π be a permutation of a phylogeny in its leaves, i.e. the same tree H with the leaves observed in a different order, according to a permutation π .
A distribution over phylogenies is exchangeable in its leaves if p ( H ) =
p ( H π ) for any permutation π .
Proof.
Outline: First, using the results from Sarkar [48], we construct an embedding for each tree into the hyperbolic plane, being careful that the embedding preserves exchangeability.
Second, we apply de Finetti’s Theorem to obtain the conditionally independent representation of the joint distribution of Z 1 , Z 2 , ... .
Third, we use the distortion bound from Sarkar [48] to bound the Wasserstein distance between p ( ν ) and p (˜ ν ) .
First we describe the Sarkar [48] (1 + (cid:15) ) distortion embedding algorithm setup.
Vertices in phylogenetic trees have maximum degree three, and, by assumption, the minimum edge length in a tree H is greater than η > 0 with probability one.
For any (cid:15) [Equation] > 0 , choose a ρ < π/ 3 and a scale factor
[Equation]
where k is the Gaussian curvature of the hyperbolic plane H (for most hyperbolic geometry models, and in particular the Lorentz manifold, k = − 1 ).
Then, let h 1 ( H ) , h 2 ( H ) , ... be the position of the leaves in the embedding of H produced by the (1 + (cid:15) ) distortion embedding algorithm in Sarkar [48], using edge scale factor λ , and ρ separated cones with cone angle 2 π/ 3 − 2 ρ .
Taking the last line of the proof of Theorem 6 in Sarkar [48], we are guaranteed that even for a countably inﬁnite number of leaves,
[Equation]
where i, i (cid:48) ∈
N := { 1 , 2 , . . .
} , and ˜ d ( · , · ) is the hyperbolic distance function.
Next we will modify the embedding function h to ensure that the distribution of embedded leaves is exchangeable.
Let [ H ] be the set of phylogenetic trees that are equivalent to H up to reordering of the vertices.
For each equivalence class [ H ] we choose one ordering of the vertices to be the canonical tree ˆ H ([ H ]) , and for any tree H let π c ( H ) be the leaf permutation such that the reordered tree H π c
( H )
=
ˆ H ([ H ]) .
Now deﬁne the modiﬁed leaf embedding function h (cid:48)
( H ) :
= h π ( H ) ( H π c ( H ) ) where π ( H ) is the inverse permutation of π c ( H ) .
Since by assumption the prior p ( H ) on the phylogenetic tree is exchangeable, we can rewrite p ( H ) using the induced distribution over equivalence classes p ([ H ]) as
[Equation]
where Permutation is the uniform distribution over all permutations of N := { 1 , 2 , . . .
We now deﬁne the distribution over leaf embeddings as
[Equation]
which we can rewrite as
[Equation]
The distribution p ( Z 1 , Z 2 , ... ) is therefore exchangeable.
Applying de Finetti’s Theorem [29] we have a.s.
[Equation]
19) where G is a random measure distributed according to a prior G .
Moreover, the embedding distortion bounds (Eqn. 17) are preserved for each H , since
[Equation]
and by the same logic
[Equation]
We will now construct the Wasserstein bound.
Deﬁne the joint distribution over ν and ˜ ν ,
[Equation]
22) where we have chosen d ( · , · ) = 12 λ ˜ d ( · , · ) .
Note that the marginal distribution of ν matches its deﬁnition in the statement of the theorem [Equation] and that, applying Eqn. 18 and Eqn. 19, the marginal distribution of ˜ ν also matches its deﬁnition.
Using the fact that log is a monotonically increasing function, Eqn. 20 gives
[Equation]
and similarly using the bound from Eqn. 21, sup i,i
[Equation]
[˜ ν
i,i
[Equation]
( H ) − ν
i,i [Equation]
( H )] ≤ 0 .
Thus, with probability 1 under p ( H ) ,
[Equation]
Recall that the Wasserstein distance between the distribution of two random variables ν and ˜ ν can be written as
[Equation]
where J is the set of joint distributions with marginals corresponding to the distributions of ν and ˜ ν ([15], Chap. 11.8).
Using the joint distribution in Eqn. 22, the Wasserstein distance is bounded by
[Equation]
Now consider the case where
W 1 ( p ( ν ) , p (˜ ν )) = 0 .
(N.b. in this case, we do not need to assume that the minimum time between nodes in H is greater than η > 0 .)
Since the Wasserstein metric is a metric on the space of probability distributions ([15] Lemma 11.8.3), p ( ν ) = p (˜ ν ) a.e.. Using the standard properties of Gaussian processes ([58], Chap. 2), the GPLVM model (Eqn. 5) can be written as
[Equation]
which is equivalent to the OUT distribution,
[Equation]
So the distribution p ( X 1: ∞ ) produced by the GPLVM is equivalent to the distribution p ( X (cid:48) 1: ∞ ) produced by the OUT model a.e..
C.3 Proof of Theorem 4.
We start with a more basic result that captures the intuition behind Thm. 4.1, and then prove a more general result, from which Thm. 4.1 can be derived as a corollary.
In particular, we start by examining the special case where the model is well-speciﬁed with respect to p ∞ .
Proposition C.2.
Assume that the model M is log-convex and well-speciﬁed with respect to the stationary distribution, i.e. p ∞ ∈ M .
Assume q θ ∗ exists and is unique.
Then, if the model is misspeciﬁed with respect to the data distribution, i.e. p 0 / ∈ M , we have
[Equation]
But if the model is well-speciﬁed, i.e. p 0 ∈ M , we have
[Equation]
Proof.
The ﬁrst inequality in Eqn.
26 follows from the fact that M is misspeciﬁed with respect to pand so KL ( p 0 (cid:107) q θ ∗ ) > 0 .
The second inequality follows from Thm.
1 from Csiszar and Matus [11], who study the geometry of reverse I-projections.
For Eqn. 27, note that q θ ∗ =
p 0 when p 0 ∈ M .
We next extend Prop. C.2 to the case where the model may be misspeciﬁed with respect to p ∞ as well as p 0 .
Proposition C.3.
Assume the model M is log-convex and that q θ ∗ exists and is unique.
If
[Equation]
then KL ( q θ ∗ (cid:107) p ∞ ) <
KL ( p 0 (cid:107) p [Equation]
Proof.
Deﬁne the projected stationary distribution
[Equation]
Now, from the deﬁnition of the KL divergence we have
[Equation]
Applying Thm.1 from Csiszar and Matus [11] to KL ( p 0 (cid:107) p ∞∗ ) we have
[Equation]
Applying Eqn.
28, the result follows.
One way of satisfying Prop.
C.3 is for there to exist a p ∞ ∗ ∈ M that is very close to p ∞ , in the sense that log p ∞ ( x ) /p
∞∗ ( x ) ≈ 0 in areas of X with high probability under both p 0 and q θ ∗ .
Finally we, derive the more interpretable (but also more restrictive) conditions in Thm. 4.1.
Proof of Thm.
4.1 For Eqn. 8, note that q θ ∗ =
p 0 when p 0 ∈ M .
To show Eqn. 7, we will show that the conditions of Prop.
C.3 are met.
(N.b.
we will work with sums over x since we are concerned with discrete sequences, but the same derivation holds replacing sums with integrals.)
We have
[Equation]
Applying Eqn. 6 to the ﬁrst term and the deﬁnition of total variation distance to the second term
[Equation]
where the second inequality is Pinsker’s inequality.
We see that Eqn.
28 is satisﬁed, and the result follows.
D Simulation Details
In both scenarios, we generated sequences of ﬁxed length | X | = 30 , with an alphabet size of
B + 1 = 4 (corresponding to nucleotides).
Scenario 1 We simulated from a Potts model
[Equation]
l (cid:48)
[Equation]
l (cid:88) b (cid:88) [Equation]
(cid:48) [Equation]
ll (cid:48) bb (cid:48)
x lb x lb (cid:48) [Equation] where h is the sitewise energies, e is the pairwise energies, x is a one-hot sequence encoding, l indexes sequence positions and b indexes letters.
Following the simulations in Ingraham and Marks [27], which were intended to roughly match the statistics of typical real protein Potts models, we drew h lb ∼ InvGamma(2 , 0 .
8) and
[Equation]
The energies h and e were drawn once, and the same values used across independent simulations.
We sampled from the model using a Gibbs sampler with 100 steps of burn-in and 10 parallel chains using the code from Ingraham and Marks [27] ( https://github.com/debbiemarkslab/ persistent-vi ).
We shufﬂed the resulting samples to remove autocorrelation.
Scenario
We used a site-wise independent ﬁtness function:
[Equation]
with site-wise residue biases h l , where x l is a one-hot encoding of the letter at the l -th position of x .
To generate phylogenetically correlated sequences, we sampled phylogenetic trees from a Kingman
Coalescent ([3], Def. 2.1) with rate 1.
Starting from a random sequence drawn from the steady state distribution at the root, we evolved the sequence simulating a Wright process in a haploid population ([49], Eqn. 3) according to the tree and ﬁtness function.
In particular, for sequences x 0 , x that are one mutation away, the mutation rate is
[Equation]
where we set the effective population size to N eff = 10000 .
This stochastic process has steady state
[Equation]
([49], Eqn. 7).
SWI model We ﬁt the SWI model with maximum likelihood estimation.
BEAR model
In these simulations, we used a vanilla BEAR model with a uniform embedded AR model (i.e. a Bayesian Markov model) for simplicity.
We set the Dirichlet prior concentration to the constant α = 0 .
Based on the theoretical analysis in Amin, Weinstein and Marks [2] (Thm. 35), we used a prior on lags of the form
[Equation] [Equation]
where B is the alphabet size (4 for nucleotides).
We inferred the prior via empirical Bayes, marginalizing over the transition probabilities following the protocol in [2].
Conditional on lag L , sampling from the posterior over the BEAR model is straightforward thanks to Dirichlet-Categorical conjugancy.
Evaluation We deﬁned S f following standard protocols for ﬁtness estimation models.
In particular, we let S f ( p ) be the Spearman correlation between p ( x ) and f ( x ) for x ∈ Λ where Λ consists of all possible single point mutations (i.e. single letter changes) of an initial (“wild-type”) sequence.
The wild-type sequence was chosen as the most likely sequence under p ∞ , computed exactly for Scenario 2 and estimated based on the 10 6 samples for Scenario 1.
To estimate model perplexity (Fig.
3C and 9B), we used N = 10 , 000 independent sequences from p 0 and computed the per-residue perplexity
[Equation]
where | X n | is the sequence length and p ( X n ) is the probability of the sequence under the model.
To estimate the KL to the ﬁtness distribution in Scenario 2 (Fig. 3D) , we sampled N = 10 ,independent sequences from p ∞ , { X 1 , . . .
, X N } and estimated
[Equation]
where H ( p ∞ ) is the entropy of p ∞ , which can be computed analytically.
For BEAR, we plotted the
KL to the posterior predictive, which, using Jensen’s inequality can also be seen to lower bound
[Equation]
where Π BEAR ( p | X train ) is the BEAR posterior learned from the training dataset.
[Figure] [Figure]
F r ac t
i o n o f t e s t s w i t h
H
y p
o t
h e s i sacce p t e d
A
[Figure]
E Empirical Results Details E.1 Data
Prediction task #1 (functional effect)
Following standard practice, we report the absolute value of the Spearman correlation as S f ( p ) , since in some assays a negative change in the measured quantity corresponds to larger ﬁtness (note that in all cases the predicted directionality of the effect under each model was correct).
We focused on single amino acid substitutions, taking only those for which EVE was able to make a prediction (EVE is limited by its reliance on a multiple sequence alignment).
We used the same data as in Shin et al.
[50], Table 1, taking the 37 experiments performed on the following 32 proteins: UBC9_HUMAN, UBE4B_MOUSE, P84126_THETH, HIS7_YEAST, BLAT_ECOLX,
IF1_ECOLI, PTEN_HUMAN, B3VI55_LIPST, GAL4_YEAST, POLG_HCVJF, PABP_YEAST,
CALM1_HUMAN, AMIE_PSEAE, TRPC_THEMA, RASH_HUMAN, YAP1_HUMAN,
TRPC_SULSO, DLG4_RAT, BG_STRSQ, KKA2_KLEPN, HSP82_YEAST, B3VI55_LIPST (stabilized), MK01_HUMAN, HIV BF520 env, SUMO1_HUMAN, RL401_YEAST, PA_FLU,
HG_FLU, TPMT_HUMAN, HIV BG505 env, TPK1_HUMAN, and MTH3_HAEAE (stabilized).
The sequence data and assay data are publicly available for research use.
Prediction task #2 (pathogenicity)
We used the pathogenicity labels of single amino acid substitutions curated from ClinVar [31] in Frazer et al.
Note ClinVar data is freely available for public use, and labels only depend indirectly on patient data; there is no per-person label, and no way of identifying individual patients [31].
We considered labels for 87 human proteins less than 250 amino acids in length: AICDA, AQP2, ATPF2, B9D2, CAH5A, CAV3, CD40L, CF410, CHC10, CIA30,
CLD16, CLN8, COQ4, CRBB2, CRGD, CTRC, CXB1, CXB2, CXB3, CXB4, CXB6, CY24A,
DERM, DGUOK, DHDDS, EDAD, EFTS, ELNE, ETFB, ETHE1, EXOS3, FGF10, FGF23, FOXE3,
FRDA, GP1BB, HBB, HEM4, HSPB1, HSPB8, IFM5, IFT27, JAGN1, KAD2, KCNE1, KCNE2,
KITM, LITAF, MMAB, MMAC, MPU1, MYPR, NDP, NDUS8, NFU1, NKX25, NMNA1, OPA3,
PAHX, PDYN, PMM2, PMP22, PNPH, PNPO, PROP1, PSPC, PTPS, RASH, RNH2A, S5A2, SAP3,
SBDS, SCO1, SDHB, SDHF2, SIX1, SIX3, SOMA, TMM70, TNNT2, TPK1, TPM2, TR13B,
TWST1, VHL, XLRS1, ZC4H2.
Training data All models were trained on datasets of protein sequences gathered as described in [50] for pathogenicity effect prediction tasks and as described in [19] for functional effect prediction tasks.
SWI and EVE were trained on the multiple sequence alignment, while Wavenet and BEAR were trained on the raw sequences as described in [50].
All datasets were uniformly subsampled to produce a 75%/25% train/test split.
E.2 Models and code
The SWI model was trained via maximum likelihood.
The Wavenet model was trained via maximum likelihood with the default architecture, hyperparameters and training protocol described in [50], for 100,000 steps.
Code is from https: //github.com/debbiemarkslab/ SeqDesign published under the MIT licence.
We did not apply the Wavenet model to the second prediction task, as it has only previously been developed for the ﬁrst task.
The EVE model was trained via variational inference, using the same architecture, hyperparameters, and training protocol described in [19].
Code is from https://github.com/debbiemarkslab/
EVE published under the MIT licence.
To match the protocol of the original paper, EVE was – unlike
SWI, Wavenet and BEAR – (a) trained on the full dataset rather than the training set alone, and (b) used a sequence reweighting heuristic.
The BEAR model used an embedded convolutional neural network (the same architecture as used in [2], with layer 1 width of 16, ﬁlter width of 5 and 30 ﬁlters total) and a uniform prior over lags 2, 3, 5, 7, and 9.
Code is from https://github.com/debbiemarkslab/BEAR published under the MIT licence.
The model was trained using empirical Bayes, as described in [2], for 500 steps with a batch size of 500000 kmers.
Two Nvidia TeslaV100s GPUs were used, on an internal cluster; training took up to 8 hours.
To construct posterior credible intervals, we used 41 samples from the posterior for prediction task #1, and 1000 samples for prediction task #2.
[Figure]
To train and predict using BEAR one needs to transform sequences into de-Bruijn graphs.
To do so for amino acid sequences, we used code from https://github.com/jdisset/kmap .
We communicated with the author of this code to obtain permission for its use.
We computed the heldout perplexity (Eqn. 30) for the BEAR posterior predictive and for Wavenet to produce Fig. 10.
E.3 Convergence experiments
To plot the convergence of the posterior over p 0 as a function of N (Fig.
4CD, 11 and 12), we used a vanilla BEAR model, a nonparametric Bayesian Markov model.
Note that here we ﬁxed the embedded AR model, rather than reﬁtting with larger N , so that we could analyze the the convergence behavior with reference to the asymptotic results of Thm. 35 in [2], which does not take into account empirical Bayes.
We set the Dirichlet concentration to 10 and used a prior over lags as in Eqn. 29.
E.4 Interpolation experiments
We ﬁt a BEAR model using the architecture and training protocol described in Sec.
E.2, optimizing both the parameters of the AR model and h via empirical Bayes.
We then varied h from its optimized value, and recalculated the total marginal likelihood and the posterior distribution over S f ( p ) (Fig. 4EF and 13).
We also computed the value of S f ( q ˆ θ ) for the ﬁt BEAR model in the h → 0 limit (Fig. 14).
[Figure]
[Figure]
F Supplementary code
The supplementary code provides a Jupyter notebook ( example.ipynb ) illustrating the application of our BEAR diagnostic test on simulated data.
It is available under an MIT License.
G Discussion of Protein Language Models
In this section we discuss the relevance and relationship of our results to large-scale “protein language models” such as ESM-1v [36], MSA Transformer [43], UniRep [1], Tranception [41], ProGen [33],
ProGen2 [40] and others. Note the term “protein language model” is something of a misnomer; these methods are far from unique in applying and extending ideas from natural language processing (NLP) to build generative protein sequence models (Wavenet [50] and BEAR [2] being just two other examples).
Rather, inspired by the recent success of large-scale “foundation models” in NLP, protein language models have two key distinguishing properties: (1) they use neural network architectures with very large numbers of parameters and (2) they are trained on very large and very diverse datasets, not just sequences from a single family [4].
Our results suggest that better density estimation for sequence distributions does not necessarily imply better ﬁtness estimation, and that a carefully chosen misspeciﬁed model can systematically outperform well-speciﬁed models.
We believe our results complicate but do not contradict lessons
[Figure]
learned on the importance of scale in natural and protein language models.
Big and Diverse Data Protein language models are trained on datasets that are bigger than those used for single family models, not only in the sense of containing more sequences but also in the sense of being more diverse.
In other words, they differ not only in N but in the data generating distribution p 0 .
The idea that using such big and diverse datasets may lead to better ﬁtness estimation is entirely in line with our theory.
First, given a particular model class M and p 0 , we expect datasets with larger N to be better, as it brings our estimated distribution q ˆ θ away from the prior and closer to the projected distribution q θ ∗ .
Second, while increasing the diversity of p 0 does not remove the non-identiﬁability problem, it can enable practical strategies for reducing the effects of phylogeny.
Misspeciﬁed models ﬁt to many different protein families may be more robust to the “noise” introduced by phylogeny, as they must be accurate across families with very different phylogenetic structure.
Moreover, protein language models are often trained on datasets that have been thinned, with very similar sequences removed (e.g. UniRef90 [36] rather than UniRef100 [44]); this shifts the target distribution p 0 , and can potentially ameliorate the effects of phylogeny.
Downstream Tasks
The problem of ﬁtness estimation using generative sequence models can be thought of as solving a downstream task of predicting an external label by learning an unsupervised one-dimensional representation for each sequence (namely, its log likelihood under the model).
In general, for problems that involve using unsupervised representations for downstream tasks, progress often comes from directly optimizing the hyperparameters of the unsupervised model to improve downstream task performance, without regard to whether such hyperparameter changes increase or decrease density estimation performance of the unsupervised model.
Our results suggest that focusing on optimizing hyperparameters and architectures for downstream performance instead of density estimation, perhaps by using meta-learning, may also be a successful way forward for ﬁtness estimation [37].
Our results are most directly in tension with recommendations from NLP for model size: we show theoretically and empirically that in the large N limit, misspeciﬁed models can be more accurate ﬁtness estimators than well-speciﬁed models, which can conﬂict with the standard recommendation in large-scale natural and protein language models to give models as many parameters as possible. Our prediction is born out in the recent results of Nijkamp et al.
[40], which demonstrate that a protein language model with more parameters and better density estimation can be worse at ﬁtness estimation. Still, a systematic understanding of model size and its tradeoffs in protein language models is lacking. For example, it may be advantageous to use misspeciﬁed models which have a very large numbers of parameters, as high-parameter models can possess other desirable properties beyond expressiveness, such as smoothness [7].
Indeed, many modern neural architectures (including attention layers in transformers) enforce invariance to certain symmetries (such as permutation or translation), ensuring that they are not universal approximators even when they have very large numbers of parameters [5].
Regularization and early stopping can also effectively restrict the model class M .
Broadly, our results ﬁt with the idea that neural architecture and hyperparameter choice can substantially impact downstream tasks even in an age of massive models and data.
The question of whether protein language models can systematically outperform the true data distribution p 0 at ﬁtness estimation remains an open problem for future work.
Here we describe some of the difﬁculties in addressing the problem and present preliminary results.
The ﬁrst challenge is scaling the diagnostic test to massive datasets.
It is crucial, in particular, to work with datasets with large enough N that the test has power to reject the null hypothesis; convergence of the BEAR diagnostic is expected to be slower with (roughly speaking) high diversity p
(Thm. 35, [2]).
Computationally, scaling the test to massive datasets will require the use of large scale amino acid kmer counters, which are currently less performant than nucleotide kmer counters [30].
A second challenge is evaluating the density estimation performance of protein language models.
Models trained using masked language modeling objectives (such as ESM-1v [36] and MSA Transformer [43]) do not specify a probability distribution over full sequences, i.e. they do not estimate a well-deﬁned density [20].
Moreover, in some cases ultra-large scale models and/or training procedures are kept private or semi-private, making it difﬁcult to understand what exactly was in the training and test sets [33, 6].
Nevertheless, we can rule out the idea that protein language models all achieve excellent ﬁtness estimation and excellent density estimation on single family datasets.
We studied UniRep, an LSTM-based protein language model trained on a diverse evolutionary dataset (UniRef50), since UniRep provides a well-deﬁned density over sequences (unlike ESM-1v and MSA Transformer) and is publicly available (unlike ProGen)
We evaluated UniRep on the experimental assay prediction task, comparing to the BEAR posterior predictive distribution and to Wavenet, both of which were inferred from single family datasets (as described in Sec. 7).
If UniRep were to show similar or better perplexity to the BEAR posterior predictive, and similar or better ﬁtness estimation performance to Wavenet, it could potentially represent a counterexample to our theory on the beneﬁts of misspeciﬁcation.
Instead, we ﬁnd that UniRep has substantially worse perplexity as compared to the BEAR posterior predictive (Fig. 15) and similar ﬁtness estimation performance to the BEAR posterior predictive (median Spearman of 0.23 for UniRep, 0.20 for BEAR and 0.55 for Wavenet).
We therefore see no evidence that protein language models contradict our ﬁndings on single family models, though much further work remains to be done.