Non-identiﬁability and the Blessings of Misspeciﬁcation in Models of Molecular Fitness
Abstract
Understanding the consequences of mutation for molecular ﬁtness and function is a fundamental problem in biology.
Recently, generative probabilistic models have emerged as a powerful tool for estimating ﬁtness from evolutionary sequence data, with accuracy sufﬁcient to predict both laboratory measurements of function and disease risk in humans, and to design novel functional proteins.
Existing techniques rest on an assumed relationship between density estimation and ﬁtness estimation, a relationship that we interrogate in this article.
We prove that ﬁtness is not identiﬁable from observational sequence data alone, placing fundamental limits on our ability to disentangle ﬁtness landscapes from phylogenetic history.
We show on real datasets that perfect density estimation in the limit of inﬁnite data would, with high conﬁdence, result in poor ﬁtness estimation; current models perform accurate ﬁtness estimation because of, not despite, misspeciﬁcation.
Our results challenge the conventional wisdom that bigger models trained on bigger datasets will inevitably lead to better ﬁtness estimation, and suggest novel estimation strategies going forward.
Introduction
The past decades have witnessed a tremendous increase in the scale of genome sequence data available from across life.
Recently, methods for estimating molecular ﬁtness using generative sequence models have seen widespread success at translating this evolutionary data into predictions of the functional consequences of mutation.
Such models have been shown to accurately predict the outcomes of experimental assays of protein function [23, 44, 36] and have been applied to infer 3D structures of RNA and protein [35, 56] and to design novel proteins [50, 47, 33].
The models have also been used to predict whether human mutations are pathogenic, directly informing the diagnosis of genetic disease [19].
In this paper, we investigate how and why generative sequence models ﬁt to evolutionary sequence data are successful at estimating molecular ﬁtness, and how they might be improved and generalized going forward.
Existing approaches to ﬁtness estimation with generative sequence models rest on an assumed relationship between density estimation and ﬁtness estimation.
Given a dataset of sequences X 1 , . . .
, X N , assumed to be drawn i.i.d.
from some underlying distribution p 0 , ﬁtness models proceed by (1)
[Table]
+HOv3HaZqGtBy4czrmXe+/xY0alsqxvY2l5ZXVtPbeR39za3tk19/ZbMkoEJk0csUg4PpKE0ZA0FVWMOLEgiPuMtP3h9cRv3xMhaRQ21CgmLkf9kAYUI6Ulzzx0PBt2JeWwfpc2itXSuPhwCh2vWvLMglW2poCLxM5IAWSoe+ZXtxfhhJNQYYak7NhWrNwUCUUxI+N8N5EkRniI+qSjaYg4kW46fWEMT7TSg0EkdIUKTtXfEyniUo64rzs5UgM5703E/7xOooJLN6VhnCgS4tmiIGFQRXCSB+xRQbBiI00QFlTfCvEACYSVTi2vQ7DnX14krUrZPitXbs8Ltassjhw4AsegCGxwAWrgBtRBE2DwCJ7BK3gznowX4934mLUuGdnMAfgD4/MHkY+U9w==</latexit>
C
D ﬁtting a probabilistic model q ✓ to X 1: N and (2) using the inferred density log q ˆ ✓ ( x ) ⇡ log p 0 ( x ) as an estimate of the ﬁtness f ( x ) of a sequence x ; this estimate in turn is used to predict other covariates such as whether the mutated sequence is pathogenic [23, 44, 19].
Innovation in ﬁtness models has come out of a trend of building increasingly ﬂexible models ﬁt to increasing amounts of data: simple models that treat each column of a sequence alignment independently were improved by energy-based models that accounted for epistasis [23], which in turn were improved by deep variational autoencoders [44], which in turn were improved by deep autoregressive alignment-free models [50, 33, 36].
Naively, one might assume that these improvements have come from obtaining better and better estimates of the data distribution p 0 , and improvements will continue with bigger models and bigger datasets.
In this article, we argue that this presumption is incorrect.
Technical summary
First, we show that that the true data distribution p 0 may not reﬂect ﬁtness, and argue instead that we should be focused on estimating another distribution that does, p 1 (the “stationary distribution”, to be deﬁned below).
In particular, we demonstrate that phylogenetic effects – i.e. the history of how current sequences evolved over time – can “distort” the observed data, leading to a situation where p 0 6 = p 1 (Sec. 2).
Second, we show in this situation that p 1 and ﬁtness f are non-identiﬁable: even with inﬁnite data, there always exists some alternative ﬁtness function ˜ f
that explains the same data just as well as f .
This sets fundamental limits on what we can learn about ﬁtness from evolutionary data (Sec. 3).
Third, although exact estimation of pis impossible, we show that it is still possible to get closer to p 1 than p 0 , that is, to ﬁnd a better estimator of ﬁtness than the true data density p 0 .
This can be done by ﬁtting to data a parametric generative sequence model M = { q ✓ : ✓ 2 ⇥ } that is (approximately) well-speciﬁed with respect to p 1 (i.e. p 1 2 M ) but misspeciﬁed with respect to the data distribution p 0 (i.e. p 0 / 2 M ), thus illustrating how misspeciﬁcation can be a blessing rather than a curse (Sec. 4).
Fourth, we construct a hypothesis test to determine whether the blessings of misspeciﬁcation occur on real data, for existing ﬁtness estimation models; our test uses a recently developed Bayesian nonparametric sequence model to construct a credible set for p
(Sec. 6).
Fifth, we apply our test to over 100 separate sequence datasets and ﬁtness estimation tasks, to conclude that existing ﬁtness estimation models systematically outperform the true data distribution p 0 at estimating ﬁtness (Sec. 7).
The takeaway is that better ﬁtness estimation (i.e. better p 1 estimation) will not come from better density estimation (i.e. better p 0 estimation); bigger models and bigger datasets are not enough.
Instead, better ﬁtness estimation can come from developing models that describe p 1 better but the data density p 0 worse .
Models of Fitness and Phylogeny
In this section we illustrate how p 0 may not accurately reﬂect the true ﬁtness landscape, by analyzing a generative model of sequence evolution that takes into account not only ﬁtness but also phylogeny.
Our model is general: it allows for arbitrarily complex epistatic ﬁtness landscapes, and recovers standard phylogenetic and ﬁtness models as special cases.
Our concerns about the effects of phylogeny on ﬁtness estimation are motivated by the widespread use – and trust – of phylogenetic models for evolutionary sequence data (phylogenetic models are far more widely applied than ﬁtness models) [21, 12, 17, 18].
Note, however, that phylogeny is only one reason why p 0 may not not reﬂect the true ﬁtness landscape, and that our later results on the beneﬁts of misspeciﬁcation in ﬁtness models (Sec. 4) do not depend on the speciﬁc cause of mismatches between p 0 and the ﬁtness landscape.
Joint ﬁtness and phylogeny models We deﬁne “joint ﬁtness and phylogeny models (JFPMs)” using two elements: a description of how individual species (or populations or individuals) change over time, which depends on ﬁtness f , and a description of the species’ relationship to one another, a phylogeny H .
To describe the dynamics of individual species, let P ⌧ ( x, x 0 ) denote the probability of sequence x 0 evolving into sequence x after time ⌧ ; in particular, P ⌧ ( x, x 0 ) is assumed to be the transition probability of an irreducible continuous-time Markov chain deﬁned over sequence space X .
For example, under neutral evolution (i.e. without selection based on ﬁtness), P ⌧ ( x, x 0 ) may follow a Jukes-Cantor model [18].
With selection, for simple population genetics models (e.g. Moran or
Wright processes), Sella and Hirsh [49] demonstrate under general conditions that for any x 0 ,
[Equation]
where f ( x ) is the log ﬁtness of the sequence x and  > 0 is a constant (Appx. B).
The implication of Eqn.
1 is that the stationary distribution of the evolutionary dynamics follows a Boltzmann distribution, with energy proportional to the log ﬁtness of the sequence.
Estimating p 1 is of interest because it provides a direct estimate of log ﬁtness, up to a linear transform, since f ( x ) =
1 (log p 1 ( x ) + log Z ) .
(N.b. in the remainder of the paper, when we say “estimate ﬁtness” we mean, implicitly, “estimate log ﬁtness up to a linear transform”.)
The sequences we observe, however, do not necessarily come from the stationary distribution.
Instead, they are correlated with one another according to their evolutionary history.
This is described by a phylogeny H =
( V, E, T ) consisting of a directed and rooted full binary tree with edges E and nodes V ,
along with time labels for the nodes, T : V !
R + (Fig.
1A).
Each node v is associated with a sequence X v , drawn as X v ⇠ P  t ( x, X v 0 ) , where X v 0 is the sequence of the parent node, v is the child node, and  t =
T ( v 0 )
T ( v ) is the length of the edge between them (Fig. 1B).
The root sequence is drawn from p 1 .
The observed datapoints X 1 , . . .
, X N correspond to the leaf nodes.
In general we will assume all leaves are observed at effectively the same time, the present day T = 0 .
Special cases Standard probabilistic phylogenetic models ignore ﬁtness and assume
Assumption 2.1 (Pure phylogeny models (PMs)) .
Constant ﬁtness: f ( x ) = C .
Example models that ﬁt this form include most of those used in BEAST [14], MrBayes [25],
RaxML [51], etc.
Standard probabilistic ﬁtness models, on the other hand, ignore phylogenetic history and assume that the stationary distribution has been reached,
Assumption 2.2 (Pure ﬁtness models (FMs)) .
Let ⌧ i be the distance in time between observed sequence X i and its parent node.
Take ⌧ i !
1 for all i , which implies that
[Equation]
The key implication of this assumption is that density estimation and ﬁtness estimation are linked: the data follows X 1 , . . .
, X N ⇠ iid p
= p 1 , and so if we can estimate p
we can estimate the ﬁtness.
Example models include EVMutation [23], DeepSequence [44], EVE [19], etc.
Note although
Assumptions 2.1 and 2.2 do not conﬂict directly, conclusions made based on them conﬂict in practice:
PMs typically infer ﬁnite and different lengths for branches (i.e. ⌧ i < 1 ), while FMs typically infer differences in ﬁtness (i.e. f ( x ) 6 = C ), even when applied to the same dataset.
1D Example
If Asm.
2.2 does not hold, then there is no reason for the distribution of observed sequences X 1 , X 2 , . . .
to follow p 1 .
We illustrate this with an example, the most widely used JFPM that does not use Assumptions 2.1 or 2.2: the Ornstein-Uhlenbeck tree (OUT) model [18, 8].
In this model, X is continuous, i.e. X 2 R , and evolves on a quadratic ﬁtness landscape of the form f ( x ) /
( x  µ ) 2 + C according to the dynamics P ⌧ ( x, x 0 ) =
Normal
[Equation]
The stationary distribution p 1 is Normal( µ,  2 ) .
One can show (Appx. C.1)
that for any H ,
Proposition 2.3 (OUT observations) .
The distribution of observed genotypes X 1: N is drawn from a multivariate normal distribution with mean µ ~ 1 N and covariance ⌃ where
[Equation]
and t ij ( H ) is the total time of the shortest path between leaves i and j along the phylogeny H .
We drew samples from the OUT with a Kingman coalescent prior on H ([3] Def. 2.1) and plotted their density (Fig. 1C).
Even as N ! 1 , the distribution of samples does not follow p 1 .
Moreover, rerunning the process with a new sample from the prior yields a very different distribution (Fig.
1D).
Non-identiﬁability
In this section we investigate whether, given inﬁnite sequence data, it is possible to infer ﬁtness f without Asm. 2.2, and conversely, whether it is possible to infer phylogeny H without Asm. 2.1.
That is, we are interested in whether ﬁtness and phylogeny are identiﬁable in general JFPMs.
We conclude they are not: given inﬁnite data generated with any f and H , there exists some alternative ˜ f and ˜ H , where ˜ H satisﬁes Asm. 2.2, that explains the data equally well.
Naively, this result may be surprising: in FMs, each sequence is drawn independently, i.e.
X i ??
X j | H , f , while in JFPMs and PMs there is (in general) correlation between sequences, i.e. X
i 6??
X j | H , f .
One might then hope that examining correlations between sequences would enable us to infer whether Asm. 2.2 holds.
However, we can show that these correlations are uninformative due to a symmetry in JFPMs, exchangeability.
Assumption 3.1 (Exchangeability) .
Let m ( X 1 , X 2 , . . . )
denote the marginal probability of an inﬁnite set of sequences X 1 , X 2 , . .
integrating over all phylogenies, i.e. m ( X 1 , X 2 , . . . )
=
R p ( X 1 , X 2 , . . .
| H ) p ( H ) d H .
Then, for any permutation ⇡ of the integers,
[Equation]
Exchangeability says that if we had observed the sequences in a different order, it would not change their probability.
Exchangeability is a ubiquitous assumption in machine learning and statistics models, and its application depends primarily on the information available in a dataset: it is a sensible assumption whenever the ordering of the datapoints provides no useful information.
In typical datasets used for ﬁtness estimation, sequences are separated by millions of years of evolution, and are thus all effectively observed at the same time: the present day, T = 0 .
In other words, there is no a priori way of ordering the sequences in the dataset, and so we must assume exchangeability.
Standard priors on phylogenetic trees, such as the Kingman coalescent, are explicitly constructed to enforce exchangeability [3, 14].
Exchangeability implies that ﬁtness and phylogeny are not identiﬁable.
Even if X 1 , X 2 , . . . are generated from a JFPM with a ﬁnite branch length phylogeny H , we can describe the same data just as well using an FM model with an inﬁnite branch phylogeny ˜ H :
Theorem 3.2 (Non-identiﬁability) .
Assume X 1 , X 2 , . .
satisfy Assumption 3.1.
Then with probability 1 there exists some function ˜ f such that
[Equation]
Proof.
Applying de Finetti’s Theorem ([29], Thm. 11.10), a.s. there exists a random measure G such that for i 2 { 1 , 2 , ... } , X i iid ⇠ G .
Let p G ( x ) be the pmf of G .
(We assume x is a ﬁnite discrete sequence; we can also use continuous x assuming the pdf p G ( x ) exists.)
Set ˜ f
( x ) =
[ p G ( x )] 1 /  .
This result says that the observed sequences from an exchangeable JFPM, X 1 , X 2 , . .
. , are precisely i.i.d.
samples from some p 0 .
Although in the standard tree representation X i 6??
X j | H , f , there must be some alternative description of the same process where X i ??
X j | ˜ H , ˜ f .
Fitness and phylogeny are thus non-identiﬁable: data generated from a JFPM with ﬁtness f and phylogeny H can be described just as well using ˜ f and ˜ H , and vice versa.
We emphasize that this non-identiﬁability result is highly general, and does not depend on the speciﬁc choice of evolutionary dynamics P ⌧ , only on the assumption of exchangeability.
The biological intuition behind Thm.
3.2 is that if two sequences are similar to each other and distant from a third, they may be similar either because they are closely related (i.e. the distance ⌧ to the most recent common ancestor is small) or because they are in a local maximum of the ﬁtness landscape.
Without further assumptions, we cannot tell the difference between these two explanations.
The machine learning intuition is that evolution, as described by a JFPM, is in effect a Markov chain
Monte Carlo process whose stationary distribution gives the ﬁtness.
However, the samples we observe may not be fully independent: each pair of samples was initialized from the same point (the most recent common ancestor), and the burn-in since that point may not be sufﬁciently long.
Without independent samples, our estimate of the stationary distribution will be biased.
Fitness inference as hyperparameter inference While general, Thm. 3.2 is not constructive, and does not tell us what the distribution p 0 actually is, or how exactly it differs from p 1 .
Thm. 3.2 also
[Table]
[Figure]
[Table]
[Figure]
[Table]
+x+NqjW3Lo7B1olXkFqUKA1qH71h5KkMRWGcKy177mJCTKsDCOczir9VNMEkwkeUd9SgWOqg2weeYbOrDJEkVT2CYPm6u+NDMdaT+PQTuYR9bKXi/95fmqi6yBjIkkNFWTxUZRyZCTK70dDpigxfGoJJorZrIiMscLE2JYqtgRv+eRV0mnUvYt64+Gy1rwp6ijDCZzCOXhwBU24gxa0gYCEZ3iFN8c4L86787EYLTnFzjH8gfP5A4PYkWc=</latexit>
[Figure]
leaves unclear how much we need to know to learn the ﬁtness landscape: could we infer ﬁtness f
if we knew the parametric form of p 1 , i.e. if we had some model M and knew that p 1 2 M ?
What if we also knew the underlying phylogeny H ?
In the long branch limit (Asm. 2.2), ﬁtness is identiﬁable if H is known; if M is also known, learning ﬁtness is a matter of inferring model parameters.
In the limit where all the branch lengths in the phylogeny are zero, the distribution of observations from a JFPM reduces to X 1 ⇠ p 1 and X 1 = X 2 = X 3 = . . . .
Here ﬁtness is non-identiﬁable even if H and M are known; learning ﬁtness is a matter of learning from a single sample.
In the realistic intermediate branch length case, if H and M are known, we will show that learning ﬁtness is essentially a matter of hyperparameter rather than parameter inference.
We demonstrate this last claim in the context of the OUT example, by approximating the OUT model as a Gaussian process latent variable model (GPLVM).
We ﬁnd that ﬁtness only appears as a hyperparameter of the derived GP, not as a parameter.
The GPLVM has latent variables Z 1 , Z 2 , . . .
that lie on the hyperbolic plane H , and uses the Gaussian process kernel k ( · , · ) =
exp(  d ( · , · )) , where d ( · , · ) is a distance metric over H .
Let W 1 ( · , · ) be the Wasserstein metric for distributions over inﬁnite matrices, i.e. over R 1⇥1 , using the sup norm on matrices.
Theorem 3.3 (GPLVM approximation of OUT) .
Assume a prior over phylogenies H that is exchangeable in its leaves and where the minimum time between any pair of nodes is greater than ⌘ > 0 with probability 1.
Deﬁne the leaf distance matrix ⌫ ij = log( 12 t ij ( H )) .
For any ✏ > 0 , there exists a.s. a GPLVM of the form,
[Equation]
where G is a random measure over H , such that W 1 ( p ( ⌫ ) , p (˜ ⌫ )) <
✏ , where ˜ ⌫ ij = log( d ( Z i , Z j )) .
If W 1 ( p ( ⌫ ) , p (˜ ⌫ )) = 0 , the OUT and GPLVM produce identical distributions over X 1 , X 2 , . . .
a.e..
The proof is in Appx.
C.2, and uses the embedding of Sarkar [48].
This result says that, by embedding phylogenies H in a metric space, we can approximate an OUT arbitrarily well with a GPLVM; as the
Wasserstein bound gets smaller, the distribution of covariance matrices of the two models get closer.
In the GPLVM, the observations are conditionally independent, X i ??
X j | s, G , in line with Thm. 3.2.
The phylogeny H enters the GPLVM only through the latent space embedding Z 1 , Z 2 , . . . .
Learning phylogeny, given the ﬁtness landscape, is thus essentially a matter of inferring latent variables [44, 13].
The ﬁtness landscape enters the GPLVM only through the prior on the Gaussian process (i.e. through µ and  ).
Inferring ﬁtness given phylogeny is thus essentially a matter of inferring hyperparameters.
This is both good and bad news for ﬁtness inference.
On the one hand, hyperparameters are often learned in practice, and doing so can yield substantially better predictions, so we should be able learn something about µ and  given data ([58], Chap. 5).
On the other hand, hyperparameters are in general (though not always) non-identiﬁable, and therefore so is ﬁtness [34].
Ho and Ané [22] describe non-identiﬁability conditions for the OUT in particular.
We conclude that even when H and
M are known, ﬁtness inference in JFPMs is fundamentally challenging.
Beneﬁts of misspeciﬁcation
We have demonstrated that a plausible biological mechanism – namely, phylogenetic effects – can produce a data distribution p 0 that does not reﬂect ﬁtness, and can make exact inference of ﬁtness impossible even given inﬁnite data.
Nonetheless, the practical success of ﬁtness estimation methods suggest it is possible to at least approximate ﬁtness landscapes from observational sequence data.
Recall that existing methods proceed by ﬁtting a probabilistic model q ✓ 2 M = { q ✓ : ✓ 2 ⇥ } to data X 1: N , typically via maximum likelihood estimation or approximate Bayesian inference, and then using the predicted log density log q ˆ ✓ ( x ) as an estimate of the ﬁtness of a sequence x .
Why is this approach empirically successful?
In this section we consider two hypotheses, either of which may hold true in theory.
In Secs.
6-7 we develop and apply tests to evaluate them on real data.
Note that our results in this and the following sections are independent of the speciﬁc evolutionary mechanisms that generate a mismatch between p 0 and p 1 , i.e. they are not speciﬁc to phylogenetic effects or JFPMs, nor do they even depend on the existence of a stable reproductive ﬁtness function f over evolutionary time.
We can, in fact, redeﬁne p 1 to be an arbitrary “target distribution”, with log p 1 proportional to any chosen measure of molecular ﬁtness or function (such as enzyme activity, ﬂuorescence, etc.).
For the sake of concrete illustration, however, we will continue to focus on JFPMs as our primary example of why the data distribution, p 0 , may not equal the target distribution we want to estimate, p 1 .
We consider two hypotheses for the empirical success of existing ﬁtness estimation methods.
Hypothesis #1 (informal).
Fitness estimation methods succeed by ﬁnding q ˆ ✓ ⇡ p 0 , since for all practical purposes on real data, p 0 =
p 1 .
This hypothesis would make sense, in JFPMs, if Asm.
2.2 held, i.e. branch lengths were long enough in real datasets for P ⌧ i ( x, x 0 ) to be close to its stationary distribution.
Under this hypothesis, better density estimators have been, and will continue to be, better ﬁtness estimators.
We should focus on developing models M that are well-speciﬁed with respect to the data, i.e. p 0 2 M (Fig.
2A).
Hypothesis #2 (informal).
Fitness estimation methods succeed by using models M that are misspeciﬁed with respect to p 0 , i.e. p 0 / 2 M .
The inferred model q ˆ ✓ is then closer to p 1 than p 0 is.
To show this second hypothesis is plausible, we prove that it is guaranteed to hold under general conditions.
We study the projection of p 0 onto M via the Kullback-Leibler (KL) divergence, q ✓ ⇤ =
argmin q ✓ 2 M KL ( p 0 k q ✓ ) .
The KL projection is relevant because maximum likelihood estimation minimizes the approximate KL divergence between the data and the model, and the posterior in
Bayesian inference asymptotically concentrates around the maximum likelihood estimator [38].
We thus expect the ﬁt model q ˆ ✓ to be close to q ✓ ⇤ , and get closer with N .
Assume that M is “log-convex”, meaning that for any ✓ , ✓ 0 2 ⇥ and 0 < r < 1 , there exists some ✓ 00 such that q ✓ 00 ( x ) =
q ✓ ( x
[Equation]
q
[Equation] [Equation]
) 1  r ; examples of log-convex models include the Potts model, as well as all other exponential family models.
Let TV ( p k q ) be the total variation distance between p and q , and let k g k 1 = sup x | g
( x ) | be the uniform (sup) norm of g .
Theorem 4.1 (Beneﬁts of misspeciﬁcation) .
Assume that the model M is log-convex and that q ✓ ⇤ exists and is unique.
If the model is “less misspeciﬁed” with respect to the stationary distribution pthan with respect to the data distribution p 0 , in the sense that
[Equation]
6) then,
[Equation]
However, if the model is well-speciﬁed with respect to the data distribution, i.e. p 0 2 M , we have,
[Equation]
Sec.
C.3 contains the proof, and explains how Thm.
4.1 can be extended with more general conditions (we also emphasize again that the proof does not make any assumption that p 0 and p 1 follow a
JFPM).
Thm.
4.1 says that if the model is less misspeciﬁed with respect to the target distribution than with respect to the data distribution, then projecting the data distribution onto the model will yield a distribution q ✓ ⇤ closer to the target distribution.
In other words, the best models for ﬁtness estimation are those at a “sweet spot” of complexity, ﬂexible enough to capture p 1 but not so ﬂexible as to capture p 0 .
To understand the biological intuition behind this result, consider a situation where two neutral mutations with no effect on ﬁtness occur successively at different sites (Fig.
2C).
Due to phylogenetic correlation, there is no observed sequence x ⇤ in which the second mutation is present but not the ﬁrst, so an accurate density estimator will ﬁnd p 0 ( x ⇤ ) ⇡ 0 .
However, if we can guess correctly that the ﬁtness landscape is independent across sites, then ﬁtting a site-wise independent model M will imply the mutation is allowed, q ✓ ⇤ ( x ⇤ ) > 0 , correctly inferring p 1 ( x ⇤ ) > 0 .
Under Hypothesis 2, progress in the ﬁeld of ﬁtness estimation has not come from building better density estimators (Hypothesis 1), but rather from an iterative process of (1) hypothesizing, based partly on biophysical knowledge, models that are approximately well-speciﬁed with respect to pbut poorly speciﬁed with respect to the data distribution p 0 , and then (2) comparing their density estimates against experimental ﬁtness measurements.
We will show that on real data, Hypothesiscan often be rejected in favor of Hypothesis 2.
Related Work
Efforts to account for the effects of phylogeny in ﬁtness estimation have a long history [32].
Practical generative sequence models that explicitly account for both epistatic ﬁtness landscapes and phylogeny have long been sought, but stymied primarily by computational challenges [28, 46].
In their place, a variety of non-generative (and often heuristic) methods for correcting for phylogeny have been proposed, including data reweighting schemes [35, 46], data segmentation schemes [10], postinference parameter adjustments [16], covariance matrix denoising methods [42], simulation based statistical testing [45], and more.
In this article, we show that deconvolving ﬁtness and phylogeny is not just computationally hard, but also in general statistically impossible: ﬁtness and phylogeny are non-identiﬁable.
We further show that use of a misspeciﬁed parametric model can on its own (without further corrections) partially adjust for phylogenetic effects.
Our results also intersect with the literature on robust statistics: we can think of the observed data distribution p 0 as a “distorted” version of the true distribution of interest p 1 .
However, in typical robust inference frameworks (e.g. Huber’s epsilon contamination model), the observed distribution differs from the true distribution by the addition of outliers [24, 52].
Our setup is, in some sense, the opposite: inliers are deleted, as phylogenetic correlations can result in an effective support of p 0 that is smaller than that of p 1 (Fig.
1CD).
Diagnostic Method
In this section, we develop diagnostic methods to discriminate between Hypothesis 1 and Hypothesis 2 (Sec. 4) based on observational sequence data and experimental ﬁtness measurements, and validate these diagnostics in simulation.
Recall that under Hypothesis 2, the estimate q ˆ ✓ from a parametric ﬁtness model is a better estimate of ﬁtness than the true data density p 0 , while under Hypothesis 1, p 0 is better.
Discriminating these two hypotheses on real data is nontrivial because we do not have access to p 0 .
Ideally, then, a diagnostic test would evaluate the probability that the true density poutperforms q ˆ ✓ at predicting ﬁtness, taking into account uncertainty in what p 0 could actually be, given the data.
To accomplish this, we compute a posterior over p
using a Bayesian nonparametric sequence model.
In particular, we apply the Bayesian embedded autoregressive (BEAR) model, which can be scaled to terabytes of data and satisﬁes posterior consistency ([2], Thm. 35):
Theorem 6.1 (Summary of BEAR posterior consistency) .
Assume p 0 is subexponential, i.e. for some t > 0 , E X ⇠ p
[exp( t | X | )] < 1 , where | X | is the length of sequence X .
Assume the conditions on the prior detailed in Amin, Weinstein and Marks [2].
If X 1 , X 2 , . . .
⇠ p 0 i.i.d., then for M >sufﬁciently large and ✏ 2 (0 , 1 / 2) sufﬁciently small,
[Equation]
in probability, where B ( p, r ) is a Hellinger ball of radius r centered at p , and ⇧ BEAR ( ·| X 1: N ) is the
BEAR posterior.
Crucially, this result implies that the BEAR posterior will converge to effectively any value of p 0 , no matter what p 0 is (unlike a parametric model’s posterior).
Moreover, BEAR quantiﬁes uncertainty in its estimates, giving the range of possible values of p 0 that are consistent with the evidence.
We construct our diagnostic test by comparing the ﬁtness estimation performance of q ˆ ✓ to the range of possible performances of p 0 estimated by BEAR.
Let S f ( p ) be a scalar score evaluating how accurately a density p predicts ﬁtness f .
In practice, S f will be based on experimental and clinical measurements of quantities directly related to ﬁtness.
Diagnostic test (Test Hypothesis 1 vs. Hypothesis 2.)
Hypothesis
[Equation] [Equation]
Hypothesis
[Equation]
) >
[Equation]
p
[Equation]
Accept Hypothesis 2 at signiﬁcance level ↵ >
if
[Equation]
[Figure]
Accept Hypothesis 1 at signiﬁcance level ↵ if
[Equation]
So long as S f ( p ) is a well-behaved function of p (in particular, so long as S f is continuous in a neighborhood of p 0 with respect to the topology of convergence in total variation), Thm.
6.1 implies that this diagnostic test will be asymptotically consistent, in the sense that it converges to the correct hypothesis in probability.
Simulations We next evaluate the performance of our diagnostic test on simulated data.
We considered two scenarios, the ﬁrst in which Hypothesis 1 holds, and the second in which Hypothesis 2 holds.
In both, we let M be a site-wise independent (SWI) model, in which each position of the sequence is drawn independently, i e. X l ⇠ Categorical( v l ) for l 2 { 1 , . . .
, | X |} .
The parameter v l is in the simplex
B , where B + 1 is the alphabet size.
(Further details in Appx.
D.)
In Scenario 1, the true data are generated according to a Potts model and p
= p 1 .
In this scenario, the SWI model is misspeciﬁed, and misspeciﬁcation is bad : using a more ﬂexible model will produce an asymptotically more accurate estimate of p 1 .
We ﬁnd that our diagnostic test asymptotically correctly accepts
Hypothesis 1, in line with Thm.
6.1 (Figs.
3A and 7A).
In Scenario 2, the true data are generated according to a JFPM with ﬁnite branch lengths, and p 1 2 M while p 0 / 2 M .
The mutational dynamics P ⌧ follow the Sella and Hirsh [49] process.
The phylogeny
H is drawn from a Kingman coalescent.
In this scenario, the SWI model is again misspeciﬁed, but misspeciﬁcation is good : while the nonparametric BEAR model can achieve better density estimates than the SWI model (Fig.
3C), the SWI model outperforms BEAR at ﬁtness estimation (Figs.
3D and 8).
We ﬁnd that our diagnostic test correctly accepts Hypothesis 2 (Figs.
3B and 7B).
A possible point of concern is that the test is poorly calibrated from a frequentist perspective, and in the low N regime accepts Hypothesis 2 in Scenario 1 more than 100 ↵ % of the time when the data is resampled from p 0 (Fig.
9A).
This behavior is common in nonparametric Bayesian tests, and not necessarily a problem: the test is still valid from a purely Bayesian perspective.
Nevertheless, on real data we will check that we are close to the large N regime by (1) checking that the BEAR posterior predictive is at least as close to p 0 as q ˆ ✓ is (as measured by perplexity on held out data;
Figs.
3C and 9B) and (2) examining the plot of the BEAR posterior over S f ( p ) as a function of N (as in Fig.
3AB), to check that it has converged.
Empirical Results
We now evaluate whether existing ﬁtness estimation methods outperform the true data density p 0 , i.e. whether we can reject Hypothesis 1 in favor of Hypothesis 2 on real data.
Tasks We consider two key prediction tasks where ﬁtness models are applied in practice.
The ﬁrst task is to predict whether variants of a protein are functional, according to an experimental assay of protein function; the metric S f ( · ) is the Spearman correlation between p ( x ) and the assay result [23].
There are typically ⇠ 1000s of measurements per assay.
The second task is to predict whether a variant of a protein observed in humans causes disease, according to clinical annotations; the metric
S f ( · ) is the area under the ROC curve when p ( x ) is used to predict whether or not a variant is
[Figure]
pathogenic [19].
There are typically only a handful of labels for each gene.
For the ﬁrst task, we considered 37 different assays across 32 different protein families, and for the second task, 97 genes across 87 protein families; for each protein family, we assembled datasets of evolutionarily related sequences, following previous work.
Note that across the 37 assays and 97 genes, the data used for
S f comes from different experiments and different clinical evidence, often collected by different laboratories or doctors.
Thus, our overall conclusions should be robust to the choice of S f .
Models We considered three existing ﬁtness estimation models: a site-wise independent model (SWI), a Bayesian variational autoencoder (EVE [19], which is similar to DeepSequence [44]), and a deep autoregressive model (Wavenet) [50].
Note that SWI and EVE, unlike Wavenet, require aligned sequences as training data.
Details in Appx.
E.
Results Applied to the ﬁrst prediction task, our diagnostic test accepts Hypothesis 2 at signiﬁcance level ↵ = 0 .
025 in 35/37 assays (95%) for SWI, 35/37 assays (95%) for EVE, and 36/37 assays (97%) for Wavenet (Fig. 4A).
Applied to the second prediction task, our diagnostic test accepts Hypothesis 2 at signiﬁcance level ↵ = 0 .
025 in 31/97 genes (32%) for SWI and 46/97 genes (47%) for EVE (Fig.
4B).
Thus, ﬁtness estimation models are capable of outperforming the true data distribution p 0 .
We found evidence for Hypothesis 1 in only a handful of examples: on the ﬁrst task, Hypothesis 1 was accepted at signiﬁcance level ↵ = 0 .
025 in 0/37 assays for SWI, 1/37 assays (3%) for EVE, and 0/37 assays for Wavenet, while on the second task, Hypothesis 1 was accepted for 5/97 genes (5%) for SWI and 4/97 genes (4%) for EVE.
We conﬁrmed that the diagnostic test was in the large N regime:
BEAR outperformed Wavenet at density estimation, providing better predictive performance on 27/37 assays (73%) and similar performance on the remaining 10 assays (Fig. 10).
(Note that we cannot do this comparison for SWI or EVE since they are alignment-based [57].)
Example plots of the BEAR posterior’s convergence with N on the ﬁrst prediction task showed convergence to values of S f well below that for parametric ﬁtness estimation models (Figs.
4C and 11-12).
Overall, we conclude that there is strong evidence that existing ﬁtness estimation methods reliably outperform the true data distribution p 0 across a range of datasets and tasks.
To study the tradeoffs between density estimation and ﬁtness estimation in more depth, we smoothly and nonparametrically relaxed a parametric autoregressive (AR) model (Appx. E.4).
We embedded the AR model (a convolutional neural network) into a BEAR model, and ﬁt the BEAR model with empirical Bayes.
We found evidence that the AR model was misspeciﬁed on every dataset, following the methodology of Amin, Weinstein and Marks [2]: the optimal h selected by empirical Bayes was on the order of 1  10 in each dataset.
Now, in the limit as the hyperparameter h ! 0 , the
BEAR model collapses to its embedded AR model; so by scanning h from low to high values we can interpolate between the parametric and nonparametric regime.
We ﬁnd a smooth tradeoff between
S f ( p ) and the likelihood of the data under the BEAR model, with higher h corresponding to better density estimation but worse ﬁtness estimation (Fig.
4EF and 13).
This relationship held across many datasets: the diagnostic test, evaluated against the AR model (the h ! 0 limit), accepts Hypothesisin 28/37 assays (76%), but Hypothesis 1 in only 6/37 (16%) (Fig. 14).
These results conﬁrm that making a model well-speciﬁed (relaxing from a parametric to a nonparametric model) can bring improved density estimation at the cost of worse ﬁtness estimation.
Discussion
In this article, we have argued that better density estimation does not necessarily lead to better ﬁtness estimation.
Further, we estimate with high probability that existing ﬁtness estimation methods systematically outperform the true training data density.
Although existing methods rely on ﬂexible, high-parameter deep neural network models, they can nonetheless be misspeciﬁed; but this misspeciﬁcation acts as a blessing, rather than a curse for ﬁtness estimation.
Successful models are at a sweet spot of complexity, ﬂexible enough to capture the target ﬁtness distribution well but not so ﬂexible as to match the data distribution itself.
We have focused on state-of-the-art ﬁtness estimation methods which are trained on data from individual protein families [44, 50, 19].
Recently, large-scale generative sequence models (“protein language models”) trained on more diverse datasets (containing proteins from many different families) have show ﬁtness estimation performance comparable to, and in some settings surpassing, single family models [36, 40, 41].
Although applying our diagnostic test to these datasets requires further work, there is no reason to expect the same limitations of density estimation do not hold for such models.
Indeed, following a preprint of this paper, Nijkamp et al.
[40] presented evidence of the beneﬁts of misspeciﬁcation in a protein language model: past a certain number of parameters, density estimation improved while ﬁtness estimation deteriorated.
See Appx.
G for further discussion.
One future direction is to further explore models M that are less ﬂexible than existing models and worse at density estimation, since they can increase the gap between KL ( q ✓ ⇤ k p 1 ) and KL ( p 0 k p 1 )
(Thm. 4.1).
There may also be opportunity to improve model geometry: while exponential family models are guaranteed to be log-convex (and thus can satisfy Thm. 4.1), we have no such guarantee for variational autoencoders or other neural network methods.
Meanwhile, uncertainty quantiﬁcation is crucial for applications such as those in clinical genetics, but challenging in misspeciﬁed models [53, 39, 26].
Alternatively, it might be useful to abandon the strategy of using misspeciﬁed models for ﬁtness estimation altogether, and instead construct JFPM models where the ﬁtness landscape is represented explicitly as a latent variable.
Recent progress on amortized variational inference for phylogenetic models is promising for building ﬂexible and scalable JFPMs [55].
However, handling non-identiﬁability is challenging, and may require new assumptions and/or new methods of sensitivity analysis to infer the full set of ﬁtness landscapes consistent with the data [9].
Although this article has focused on technological applications of ﬁtness models in solving prediction problems, ﬁtness models also have implications for our fundamental understanding of evolution.
Pure phylogeny models and pure ﬁtness models present very different pictures of the past history of life: in
PMs, similarities and differences among genetic sequences are determined primarily by history and ancestry (Asm. 2.1), while in FMs they are primarily determined by functional constraints (Asm. 2.2).
PMs and FMs also present very different implications for the future of life: in PMs, the diversity of sequences seen in nature will likely expand dramatically going forward, while in FMs, the landscape of functional sequences has already been well-explored.
Our results emphasize that where and to what extent each model offers an accurate picture of reality remains an open question.
Acknowledgements
We thank members of the Marks lab, Pascal Notin, Ali Madani, John Ingraham and the anonymous reviewers for insights and suggestions.
E.N.W.’s work was supported by the Fannie and John Hertz
Foundation.
D.S.M. is supported by the Chan Zuckerberg Initiative.