Power posteriors do not reliably learn the number of components in a ﬁnite mixture
Abstract
Scientists and engineers are often interested in learning the number of subpopulations (or components) present in a data set.
Data science folk wisdom tells us that a ﬁnite mixture model (FMM) with a prior on the number of components will fail to recover the true, data-generating number of components under model misspeciﬁcation.
But practitioners still widely use FMMs to learn the number of components, and statistical machine learning papers can be found recommending such an approach.
Increasingly, though, data science papers suggest potential alternatives beyond vanilla FMMs, such as power posteriors, coarsening, and related methods.
In this work we start by adding rigor to folk wisdom and proving that, under even the slightest model misspeciﬁcation, the FMM component-count posterior diverges: the posterior probability of any particular ﬁnite number of latent components converges to 0 in the limit of inﬁnite data.
We use the same theoretical techniques to show that power posteriors with ﬁxed power face the same undesirable divergence, and we provide a proof for the case where the power converges to a non-zero constant.
We illustrate the practical consequences of our theory on simulated and real data.
We conjecture how our methods may be applied to lend insight into other component-count robustiﬁcation techniques.
Introduction
In any probabilistic model, simplifying assumptions are made out of necessity.
Some models and conclusions nonetheless lead to reliable and useful inference.
But some misspeciﬁcation will have undesirable consequences.
In this work we focus on the case of mixture models — which are widely used to discover latent groups, or components, within a population.
Often the number of components is unknown in advance and one of the principal inferential goals is estimating and interpreting this number.
For example, practitioners might wish to ﬁnd the number of latent genetic populations (Pritchard et al., 2000; Lorenzen et al., 2006; Huelsenbeck and Andolfatto, 2007), gene tissue proﬁles (Yeung et al., 2001; Medvedovic and Sivaganesan, 2002), cell types (Chan et al., 2008; Prabhakaran et al., 2016), haplotypes (Xing et al., 2006), switching Markov regimes in US dollar exchange rate data (Otranto and Gallo, 2002), gamma-ray burst types (Mukherjee et al., 1998), or segmentation regions in an image (e.g., tissue types in an MRI scan (Banﬁeld and Raftery, 1993)).
In all of these cases, it is typical to assume some standard parametric form for the mixture likelihoods, and we can expect that this form will be at least slightly misspeciﬁed.
A common approach to learning mixture components and their cardinality is to use a ﬁnite mixture model with a prior on the number of components (FMM) (Nobile, 1994; Stephens, 2000; Green and
Richardson, 2001; Nobile, 2004, 2007; Nobile and Fearnside, 2007; Miller and Harrison, 2013, 2014, ; Grazian et al., 2020).
Typically the prior on component counts is supported on all possible strictly-positive integers, and we compute a posterior distribution over the number of components (Pritchard et al., 2000; Lorenzen et al., 2006; Huelsenbeck and Andolfatto, 2007).
But empirical evidence (Miller and Dunson, 2019, e.g.) suggests that the posterior number of components is very sensitive to misspeciﬁcation of the component likelihoods.
Indeed, data science folk wisdom tells us that ﬁnite mixture models will tend to choose too many components in practice (Frühwirth-Schnatter, 2006, Chapter 7).
One proposal to address this mis-estimation is to use an α -posterior, or power posterior , where we replace the likelihood with the same likelihood but raised to a ﬁxed power α > 0 , often between 0 and 1 (Grünwald, 2006; Grünwald and van Ommen, 2017; Royall and Tsou, 2003; Ghosh and Sudderth, 2012; Holmes and Walker, 2017).
Power posteriors have much more general application than just to mixture models; we focus only on mixture models here and note that behavior of power posteriors for other models may be very different than for mixture models.
In a separate line of work, Miller and Dunson (2019) propose a coarsened posterior , which they show can be closely approximated by a variant of the power posterior with exponent α N → 0 as the number of data points N → ∞ ; in fact, they use this approximation in all of their experiments.
Note that we use the terminology “power posterior” or α -posterior throughout to refer to the ﬁxed power case, and the terminology α N -posterior to refer to the case where the power may depend on N .
In the present work, we start by adding rigor to data science folk wisdom and prove that, under even the slightest model misspeciﬁcation, the posterior number of components in a classic ﬁnite mixture analysis diverges; that is, the posterior probability of any particular ﬁnite number of latent components converges to 0 in the limit of inﬁnite data.
We use a similar analysis to lend insight into the power posterior for ﬁnite mixtures; in particular, we ﬁnd that the component-count power posterior, with power that is constant in N , diverges in the same way.
We give a proof that an α N -posterior, where α N converges to a nonzero constant, also diverges.
We support our theory with experiments on simulated and real data.
We leave analysis of powers α N → 0 to future work but provide some discussion here.
Main results
We begin with a brief description of the ﬁnite mixture model we consider in this work and a statement of the main results.
We defer precise details to Section 3.
Let g be a mixing measure g :=
(cid:80) kj =1 w j δ θ j on a parameter space Θ with w j ∈ [0 , 1] and (cid:80) kj =1 w j = 1 , and let Ψ
= { ψ θ : θ ∈ Θ } be a family of component distributions dominated by a σ -ﬁnite measure µ .
Then we can express a ﬁnite mixture f of the components as
[Equation]
Consider a Bayesian model with a prior distribution Π on the set of all mixing measures G on Θ with ﬁnitely many atoms, i.e., g ∼ Π , and likelihood corresponding to conditionally i.i.d.
data from f = (cid:82) ψ θ dg ( θ ) .
The model assumes the likelihood is f , but the model is misspeciﬁed ; i.e., the observations X 1: N :=
( X 1 , . . .
, X N ) are actually generated conditionally i.i.d.
from f 0 , which is not itself a ﬁnite mixture of distributions in Ψ .
For example, the likelihood f might be a ﬁnite mixture of Gaussian distributions, but f 0 might represent a ﬁnite mixture of Laplace distributions.
Let Π ( α ) ( k | X 1: N ) denote the posterior marginal number of components induced by raising the likelihood to a ﬁxed power α > 0 .
Our main result is that under this misspeciﬁcation of the likelihood, for any α ∈ (0 , 1] , the α -posterior on the number of components Π ( α ) ( k | X 1: N ) diverges ; i.e., for any ﬁnite k ∈ N , Π ( α ) ( k | X 1: N ) → 0 as N → ∞ .
We make only two requirements of the mixture model to guarantee this result: (1) the true datagenerating distribution f 0 must be arbitrarily well-approximated by ﬁnite mixtures of Ψ , and (2) the family Ψ must satisfy mild regularity conditions that hold for popular mixture models (e.g., the family Ψ of Gaussians parametrized by mean and variance).
We provide precise deﬁnitions of the assumptions needed for Theorem 2.1 to hold in Section 3, and a proof in Appendix A.1.
Theorem 2.1 (Main result) .
Suppose observations X 1: N are generated i.i.d.
from a distribution fthat is not a ﬁnite mixture of Ψ .
Assume that:
Assumption 3.1: f 0 is in the KL-support of the prior Π , and
Assumption 3.6: Ψ is continuous, is mixture-identiﬁable, and has degenerate limits.
Then for any α ∈ (0 , 1] , the α -posterior on the number of components diverges; i.e., for all k ∈ N ,
[Equation]
Note that the conditions of the theorem—although technical—are satisﬁed by a wide class of models used in practice.
Assumption 3.1 requires that the prior Π places enough mass on mixtures near the true generating distribution f 0 .
Assumption 3.6 enforces regularity of the component family and is satisﬁed by many popular models used in practice, such as the multivariate Gaussian family (Proposition B.2) and, more generally, mixture-identiﬁable location-scale families (Proposition B.3).
While the result of Theorem 2.1 assumes that the model uses a ﬁxed prior Π , in many practical modeling scenarios it is common to specify a prior Π N that depends on observed data X 1: N (e.g., to establish a parameter range).
If f 0 satisﬁes a modiﬁed KL-support condition with respect to the sequence of priors Π N , it is straightforward to show that the α -posterior number of components also diverges in this setting by Corollary A.5. 2.1 Extension to sequences α N
Much of the existing literature on power posteriors focuses on the case where the power is ﬁxed between 0 and 1 (Grünwald, 2006; Grünwald and van Ommen, 2017; Walker and Hjort, 2001; Royall and Tsou, 2003).
In Theorem 2.1, we considered the asymptotic behavior of the α -posterior number of components with a ﬁxed α ∈ (0 , 1) .
We also provide a result about the α N -posterior number of components, where the sequence α N may depend on the sample size N .
Corollary 2.2 states that if α N converges to a value between 0 and 1, then the α N -posterior number of components diverges; a proof is presented in Appendix A.2.
Corollary 2.2. Suppose the conditions of Theorem 2.1 hold. Let α N → α , where α ∈ (0 , 1) . Then for any k ∈ N ,
[Equation]
Note that the above statement does not include sequences α N that converge to 0.
A notable instance of a sequence α N → 0 is studied by Miller and Dunson (2019) as an approximation to a coarsened posterior.
Speciﬁcally, Miller and Dunson (2019) consider a particular sequence α N = γ/ ( γ + N ) , where γ is a coarsening parameter.
The interpretation of this particular sequence is that the posterior will, for any number N of observations, behave as though there were roughly γ number of data points.
Additionally, characterizing the behavior of the α N -posterior number of components for other rates of convergence of α N → 0 beyond the one proposed by Miller and Dunson (2019) remains an open question.
Setup and assumptions in Theorem 2.
This section makes the details of the modeling setup and each of the conditions in Theorem 2. precise.
3.1 Notation and setup
Let X and Θ be Polish spaces for the observations and parameters, respectively, and endow both with their Borel σ -algebra.
For a topological space ( · ) , let C ( · ) be the bounded continuous functions from ( · ) into R , and P ( · ) be the set of probability measures on ( · ) endowed with the weak topology metrized by the Prokhorov distance d (Ghosal and van der Vaart, 2017, Appendix A, p. 508), and Borel σ -algebra.
We use f i ⇒ f and f i ⇐⇒ f (cid:48)
i to denote lim
i →∞ d ( f i , f ) =
and lim i →∞ d ( f i , f (cid:48) i ) = 0 , respectively, for f i , f (cid:48)
i , f ∈ P ( · ) .
We assume that the family of distributions Ψ = { ψ θ : θ ∈ Θ } is absolutely continuous with respect to a σ -ﬁnite base measure µ , i.e., ψ θ (cid:28) µ for all θ ∈ Θ , and that for measurable A ⊆ X , ψ θ ( A ) is a measurable function on Θ .
Deﬁne the measurable mapping F : P (Θ) → P ( X ) from mixing measures to mixtures of Ψ , F ( g ) = (cid:82) ψ θ dg ( θ ) .
Let G be the set of atomic probability measures on Θ with ﬁnitely many atoms, and let F be the set of ﬁnite mixtures of Ψ .
In the Bayesian ﬁnite mixture model from Section 2, a mixing measure g ∼ Π is generated from a prior measure Π on G , and f = F ( g ) is a likelihood distribution.
The α -posterior distribution on the mixing measure is
[Equation]
where dfdµ is the density of f = F ( g ) with respect to µ and α > 0 .
The α -posterior on the mixing measure g ∈ G induces an α -posterior on the number of components
k ∈ N by counting the number of atoms in g ,
and it also induces a posterior on mixtures f ∈ F via the pushforward through the mapping F .
We overload the notation Π ( α ) ( · | X 1: N ) to refer to all of these α -posterior distributions and Π( · ) to refer to prior distributions; the meaning should be clear from context.
3.2 Model assumptions
The ﬁrst assumption of Theorem 2.1 is that while the true data-generating distribution f 0 is not contained in the model class f 0 / ∈ F , it lies on the boundary of the model class.
In particular, we assume f 0 is in the KL-support of the prior Π .
Denote the Kullback-Leibler (KL) divergence between probability measures f 0 and f as
[Equation]
Assumption 3.1.
For all (cid:15) > 0 , the prior distribution Π satisﬁes
[Equation]
We use Assumption 3.1 in the proof of Theorem 2.1 primarily to ensure that the Bayesian posterior is consistent for f 0 .
Note that Assumption 3.1 is fairly weak in practice.
Intuitively, it just requires that the family Ψ is rich enough so that mixtures of Ψ can approximate f 0 arbitrarily well, and that the prior Π places sufﬁcient mass on those mixtures close to f 0 .
For Bayesian mixture modeling, Ghosal et al.
(1999, Theorem 3), Tokdar (2006, Theorem 3.2), Wu and Ghosal (2008, Theorem 2.3), and
Petralia et al.
(2012, Theorem 1) provide conditions needed to satisfy Assumption 3.1.
The second assumption of Theorem 2.1 is that the family of component distributions Ψ is wellbehaved.
This assumption has three stipulations.
First, the mapping θ
(cid:55)→ ψ θ must be continuous; this condition essentially asserts that similar parameter values θ must result in similar component distributions ψ θ .
Deﬁnition 3.2.
The family Ψ is continuous if the map θ (cid:55)→ ψ θ is continuous.
Second, the family Ψ must be mixture-identiﬁable , which guarantees that each mixture f ∈ F is associated with a unique mixing measure G ∈ G .
Deﬁnition 3.3 (Teicher (1961, 1963)) .
The family Ψ is mixture-identiﬁable if the mapping F ( g ) =
(cid:82) ψ θ dg ( θ ) restricted to ﬁnite mixtures F : G → F is a bijection.
In practice, one should always use an identiﬁable mixture model for clustering; without identiﬁability, the task of learning the number of components is ill posed.
And many models satisfy mixtureidentiﬁability, such as ﬁnite mixtures of the multivariate Gaussian family (Yakowitz and Spragins, 1968), the Cauchy family (Yakowitz and Spragins, 1968), the gamma family (Teicher, 1963), the generalized logistic family, the generalized Gumbel family, the Weibull family, and von Mises family (Ho and Nguyen, 2016, Theorem 3.3).
A number of authors (e.g., Chen (1995); Ishwaran et al.
Nguyen (2013); Ho and Nguyen (2016); Guha et al.
; Heinrich and Kahn (2018)) appeal to stronger notions of identiﬁability for mixtures than Deﬁnition 3.3.
But, to show posterior divergence in the present work, we do not require conditions stronger than Deﬁnition 3.3.
The third stipulation—that the family Ψ has degenerate limits —guarantees that a “poorly behaved” sequence of parameters ( θ i ) i ∈ N creates a likewise “poorly behaved” sequence of distributions ( ψ θ i ) i ∈ N .
This condition allows us to rule out such sequences in the proof of Theorem 2.1, and is the essential regularity condition to guarantee that a sequence of ﬁnite mixtures of at most k components cannot approximate f 0 arbitrarily closely.
Deﬁnition 3.4.
A sequence of distributions ( ψ i ) ∞
i =1 is µ -wide if for any closed set C such that µ ( C ) =
and any sequence of distributions ( φ i ) ∞
i =1 such that ψ
i ⇐⇒ φ
i , lim sup i →∞ φ
i ( C ) = 0 .
Deﬁnition 3.5.
The family Ψ has degenerate limits if for any tight, µ -wide sequence ( ψ θ i ) i ∈ N , we have that ( θ i ) i ∈ N is relatively compact.
The contrapositive of Deﬁnition 3.5 provides an intuitive explanation of the condition: as i → ∞ , for any sequence of parameters θ i that eventually leaves every compact set K ⊆ Θ , either the ψ θ i become “arbitrarily ﬂat” (not tight) or “arbitrarily peaky” (not µ -wide).
For example, consider the family Ψ of Gaussians on R with Lebesgue measure µ .
If the variance of ψ θ i shrinks as i grows, the sequence of distributions converges weakly to a sequence of point masses (not dominated by the
Lebesgue measure).
If either the variance or the mean diverges, the distributions ﬂatten out and the sequence is not tight.
We use the fact that these are the only two possibilities when a sequence of parameters is poorly behaved (not relatively compact) in the proof of Theorem 2.1.
These three stipulations together yield Assumption 3.6.
Assumption 3.6.
The mixture component family Ψ is continuous, is mixture-identiﬁable, and has degenerate limits.
Proof of Theorem 2.1 and extension to sequences α N 4.1 Proof of Theorem 2.
The proof has two essential steps.
The ﬁrst is to show that for any α ∈ (0 , 1] , the Bayesian posterior is weakly consistent for the mixture f 0 ; i.e., for any weak neighborhood U of f 0 the sequence of posterior distributions satisﬁes
[Equation]
Weak consistency for f 0 is guaranteed directly by Assumption 3.1 and the fact that Ψ is dominated by a σ -ﬁnite measure µ .
For α = 1 , Assumption 3.1 implies weak consistency for f 0 , i.e., Equation (3) (Ghosh and Ramamoorthi, 2003, Theorem 4.4.2).
For α ∈ (0 , 1) , if Assumption 3.1 holds, by Ghosal and van der Vaart (2017, Theorem 6.54), with f 0 -probability 1, for any Hellinger neighborhood V of f 0 and any α ∈ (0 , 1) , Π ( α )
( V | X 1: N ) N →∞ −→ 1 .
Thus, since V ⊆ U,
[Equation]
The second step is to show that for any ﬁnite k ∈ N , there exists a weak neighborhood U of fcontaining no mixtures of the family Ψ with at most k components.
Together, these steps show that the posterior probability of the set of all k -component mixtures converges to 0 f 0 -a.s.
as the amount of observed data grows.
We provide a proof of the second step.
To begin, note that Assumption 3.1 has two additional implications about f 0 beyond Equation (3).
First, f 0 must be absolutely continuous with respect to the dominating measure µ ; if it were not, then there exists a measurable set
A such that f 0 ( A ) > 0 and µ ( A ) = 0 .
Since µ dominates Ψ , any f ∈ F satisﬁes f ( A ) = 0 .
Therefore KL( f 0 , f ) = ∞ , and the prior support condition cannot hold.
Second, it implies that f 0 can be arbitrarily well-approximated by ﬁnite mixtures under the weak metric, i.e., there exists a sequence of ﬁnite measures f i ∈ F , i ∈ N such that f i ⇒ f
0 as i → ∞ .
This holds because (cid:113) 12 KL( f 0 , f ) ≥ TV ( f 0 , f [Table] ≥ d ( f 0 , f ) .
Now suppose the contrary of the claim for the second step, i.e., that there exists a sequence ( f i )
∞
i = of mixtures of at most k components from Ψ such that f i ⇒ f 0 .
By mixture-identiﬁability, we have a sequence of mixing measures g i with at most k atoms such that F ( g i ) =
f i .
Suppose ﬁrst that the atoms of the sequence ( g i ) i ∈ N either stay in a compact set or have weights converging to 0.
More precisely, suppose there exists a compact set K ⊆ Θ such that
[Equation]
Decompose each g i
=
g i,K + g i, Θ \ K such that g i,K is supported on K and g i, Θ \ K is supported on Θ \ K .
Deﬁne the sequence of probability measures ˆ g i,K
= g i,K g
i,K (Θ) for sufﬁciently large i such that the denominator is nonzero.
Then Equation (4) implies
[Equation]
Since Ψ is continuous and mixture-identiﬁable, the restriction of F to the domain G is continuous and invertible; and since K is compact, the elements of (ˆ g i,K ) i ∈ N are contained in a compact set G K ⊆ G by Prokhorov’s theorem (Ghosal and van der Vaart, 2017, Theorem A.4).
Therefore F ( G K ) =
F K is also compact, and the map F restricted to the domain G K is uniformly continuous with a uniformly continuous inverse by Rudin (1976, Theorems 4.14, 4.17, 4.19).
Next since F (ˆ g i,K ) ⇒ f 0 , the sequence F (ˆ g i,K ) is Cauchy in F K ; and since F − 1 is uniformly continuous on F K , the sequence ˆ g i,K must also be Cauchy in G K .
Since G K is compact, ˆ g i,K converges in G K .
Lemma A.1 below guarantees that the convergent limit g K is also a mixing measure with at most k atoms; continuity of
F implies that F ( g K ) =
f 0 , which is a contradiction, since by assumption f 0 is not representable as a ﬁnite mixture of Ψ .
Now we consider the remaining case: for all compact sets K ⊆ Θ , g i (Θ \ K ) (cid:54)→ 0 .
Therefore there exists a sequence of parameters ( θ i ) ∞ i =1 that is not relatively compact
such that lim sup i →∞ g
i ( { θ i } ) > 0 .
By Assumption 3.6, the sequence ( ψ θ i ) i ∈ N is either not tight or not µ -wide.
If ( ψ θ i ) i ∈ N is not tight then f i =
F ( g i ) is not tight, and by Prokhorov’s theorem f i cannot converge to a probability measure, which contradicts f i ⇒ f 0 .
If ( ψ θ i ) i ∈ N is not µ -wide then f i =
F
( g i ) is not µ -wide.
Denote ( φ i ) i ∈ N to be the singular sequence associated with ( f i ) i ∈ N and C to be the closed set such that lim
sup i →∞ φ i ( C ) > 0 , µ ( C ) = 0 , and φ i ⇐⇒ f
i per Deﬁnition 3.4.
Since f 0 (cid:28) µ , f 0 ( C ) = 0 .
But f i ⇒ f 0 implies that φ i ⇒ f 0 , so lim sup i →∞ φ
i ( C ) = f 0 ( C ) = 0 by the Portmanteau theorem (Ghosal and van der Vaart, 2017, Theorem A.2).
This is a contradiction.
Experiments
For all experiments below, we use a ﬁnite mixture model with a Gaussian component family and a conjugate prior.
In particular, consider number of components k , mixture weights p ∈ R k , Gaussian component precisions τ ∈ R k + and means θ ∈ R k , labels Z ∈ { 1 , . .
. , k } N , and data X ∈ R N .
Then the probabilistic generative model is
[Equation]
where j ranges from 1 , .
k , and n ranges from 1 , . .
. , N .
For posterior inference, we used a Gibbs sampler (Miller and Harrison, 2018, Sec.
7.2.2, Algorithm 1), coupled with the approximation described in Miller and Dunson (2019, Section 5).
Note that we use this model primarily to illustrate the problem of α -posterior divergence under model misspeciﬁcation; it should not be interpreted as a carefully-speciﬁed model for the data examples that we study.
Also note that while the empirical examples below involve Gaussian FMMs, our theory applies to a more general class of component distributions.
In this section, we consider the behavior of the α -posterior only for ﬁxed α ∈ (0 , 1) .
In Appendix C.2, we show the α N -posterior number of components for α N → α ∈ (0 , 1) exhibits similar divergent behavior.
5.1 Synthetic mixture data
Here we study the effects of varying data set sizes under a misspeciﬁed model.
We generated data sets of increasing size N ∈ { 50 , 100 , 500 , 1000 , 5000 , 10000 } from a 2-component Laplace mixture models, where the 2-component distributions have means ( − 5 , 5) , scales (1 . 5 , 1) , and mixing weights [Figure]
We generated the sequence of data sets such that each was a subset of the next, larger data set in the sequence.
Following Miller and Harrison (2018, Section 7.2.1), we set the hyperparameters of the Bayesian ﬁnite mixture model as follows: m = 12 (max n ∈ [ ˜ N ] X n + min n ∈
[ ˜ N ] X n ) where ˜ N = 10 , 000 , κ = (max n ∈ [ ˜ N ] X n − min n ∈ [ ˜ N ] X n ) − 2 , α = 2 , r = 0 .
1 , γ = 1 , and β = 1 .
We ran a total of 150,000 Markov chain Monte Carlo iterations per data set; we discarded the ﬁrst 50, iterations as burn-in.
In Figure 1, we show the α -posterior number of components resulting from ﬁxed α = 1 , 0 . 8 , 0 . 5 , 0 . 2 .
Note that α = 1 is the usual posterior distribution on the number of components.
The ﬁgures show that as α decreases, the posterior mass tends to shift to smaller numbers of components and also becomes less concentrated.
However, as in the usual posterior ( α = 1 ), the α -posterior still diverges, though more slowly with lower values of α .
5.2 Galaxy mixture data
Mixture models are used in astronomy to characterize stellar populations (Nemec and Nemec, 1991), including analysis of star and galaxy clusters.
We study the Shapley galaxy data set (Drinkwater et al., 2004), which contains measurements of redshifts (i.e., velocities in km/s) for 4215 galaxies in the Shapley Concentration regions.
To examine the effect of increasing data set size on inferential results, we randomly sampled subsets of increasing size without replacement with N ∈ { 100 , 200 , 500 , 1000 , 2000 , 4215 } ; each smaller subset was contained in the next larger data set.
We set the hyperparameters of the Bayesian ﬁnite mixture model as follows: m = 12 (max n ∈ [ ˜ N ] X n + min n ∈
[ ˜ N ] X n ) where ˜ N = 4215 , κ = (max n ∈ [ ˜ N ] X n − min n ∈
[ ˜ N ] X n ) − 2 , α = 2 , r = 0 .
1 , γ = 1 , and β = 1 .
We ran a total of 100,000 Markov chain Monte Carlo iterations per data set; we discarded the ﬁrst 50,000 iterations as burn-in. [Figure] [Figure]
The α -posterior number of components for this model is displayed in Figure 2.
For α = 1 , we ﬁnd that as we examine more and more data, the posterior diverges.
Similar behavior occurs with fractional α -posteriors.
With smaller values of α , the α -posterior diverges more slowly.
Discussion and future work
We have shown that the posterior distribution for the number of components in ﬁnite mixtures diverges when the mixture component family is misspeciﬁed.
Our results show that this divergence holds even when using a power posterior, with ﬁxed power between 0 and 1, for robustness.
It follows that, with either a standard or power posterior, inferential results on the number of components will change substantively with more data, no matter how much data has been analyzed so far.
This divergence calls into question the usefulness of FMMs, and power posterior FMMs, in applications.
A number of open questions remain.
We here point to similar power-posterior behavior even when the power changes in N but converges to a constant in (0 , 1) .
However, Miller and Dunson (2019) consider powers that converge to 0 in the limit of N → ∞ .
It remains to investigate this case, especially for more general sequences of powers converging to zero, beyond the particular sequence suggested by Miller and Dunson (2019).
Because our analysis is inherently asymptotic, it is possible that the α -posterior on the number of components may still provide useful inferences for a ﬁnite sample — even when the power is ﬁxed or converging to a non-zero constant.
But our results suggest care would need to be taken to account for the inferential dependence on data size.
A Proofs
A.1 Additional details for the proof of Theorem 2.
Lemma A.1.
Suppose φ, ( φ i ) i ∈ N are Borel probability measures on a Polish space such that φ i ⇒ φ and sup i | supp φ i | ≤ k ∈ N .
Then | supp φ | ≤ k .
Proof.
Suppose | supp φ | > k .
Then we can ﬁnd k + 1 distinct points x 1 , . . .
, x k +1 ∈ supp φ .
Pick any metric ρ on the Polish space, and denote the minimum pairwise distance between the points 2 (cid:15) .
Then for each point j = 1 , . . .
, k + 1 deﬁne the bounded, continuous function
h j ( x ) = 0 ∨ (cid:0) 1 − (cid:15) − 1 ρ ( x, x j ) (cid:1) .
Since x j ∈ supp φ , we have that (cid:82) h
j dφ > 0 .
Weak convergence φ i ⇒ φ therefore implies min j =1 ,...,k +1 lim inf i →∞ (cid:82) h
j dφ i > 0 .
But the h j are nonzero on disjoint sets, and each φ i only has k atoms; the pigeonhole principle yields a contradiction.
A.2 Proof of Corollary 2.
The only difference of the proof of Corollary 2.2 with the proof of Theorem 2.1 is the ﬁrst step, i.e., weak consistency of the α N -posterior for f 0 .
Here we provide a proof of the ﬁrst step, which is a straightforward generalization of Ghosal and van der Vaart (2017, Theorem 6.54).
Let p :=
dfdµ and p 0 :
= df 0 dµ denote the density of f and f 0 with respect to µ , respectively.
For any weak neighborhood U of f 0 , we can express the α N -posterior as
[Equation]
Since f 0 is in the KL support of the prior Π , by Lemma A.2, for any β > 0 and α N → α ∈ (0 , 1) , the denominator satisﬁes
[Equation]
Suppose V is a Hellinger neighborhood of f 0 .
Then by Lemma A.3 and since V c ⊇ U c , for some β 0 > 0 , with f 0 -probability 1,
[Equation]
Finally, choose β = β 0 , and so with f 0 -probability 1, Equation (5) goes to 0 as N → ∞ .
We now prove the lemmas used for the proof above.
Lemma A.2.
Suppose that f 0 is in the KL support of the prior Π and and α N → α ∈ (0 , 1) as
N
→ ∞ , then for any β > 0 ,
[Equation]
Proof.
Let K (cid:15) :
= { f : KL( f 0 , f ) < (cid:15) } .
First note that by monotonicity, the strong law of large numbers, and the KL condition, we can bound the integral as follows
[Equation]
Thus, Ghosh and Ramamoorthi (2003, Lemma 4.4.1) implies the the result.
Lemma A.3.
Let α N → α ∈ (0 , 1) as N → ∞ .
Then for any Hellinger neighborhood V of f 0 and for some β 0 > 0 ,
[Equation]
Proof.
Denote the Hellinger transform as ρ α ( p, p 0 ) :
= (cid:82) p α p 1 − αdµ .
By Fubini’s theorem and because f 0 ( p/p 0 )
α
= ρ α ( p, p 0 ) , we have that
[Equation]
The inequality above follows because for any α ∈ (0 , 1) , min( α, 1 − α )
d 2 H ( p, p 0 ) ≤ − log p
α ( p, p 0 ) , where d H denotes the Hellinger distance (Ghosal and van der Vaart, 2017, Lemma B.5) and because for any p ∈ V c , d 2 H ( p, p 0 ) ≥ (cid:15) 2 .
Applying Markov’s inequality and Borel-Cantelli implies that for some β 0 > 0 , f
[Equation]
and so the conclusion holds.
A.3 Extension to priors that vary with N
Our main result (i.e., Theorem 2.1) applies to the setting of a ﬁxed prior Π .
However, it is often natural to specify a prior distribution that changes with N (e.g., Roeder and Wasserman (1997),
Richardson and Green (1997), and Miller and Harrison (2018, Section 7.2.1)).
Corollary A.5 below demonstrates that a result nearly identical to Theorem 2.1 holds for priors that are allowed to vary with N , provided that f 0 is in the KL-support of the sequence of priors Π N .
The only difference is that our result in this case is slightly weaker: we show that the posterior number of components diverges in probability rather than almost surely.
Assumption A.4.
For all (cid:15) > 0 , the sequence of prior distributions Π N satisﬁes
[Equation]
Corollary A.5.
Suppose in the setting of Theorem 2.1 we replace Assumption 3.1 with Assumption A.4.
Then for any α ∈ (0 , 1] , the α -posterior on the number of components diverges in f 0 -probability: i.e., for all k ∈ N ,
[Equation]
Proof.
Since for any (cid:15) > 0 , lim inf N →∞ Π N ( f : KL( f 0 , f ) < (cid:15) ) > 0 , Ghosal and van der Vaart (2017, Theorem 6.54, Lemma 6.26) imply that the α -posterior is weakly consistent at f 0 in probability: i.e., for any weak neighborhood U of f 0 ,
[Equation]
Assumption A.4 also implies that for sufﬁciently large N , f 0 is a weak limit of ﬁnite mixtures in F .
The remainder of the proof is identical to that of Theorem 2.1.
B Example component families
Consider the multivariate Gaussian family Ψ
= (cid:8) N ( ν, Σ) : ν ∈ R d , Σ ∈ S d ++ (cid:9) with parameter space Θ = R d × S d ++ , equipped with the topology induced by the Euclidean metric.
Let ( λ j (Σ))
dj
= denote the eigenvalues of the covariance matrix Σ ∈ S d ++ that satisfy ∞ > λ 1 (Σ) ≥ . .
≥ λ d (Σ) > 0 .
Since the family of Gaussians is continuous and mixture-identiﬁable (Yakowitz and Spragins, 1968, Proposition 2), the main condition we need to verify is that the family has degenerate limits (Deﬁnition 3.5).
A useful fact is that if a sequence of Gaussian distributions is tight, then the sequence of means and the eigenvalues of the covariance matrix is bounded.
Lemma B.1.
Let ( ψ i ) i ∈ N be a sequence of Gaussian distributions with mean ν i ∈ R d and covariance Σ i ∈ S
d ++ .
If ( ψ i ) i ∈ N is a tight sequence of measures, then the sequences ( ν i ) i ∈ N and ( λ 1 (Σ i ))
i ∈ N are bounded.
Proof.
Let Y i denote a random variable with distribution ψ i .
For each covariance matrix Σ i , consider its eigenvalue decomposition Σ
i =
U i Λ i
U (cid:62) i , where U i ∈
R d × d is an orthonormal matrix and Λ
i ∈ R
d × d is a diagonal matrix.
Then the random variable Z i
=
U (cid:62) i
Y i has distribution N
( U (cid:62) i
ν
i , Λ i ) .
If either (cid:107) ν
i (cid:107)
= (cid:107) U (cid:62) i
ν i (cid:107) 2 is unbounded or (cid:107)
Λ i (cid:107) F is unbounded, then Z i is not tight (Billingsley (1986, Example 25.10)).
Since Z i and Y i lie in any ball centered at the origin with the same probability, Y i is not tight.
Proposition B.2.
Let Ψ = (cid:8)
N
( ν, Σ) :
ν ∈ R d , Σ ∈ S d ++ (cid:9) be the multivariate Gaussian family, where S d ++ := { Σ ∈ R d × d : Σ = Σ (cid:62) , Σ (cid:31) 0 } is the set of d × d symmetric, positive deﬁnite matrices.
Then Ψ satisﬁes Assumption 3.6.
We now show that the multivariate Gaussian family has degenerate limits.
Proof of Proposition B.2.
If the parameters ( θ i ) i ∈ N are not a relatively compact subset of Θ , then either some coordinate of the sequence of means ν i diverges, λ 1 (Σ i ) → ∞ , or λ d (Σ i ) → 0 .
If some coordinate of the mean ν i diverges or the maximum eigenvalue diverges, i.e., λ 1 (Σ i ) → ∞ , then the sequence ( ψ θ i ) is not tight by Lemma B.1.
On the other hand, if λ d (Σ i ) → 0 as i → ∞ , then ψ θ i converges weakly to a sequence of degenerate Gaussian measures that concentrate on
C i = (cid:8) x ∈ R d : ( x − ν i ) (cid:62) u d,i = 0 (cid:9) , where u d,i is the d th eigenvector of Σ i .
Note that µ ( C i ) =
for Lebesgue measure µ ; so if we deﬁne C =
∪ i C i in the setting of Deﬁnition 3.4, the sequence is not µ -wide.
We can generalize Proposition B.2 beyond multivariate Gaussians to mixture-identiﬁable locationscale families, as shown in Proposition B.3.
Examples of such families include the multivariate
Gaussian family, the Cauchy family, the logistic family, the von Mises family, and generalized extreme value families.
The proof is similar to that of Proposition B.2.
Proposition B.3.
Suppose Ψ is a location-scale family that is mixture-identiﬁable and absolutely continuous with respect to Lebesgue measure µ , i.e.,
[Equation]
where ϕ : R d → R is a probability density function.
Then Ψ satisﬁes Assumption 3.6.
C Additional experiments C.1 Gaussian mixture data
We generated data sets of increasing size N ∈ { 50 , 100 , 500 , 1000 , 5000 , 10000 } from 2-component univariate Gaussian mixture models with means ( − 5 , 5) , scales (1 . 5 , 1) , and mixing weights (0 . 4 , 0 . 6) .
In Figure 3, we see that the posterior number of components concentrates on the generating number of components, and for α = 0 . 2 , the posterior is less concentrated when there is less data.
C.2 Results for α N → α
Using the same data of Section 5, we examined the behavior of the α N -posterior number of components.
Below, we consider sequences α N =
(1 − 1 /N )
α , where α ∈ { 1 , 0 . 8 , 0 . 5 , 0 . 2 } .
We plot the α N -posterior number of components for the Laplace mixture data in Figure 4 and the galaxy data in
Figure 5.
As in the ﬁxed α case, we observe that the α N posterior number of components diverges.
C.3 Gene expression data
Computational biologists are interested in classifying cell types by applying clustering techniques to gene expression data (Yeung et al., 2001; Medvedovic and Sivaganesan, 2002;
McLachlan et al., 2002; Medvedovic et al., 2004; Rasmussen et al., 2008; de Souto et al., 2008; McNicholas and
Murphy, 2010).
[Figure] [Figure]
In this section, we examine the behavior of the usual posterior ( α = 1 ) on gene expression data using a mixture of multivariate Gaussians with axis-aligned covariances.
In particular, consider number of components k , mixture weights p ∈ R k , Gaussian component precisions τ ∈ R k × D + and means θ ∈ R k × D , labels Z ∈ { 1 , . .
. , k } N , and data X ∈ R N × D .
Then the probabilistic generative model is
[Equation]
[Figure] [Figure]
where j ranges from 1 , .
, k , d ranges from 1 , . . .
, D , and n ranges from 1 , . . .
, N .
In our next set of experiments, we apply the Gaussian ﬁnite mixture model to two gene expression data sets: (1) single-cell RNA sequencing data from mouse cortex and hippocampus cells (Zeisel et al., 2015) with the same feature selection as Prabhakaran et al.
( N = 3008 , D = 558 , [-11,
-]11,
Gibbs sampling steps with 1,000 of those as burn-in) and (2) mRNA expression data from human lung tissue (Bhattacharjee et al., 2001)
( N = 203 , D = 1543 , and 10,000 Gibbs sampling steps with 1,000 of those burn-in).
Our experiments here represent a simpliﬁed version of previous mixture model analyses for these and other related data sets (de Souto et al., 2008; Prabhakaran et al., 2016;
Armstrong et al., 2001; Miller and Harrison, 2018).
For both data sets, we used hyperparameters α = 1 , β = 1 , m = 0 , κ jd = τ jd , r = 0 .
1 , and γ = 1 .
As these gene expression data sets contain counts, we ﬁrst transformed the data to real numerical values.
In particular, we used a base-2 log transform followed by standardization—such that each dimension of the data had zero mean and unit variance—per standard practices (e.g., Miller and
Harrison (2018)).
Then to examine the effect of increasing data set size on inferential results, we randomly sampled subsets of increasing size without replacement; each smaller subset was contained in the next larger data set.
For the single-cell RNAseq data set, the posterior on the number of components is shown in Figure 6a.
Here the ground truth number of clusters is captured when the data set size is N = 100 .
But as predicted by our theory, as we increase the number of data points, the posterior number of components diverges.
The α -posterior on the number of components for the lung gene expression data is shown in Figure 6b.
Again we ﬁnd that on the smallest data subsets, the posterior appears to capture the ground truth number of clusters, but that as we examine more and more data, the posterior diverges.
While diagonal covariance Gaussian components are likely not rich enough to model the cluster shapes, our purpose here is to capture the effect of model misspeciﬁcation on the posterior on the number of components.
Thus, these examples suggest the need for more robust analyses.