Double trouble: Predicting new variant counts across two heterogeneous populations
Introduction
Collecting genomics data across multiple heterogeneous populations has the potential to improve our understanding of disease.
For instance, we might compare cases and controls to discover biological underpinnings of disease and ultimately develop treatments.
Or we might compare multiple cancer types to understand etiology and develop more broadly-applicable therapies.
Despite sequencing advances, though, resources remain a constraint in how much data can be gathered.
If we could predict the amount of new information to be gained from different potential amounts of data, that could guide experimental design to make optimal use of the resources at hand.
And while many authors have developed prediction methods for the single-population case [Masoero et al., 2022,
Chakraborty et al., 2019, Zou et al., 2016, Gravel and National Heart Lung & Blood Institute (NHLBI)
GO Exome Sequencing Project, 2014, Ionita-Laza et al., 2009], that work will generally provide poor-quality predictions across multiple heterogeneous populations when these populations can share information.
We develop a new methodology to provide accurate predictions in the face of multiple populations.
More precisely, we predict the number of (genetic) variants, i.e., differences in an observed genome relative to some reference genome.
Variants have the potential to severely disrupt gene function and are implicated in many diseases [Tennessen et al., 2012, Cargill et al., 1999, 1000 Genomes Project
Consortium, 2015].
We consider the case where the experimenter has collected (limited) pilot data on two populations.
The experimenter is planning a follow-up experiment of a particular size and wishes to predict (1) the number of new variants shared across the two populations or (2) the total number of new variants.
Here, “new” refers to variants not already discovered in either population in the pilot experiment, and size is determined by the number of samples in each population.
For instance, the shared variants predicted in task (1) might be used to help elucidate a common genetic basis across cancers [Rafnar et al., 2009, Bojesen et al., 2013, Rashkin et al., 2020].
Or a biologist might be interested in task (2) if they plan to compare cases and controls in a rare variant association study [Zuk et al., 2014].
Related work.
Many authors have addressed the problem of predicting the number of new variants in a follow-up experiment of a particular size given a pilot experiment – when samples in both cases arise from a single, homogeneous population [Masoero et al., 2021, Chakraborty et al., 2019, Zou et al., 2016, Gravel and National Heart Lung & Blood Institute (NHLBI)
GO Exome Sequencing
Project, 2014, Ionita-Laza et al., 2009].
One option for task (2) is to treat the two populations as a single population and apply existing techniques.
But we will see in our experiments that, if the populations are heterogeneous, the relative ratios of the two populations in the follow-up affects the number of expected variants.
Similarly, while we might use existing methods for task (1) by treating shared variants as the only variants, doing so cannot account for the differential impact of more samples in one or the other population.
Our contribution.
We provide a new methodology (Section 2) for predicting the number of shared and total variants in a follow-up experiment from a pilot experiment when there are two potentiallyheterogeneous populations.
We provide theory to show that a seemingly natural extension of a state-of-the-art technique [Masoero et al., 2022] to independently model the variant frequencies in two populations fails for surprisingly fundamental reasons.
To still use ideas from Bayesian nonparametrics (BNP) as in [Masoero et al., 2022], we need to develop a new BNP model, not previously seen in any literature.
We demonstrate in experiments on real cancer data and simulations that our method provides accurate predictions (Section 3).
Setup and our model
Setup.
We consider two populations.
Let N i be the number of samples in population i ∈ { 1 , 2 } in the pilot experiment.
Let N =
( N 1 , N 2 ) , and let J N be the total number of unique variants across both sets of samples.
Let ψ j be a (unique) label for the j th unique variant.
Let x i,n,j equal 1 if the n th sample in population i exhibits the j th variant and 0 otherwise.
Let δ ψ represent a discrete mass of size 1 at location ψ .
So the measure X i,n = (cid:80)
J N j
=1 x i,n,j δ ψ j collects the variant information for sample n in population i by pairing each variant indicator x i,n,j with its variant label ψ j .
Equivalently, we can write X i,n = (cid:80) ∞
j =1 x i,n,j δ ψ j if we set x i,n,j = 0 for an imagined countable inﬁnity of possible future variants ψ j .[-
-]
We consider experimenters planning a follow-up study that will collect M i samples from population i ; let M =
( M 1 , M 2 ) .
We would like to predict the total number of new variants found in the follow-up given the pilot; call that number U
[Equation]
We would also like to predict the number of shared variants across the populations.
In fact, we predict the number of new variants appearing k 1 times in population 1 and k 2 times in population 2; call that number U ( M,k ) N
where k = ( k 1 , k 2 ) :
[Equation]
We can sum up across values of k 1 > 0 and k 2 > 0 to ﬁnd the total number of shared variants.
Our model and theoretical results.
Like all single-population methods [Masoero et al., 2022,
Chakraborty et al., 2019, Zou et al., 2016, Gravel and National Heart Lung & Blood Institute (NHLBI)
GO Exome Sequencing Project, 2014, Ionita-Laza et al., 2009], we ignore linkage disequilibrium.
We assume that samples within a single population are exchangeable.
It is natural then to assume the existence of an unknown θ i,j that represents the frequency of variant j in population i .
Let θ j = ( θ 1 ,j , θ 2 ,j ) be the vector collecting variant frequencies across populations for variant j .
We can further collect the frequencies across variants in a random vector-valued measure Θ = (cid:80) ∞ i =1 θ j δ ψ j .
We assume x i,n,j ∼ Bernoulli ( θ i,j ) independently across i, j, n .
We take a Bayesian approach, so it remains to put a prior on Θ .
Analogous to the principles from [Masoero et al., 2022, James, 2017, Broderick et al., 2018] for one population, we have the following desiderata for our prior across two populations: (A) the number of variants observed in any single sample should be almost surely (a.s.) ﬁnite, and (B) the number of latent variants should be countably inﬁnite (to ensure we never hit an upper bound during our analysis); more precisely for desideratum (B), we assume each of the two elements of each θ j in the countably inﬁnite sum above is strictly between 0 and 1.
In the one-population case, Masoero et al.
[2022] satisfy similar desiderata with a three-parameter beta process (3BP) prior [Masoero et al., 2022, James, 2017, Broderick et al., 2012, Teh and Gorur, 2009] on the single-population analogue of Θ .
One might think to put independent 3BP priors on the variant frequencies in the multiple population case.
But, surprisingly, the coupling induced via the sharing of variant labels ψ j makes it impossible to use any independent priors on the two sets of population frequencies (within a Poisson point process framework) while simultaneously meeting the desiderata.
We provide a precise theorem statement and a proof for this new result in Appendix S8.
However, we show that an alternative approach can work.
Namely, we propose that the θ j be generated from a Poisson point process with rate measure that does not factorize across the two populations: ν (
[Equation]
=
[Equation]
θ with hyperparameters γ, α, σ 1 , σ 2 , c 1 , c 2 .
We show that this prior satisﬁes our desiderata when γ, c 1 , c 2 > 0 , σ 1 , σ 2 > 0 , and α ∈ [ σ 1 + σ 2 , σ 1 + σ 2 + 1) ; for details, see Proposition S4 in Appendix S7.
The 3BP enjoys useful conjugacy properties; in particular, when the hyperparameters are known, its convenient form allows a closed-form prediction for the number of new variants in a follow-up.
Our prior is similar in structure; it resembles a beta distribution in each dimension except for the leading factor ( θ 1 + θ 2 ) − α .
But due to this leading factor, there is no longer a convenient closed form for our prediction. Nonetheless, we can use numerical approximations, which we detail in Appendix S4.
Our method for ﬁtting and making predictions.
Whether we are predicting the number of new variants or new shared variants in a follow-up, we start by learning the hyperparameters of our model by using an empirical Bayes technique with the pilot data; see Appendix S2 for details.
The hyperparameters control the behavior of the variant frequencies in Θ and hence the overall growth rate of variants and shared variants.
With these hyperparameters in hand, we can then make predictions for the follow-up; see Appendix S1 for details of how we perform prediction in each task.
Synthetic and real data experiments
We next demonstrate empirically that our method is able to predict the total number of new variants, or particular combinations of new variants, across two populations in a follow-up study – given data from a pilot study across the same two populations.
Data.
We simulate data according to our own model and check our recovery in Appendix S6.
For real data, we use the MSK-IMPACT dataset [Cheng et al., 2015].
While the full data provides samples from patients with 341 different cancer types, we here restrict our attention to the two most prevalent types in the data to form our two populations: Lung Adenocarcinoma (749 individuals) and Breast
Invasive Ductal Carcinoma (1,153 individuals).
See Appendix S3 for more details.
Comparisons.
We compare our new approach to two baselines that we derive from a state-of-the-art single-population approach [Masoero et al., 2022], which uses a three-parameter beta process (3BP) model.
In particular, we let d3BP (for “(fully) dependent 3BP”) represent the prediction we get by treating the two populations as a single population.
And we let i3BP (for “independent 3BP”) represent the case where we treat the two populations separately and assume no shared variants.
Experimental setup.
For each i , we choose N i points uniformly at random from the full available data set for the i th population.
Together, these samples form the pilot study.
We consider predicting the number of new variants in a follow-up study with M i variants in the i th follow-up population.
We take the follow-up samples chosen uniformly at random from the remaining samples not already chosen for the pilot.
We describe the details of how we ﬁt the hyperparameters of each model, including the comparison models, in Appendix S2.
Predicting the total number of new variants.
In Figure 1A, we consider a pilot of size ( N 1 , N 2 ) =
In Appendix S6, we perform the analogous experiment but with different sizes ( N 1 , N 2 ) for the pilot – including varying proportions across the two populations.
The horizontal axis of
Figure 1A shows the total number of samples across both pilot and follow-up.
Everything before 200 samples represents the (known) pilot data (shaded gray).
Beyond 200 samples, we vary both the number of samples in the follow-up as well as the proportion from each population.
Concretely, we start by introducing samples entirely from population 1 (Lung Adenocarcinoma).
So our follow-up study has size ( M 1 , M 2 ) with M 2 = 0 and increasing M 1 until we reach M 1 = 649 .
At that point, we ﬁx M 1 = 649 and add samples only from population 2 (Breast Invasive Ductal Carcinoma) until we reach 1,702 total follow-up samples.
Each successive follow-up study contains the previous
Our method (red) tracks the ground truth closely.
The d3BP (blue) is often far from the ground truth since it is not able to model the two populations separately; thus it performs particularly poorly when the overall ratio of populations in the full observed data does not closely match the ratio in the pilot data – e.g., when the follow-up is dominated by population 1 near the righthand vertical dashed line.
The i3BP is able to capture the two different growth rates in the two populations; it likely performs well since there are few shared variants across these two populations relative to the total number of variants.
For each prediction method, the shaded area represents a 95% credible interval.
Predicting the number of new shared variants.
Next we consider a pilot of size ( N 1 , N 2 ) = (100 , 100) and a follow-up of size ( M 1 , M 2 ) =
(649 , 1053) chosen uniformly at random from the two cancer populations, as above.
We are interested in predicting how many new variants appear ( k 1 , k 2 ) times in the follow-up; this number of new variants forms the vertical axis in Figure 1B.
We set k 1 = 1 and vary k 2 across the horizontal axis of Figure 1B.
Our method’s prediction appears in red, and the prediction of the d3BP appears in blue; in both cases, the shaded area represents a 95% credible interval.
The i3BP does not model sharing of variants, so its predictions are effectivelythroughout.
We consider different combinations of ( k 1 , k 2 ) in Appendix S6.
Summary.
We have seen that the i3BP can perform well in predicting the total number of new variants (in particular, when there are relatively few shared variants across populations).
And the d3BP can perform well at predicting the number of new shared variants.
But our new method is the only one that consistently performs well across all tasks and in the presence of shared variants.
Discussion.
While our model naturally extends to more than two populations, computation may pose a practical challenge when the number of populations grows.
This challenge arises since our method requires numerical integration both when ﬁtting hyperparameters and when making predictions.
We hope to explore ideas to speed up these computations in future work.
Acknowledgments and Disclosure of Funding
YS and TB were supported in part by the DARPA I2O LwLL program, an NSF Career Award, and an
ONR Early Career Grant.
Appendix S Method for prediction
In the center of our method is to predict U M,kN for k = ( k 1 , k 2 ) and M =
( M 1 , M 2 ) while N =
( N 1 , N 2 ) , that is, number of new variants exist exactly k 1 times in population 1 and k 2 times in population 2 (in a follow up study of size M ).
Shared variants are those with k 1 , k 2 > 0 .
Predicting number of new variants in either population thus the discover curve can be done via the predicting of U M,kN and sum over k > 0 (see Corollary 1).
S1.1 Near conjugate prior
We assume hyperparameters for the near-conjugate prior is known.
We describe ﬁtting of those hyperparameters in Appendix S2.
Consider the variants exists k 1 times in population 1 and k 2 times in population 2 in the follow up sequencing of size M [-
-]1 and N 2 while did not appear in the pilot study with size N =
( N 1 , N 2 ) .
Formally, we consider
[Equation]
Proposition S1 (predicting U ( M,k ) N under near conjugate prior) .
With prior being the near-conjugate prior, the number of follow-up k-r tons
[Equation]
where
[Equation]
Where B ( a, b ) is the beta function, and E X,Y is the expectation with
[Equation]
independently.
We give the proof of this result in Appendix S4.
The number of new variants in in the follow-up given the pilot
[Equation]
Corollary 1 (Predicting number of new variants) .
With the prior being the near-conjugate model, the number of new variants seen with in the follow up study
[Equation]
Where λ ( M,k ) N ’s are given in Equation (S2) .
Proof.
Use the deﬁnition of U ( M ) N and results on sum of independent Poisson random variables.
S1.2 d3BP
Under d3BP, we essentially ignore the population label and use a 3BP to model the combined population following Masoero et al.
We use the same deﬁnition of kr-tons in Equation (S1).
Proposition S2 (predicting number of k-r tons under d3BP) .
[Equation]
Where
[Equation]
For number of variants, we follow Masoero et al.
[2021], which is equivalent as using Corollary 1.
S1.3 i3BP
Th i3BP model is not capable for predicting shared variants.
The number of new variants in each population can be predicted follow Masoero et al.
and we add up the new variants as a postprocessing.
S Method for ﬁtting the hyperparameter
Regardless of predicting task, we took a empirical Bayes approach to ﬁnd the hyperparameters ( γ, α, σ 1 , σ 2 , c 1 , c 2 ) for the near-conjugate prior, ( α, σ, c ) for d3BP, ( α 1 , α 2 , σ 1 , σ 2 , c 1 , c 2 ) for i3BP.
We used likelihood function on prediction of kr tons.
S2.1 Objective function on trimmed (cid:96) -tons
From the training set of size ( N 1 , N 2 ) , we can calculate the observed number of variants exist (cid:96) =
( (cid:96) 1 , (cid:96) 2 ) times u ( N,(cid:96) ).
We can treat these as the prediction when there is no pilot data.
From Proposition S1, we have the predictive distribution of u ( N,(cid:96) )of for different (cid:96) (and (cid:96) 1 + (cid:96) 2 > 0 ) they are independent Poisson with parameters λ ( N,(cid:96) )
(Equation (S2)).
To reduce computation cost we trim (cid:96) with some upper bound v such that (cid:96) 1 , (cid:96) 2 ≤ v (we set v=10 in the real data analysis).
Denote pois ( x | λ ) as the pmf of Poisson distribution with parameter λ
The objective function is given by the log likelihood
L
[Equation]
where λ ( N,(cid:96) )is given in Equation (S2)
Then we set the hyperparameter to be
[Equation]
We optimize the objective function using double annealing from scipy with default parameters.
S2.2 Fitting i3BP and d3BP with trimmed (cid:96) -ton’s
The d3BP essentially treat the two population as one.
We used the result from Masoero et al.
[2021] on the predictive distribution of (cid:96) -ton, i.e. the number of variants exist exactly (cid:96) s in a study with N s = N {+
+} + Nsamples times being independent Poisson distributed with mean λ (cid:96) s N s = α (cid:0) N s (cid:96) s (cid:1) (
[Equation] σ ) ( (cid:96)s − [Equation] ↑ ( c + σ ) ( Ns − (cid:96)s ) ↑ ( c +1)
( Ns − 1) ↑ .
For d3BP, we can calculate the number (cid:96) s -tons u (cid:96) s N s up to an upper bound (cid:96) ≤ v by ignoring the label of two populations.
The objective function for d3BP is then
[Equation]
For i3BP, we treat the two population independently and count u (cid:96) 1 1 ,N 1 , i.e. the (cid:96) 1 -tons in populationand u (cid:96) 2 2 ,N 2 (cid:96) 2 -tons in population 2 up to an upper bound v such that (cid:96) 1 , (cid:96) 2 ≤ v .
The predictive distribution for them are Poisson with expectation λ (cid:96) [-1 1 ,N 
-][Equation]
[Equation] N [Equation] =
[Equation] and λ [Equation] =
[Table]
respectively.
The objective function for i3BP is
[Table]
We optimize the objective function using double annealing from scipy with default parameters.
S Details on experiments S3.1 Synthetic example
In the synthetic example we sample the product Bernoulli process whose rate coming from d3BP and near conjugate prior.
We simulate 300 individuals for each population and take ( D, L ) = (30 , 30) , (30 , 50) , (50 , 30) , (50 , 50) from two populations as training and the rest as test sets.
Simulation methods was described in Appendix.S5.
We set the near conjugate prior’s hyperparameters to be γ = 150 , α = 1 , σ 1 = 0 . 2 , σ 2 = 0 . 8 , c 1 = c 2 = 20 and the hyperparameters for the d3BP to be α = 20 , σ = 0 .
5 , c = 10 to generate the synthetic dataset.
We ﬁt the hyperparameters for the near-conjugate prior, d3BP and i3BP prior using the empirical Bayes method described in Appendix S2.
We then make predictions based on the posterior distribution of new variants.
We calculate the posterior distribution of new variants in each follow-up size following the method described in Appendix S1 and add them to the variants number in the training sets.
S3.2 MSK-IMPACT example
In real data example, we ﬁrst take ( D, L ) =
(50 , 100) , (100 , 50) , (100 , 100) samples from the ﬁrst and second cancer types.
With this “pilot” sample, we 1) ﬁnd the hyperparameters for near-conjugate, d3BP and i3BP via empirical Bayes method described in Appendix S2 and 2) condition on the sample, we predict new variants and new shared variants seen in the follow up samples.
In ﬁt the hyperparameter for the near-conjugate prior, d3BP and i3BP prior using the empirical Bayes method described in Appendix S2.
We then make predictions based on the posterior distribution of new variants.
We calculate the posterior distribution of new variants in each follow-up size following the method described in Appendix S1 and add them to the variants number in the training sets.
In predicting the new shared variants we calculate the posterior follow (S2) (for near conugate) and Masoero et al.
[2021] using different k 1 , k 2 values.
S Details on predicting method S4.1 Near conjugate prior
Here we give the proof of Proposition S1.
Proof.
The posterior followed a thinned Poisson process with a rate measure: ν
[Equation]
Thus the posterior distribution of k, r ton’s are Poisson with expectation:
[Equation]
Where B ( a, b ) is the beta function, and E X,Y is the expectation with
[Equation]
independently.
The numerical double integral might be speed up via special function:
[Equation]
where 2 F 1 is the Gaussian hypergeometric function and B () is the Beta function.
Further we can transform 2 F 1 using the Euler type transformation
[Equation]
S4.2 d3BP
We give the proof of Proposition S2.
The proof largely follow Masoero et al.
[Equation] by realizing that the total number follows a 3BP and allocation of the variants in two population follows a hypergeometric distribution.
Proof.
The posterior followed a thinned Poisson process with a rate measure:
[Equation]
whose integral is the expectation, following Masoero et al.
[2021], this is given by
[Equation]
S4.3 i3BP
The i3BP method cannot predict number of kr-tons, but we still could use the model to predict number of new variants.
We treat two population independently model as 3BP’s following Masoero et al.
[2021] and add the number of new variants together to get the discover curve.
That is, suppose we sample M 1 individual from population 1 and then M 2 from population 2 in the follow up study and N 1 , N 2 individuals in the pilot, the predicted number of new variants at sample m 1 ( m 1 ≤ M 1 ) is Poisson with mean
[Equation]
and for all population 2 sample, the new variants seen after sampling population 1 is then added by the number seen in population 1, i.e. the predicted number of new variants at m 2 the sample of population 2 ( m 2 ≤ M 2 ) thus the m
+ M 1 th sample of the entire followup is Poisson with mean
[Equation]
S Details on sampling
The sampling follows the general principle in Broderick
S5.1 Near conjugate prior
Proposition S3 (Marginal representation of near-conjugate-Bernoulli process) .
Suppose we have x 1 ,n,k | Θ ∼ Bernoulli( θ 1 ,k ) and x 2 ,m,k | Θ ∼ Bernoulli( θ 2 ,k ) where Θ come from a near conjugate prior, we have
[Equation]
For
[Equation]
∼ Beta
[Equation]
independently.
Proof.
Follow the result in Broderick et al.
[2018], the marginal representation for the independent likelihood process with likelihood h 1 ( x 1 ,n,k | θ 1 ) and h 2 ( x 2 ,m,k | θ 2 ) is given by h cond ( x 1 ,n,k = x | x 1 , − n,k , x 2 , 1: m,k ) =
[Equation]
substitute h 1 ( x 1 ,n,k
| θ 1 ) by θ 1 and h 2 ( x 2 ,m,k | θ 2 ) by θ 2 and ν ( θ ) , we have
[Equation]
For
[Equation]
n − 1 i =1 x 1 ,i,k ) and X 2 ∼ Beta( σ 2 + (cid:80) mj
=1 x 2 ,j,k , c 2 + m − (cid:80) mj
=1 x 2 ,j,k ) independently.
S5.2 d3BP
For a d3BP-Bernoulli process of size ( M, N ) , we sample a 3BP-Bernoulli process following Masoero et al.
[2021] with size M + N and randomly split individual into population 1 of size M and population 2 of size N .
S More experiment results S6.1 Synthetic experiments
We present the discover curve of ﬁtting different generating process.
When the data was sampled from d3BP- Bernoulli process, the near conjugate prior perform at least as well as the d3BP model while i3BP tend to over estimate due to double counting (Fig.S2).
Similarly when the data was sampled from a near conjugate-Bernoulli process the near conjugate prior perform better than other two method we tested (Fig.S3).
S6.2 More sampling schemes in MSK-IMPACT data
When the training data has relatively small sample in population 2 all methods tend to over estimate the number of new variants at the latter stage but near-conjugate is close to the ground truth (Fig.S4).
In this case the i3BP has good performance because the shared variants are rare in cancer dataset.
S View as multivariate complete random measure
In a multivariate case, we consider an atom has vector valued frequencies, and we collect the atom-frequencies pair in a vector valued random measure as simply an analogues of (scalar valued) measure, consider some θ k ∈ R p + and x i,n,k ∈ { 0 , 1 , . . . }
qi =1 for a ﬁnite p, q , for K atoms that can be inﬁnite:
[Equation]
s.t.
we have x n,k ∼ H
(Π i dx i | θ )
Consider the ordinary components is a Poisson process with rate measure ν .
There are some requirements for this construction, we wish to have ﬁrst:
[Equation]
We also want at each coordinate of the feature vector, we see ﬁnitely many features when only ﬁnitely sampled.
For coordinate 1 ≤ j ≤ p , we want:
[Equation]
Equivalently, we can denote g ( x j | θ ) being the marginal likelihood of x j ,
[Equation]
The several method for two independent Bernoulli process, i.e. H ( x i,k | θ ) =
Ber ( θ i ) independent, can then be viewed in a similar way, i.e. a CRM whose ordinary component is a Poisson process on the unit square.
Example 1 (Single 3BP: diagonal model) .
The one 3BP ignoring label can be viewed as a “diagonal measure”.
I.e. we generate frequencies in population 1 follow a 1-d
3BP, then dictate the frequencies in the other population for the same variant to have the same value
[Equation]
Example 2 (Separate two 3BPs: axis model) .
In the model we have two separate 3BPs, we essentially assumed all variants are “private”, i.e. will only appear in one population, we can take each 3BP and have a δ measure product
[Equation]
Example 3 (Hierarchical-3BP by Masoero et al.
This model was proposed by Masoero et al.
[2021] as a prior of Bernoulli process.
In their formulation, f j ( θ j | θ ) takes a Beta form, that is
[Equation]
The inﬁnite mass can be shown by ﬁrst applying Tonelli theorem and then observing that the integral on θ j ’s are factorized and the overall integral is the integral of 3BP mass.
The ﬁnite expectation can be shown by evaluating the expectations
[Equation]
For any a 1 , b 1 > 0 the integral is ﬁnite since θν ( dθ ) is proper Beta distribution up to a constant scaling while the denominator b 1 + ( a 1 − b 1 ) θ is bounded.
Importantly we show our proposed method is proper
Proposition S4 (near conjugaet prior is proper) .
For the near conjugate prior with rage measure ν ( θ ) = γ ( θ 1 + θ 2 ) − α θ σ 1 − 1θ σ 2 − 1(1 − θ 1 ) c 1 − 1 (1 − θ 2 ) c 2 − 1 .
When γ, c 1 , c 2 > 0 , σ 1 , σ 2 > 0 and α ∈ [ σ 1 + σ 2 , σ 1 + σ 2 +1) , the prior satisﬁes our desiderata of inﬁnite mass and ﬁnite expectation on both axis, i.e. (cid:82)(cid:82) [0 , 1] 2 ν ( θ ) = ∞
while (cid:82)(cid:82) [0 , 1] 2 θ i ν ( d θ ) <
∞ for i = 1 , 2 .
Proof.
We take a small (cid:15) > 0 .
Deﬁne the set A = [0 , (cid:15) ] 2 .
Observe that we have θ σ 1 − {+
+}
[Equation]
θ 2 ) c 2 − 1 being a (unormalized) independent Beta densities.
We have (cid:90)(cid:90)
[Equation]
) − α θ σ 1 − 1θ σ 2 − 1(1 − θ
[Equation]
c 1 − 1 (1 −
[Equation] [Equation] c 2 − 1 d θ Consider the second term, we have I A c ( θ ) γ ( θ 1 + θ 2 ) − α ≤ γ(cid:15) − α < ∞ , thus (cid:90)
[Equation]
A
[Equation] [Equation] [Equation] where B is the beta function and the ﬁrst equal is by writing the integral as an expectation of two independent Beta distributions.
Similarly for the expectation requirement, without lose of generality we consider θ 1 , as our requirement and the rate measure is symmetric (cid:90)(cid:90)
[Equation]
Consider the second term, we have I A c ( θ ) γθ
[Equation]
θ [Equation] + θ
[Equation]
≤ γ(cid:15) − α < ∞ as well, thus use the same argument, we have
[Equation]
Thus, it sufﬁce to check under the parameter range, we have (cid:82)
A ν ( θ ) = ∞ and (cid:82)
A θ i ν ( θ ) < ∞ .
Observe that for c i > 0 , we have (1 − θ 1 ) c 1 − 1 , (1 − θ 2 ) c
2 − 1 being bounded on A , thus it sufﬁce to check (cid:82) A ( θ 1 + θ 2 ) − α θ
[Equation]
− 1θ
[Equation]
[Equation]
1d θ = ∞ and (cid:82)
A θ 1 ( θ 1 + θ 2 ) − α θ σ 1 −
[Equation]
d θ < ∞ .
For the ﬁrst requirement, consider the range where θ 1 ≥ θ 2 , we have
[Equation]
We used α ≥ σ 1 + σ 2 in the ﬁrst inequality.
Suppose α ≤ σ 1 + σ 2 + 1 − δ for an arbitrary δ > 0 , consider the expectation (cid:90)
[Equation]
[Table]
[Equation]
S Product measure in prior process
We can show that if the prior process is a product measure (not having δ measure) and the likelihood factorize in both data and parameters, then the prior cannot be properly normalized.
This result tells us that, in the two population problem, atoms’ behavior in the ﬁrst population must contain information about their behavior in the other population, either through likelihood or through prior.
Theorem S1 (Incompatibility between product measure, inﬁnite mass and ﬁnite expectations when likelihood factorizes) .
If the likelihood process are factorized for all dimensions (i.e. g ( x | θ ) = (cid:81) j g j ( x j | θ j ) ), then the prior process (rate measure ν ( d θ ) ) can not have all three properties: (1) being product measure (2) inﬁnite mass (3) ﬁnite positive expectations on all coordinates
Proof.
By assumption, we have (3) means:
[Equation]
Suppose we have (1) and (2).
By (1) we have ν ( d θ ) =
(cid:81) j ν j ( dθ j ) , i.e. a product measure, then by (2), we have there exist at least one coordinate k , such that (cid:82) ν
k
( dθ k ) = ∞ and all other coordinates are positive.
We show (3) is not true for coordinate j (cid:54) =
k :
[Equation]
The ﬁrst factor is positive and second is ∞ , while the last one is non-negative, thus the expectation can only be either 0 or ∞ .
Suppose we have (1) and (3), take coordinate j , we have:
[Equation]
We must have for all i (cid:54) =
j (cid:82) ν
i ( dθ i ) < ∞ , then we can take k (cid:54) =
j , which will imply for all i (cid:54) =
k (cid:82) ν
i ( dθ i ) < ∞ , i.e. for all coordinate we must have (cid:82) ν
i ( dθ i ) < ∞ .
Since (1) implies ν ( d θ ) is a product measure, we must have (cid:82) ν
( d θ ) < ∞ .
Suppose we have (2) and (3), use the above two results, by contradiction we cannot have (1).
Note that the problem only happen when the likelihood factorize (in both data and parameters), which is stronger than independent in data.
Independent is not enough to cause the problem.
One simple counter example is when one coordinates actually contains information of the parameter that determines the behavior of the second coordinates
Example 4 (Independent Bernoulli( θ 1 θ 2 ) ) .
Suppose we have g ( x | θ ) =
( θ 1 θ 2 ) x 1 + x 2 (1 − θ 1 θ 2 ) 1 − x 1 +1 − x 2 , i.e. x 1 , x 2 ∼ Bernoulli( θ 1 θ 2 ) independently.
The product measure of 3BP’s (denote as ν ( θ ) ) are properly normalized.
Because both have ﬁnite expectations.
[Equation]