Beyond Black Box Densities: Parameter Learning for the Deviated Components
Anonymous Author(s)
AfﬁliationAddress email
Abstract
As we collect additional samples from a data population for which a known densityfunction estimate may have been previously obtained by a black box method, theincreased complexity of the data set may result in the true density being deviatedfrom the known estimate by a mixture distribution.
To model this phenomenon, weconsider the deviating mixture model (1   ⇤ )
h
+  ⇤ (
P ki =1 p
⇤ i f
( x | ✓ ⇤ i )) , whereh 0 is a known density function, while the deviated proportion  ⇤ and latent mixingmeasure G ⇤ =
P ki =1 p
⇤ i  ✓ ⇤ i associated with the mixture distribution are unknown.
Via a novel notion of distinguishability between the known density h 0 and thedeviated mixture distribution, we establish rates of convergence for the maximumlikelihood estimates of  ⇤ and G ⇤ under Wasserstein metrics.
Simulation studiesare carried out to illustrate the theory.
Introduction
Most data-driven learning processes typically consist of iterative steps that involve model trainingand ﬁne-tuning, with more data in-take leading to further model re-training and reﬁnement.
Asmore samples come in and exhibit more complex patterns, the initial model may be obsolete, risksbeing discarded or absorbed into a richer class of models to adapt better to increased complexity.
Ittakes much resources to train complex models on a rich data population.
Moreover, many successfulmodels in modern real-world applications have become so complex that make them hard to properlyevaluate and interpret; aside from the predictive performance they may as well be considered as blackboxes.
Nonetheless, as data populations evolve and so must the learning models, several desiderataremain worthy: the ability to adapt to new complexity while retaining aspects of old "wise" model,and the ability to interpret the changes.
In this paper we will investigate a class of complex models for density estimation that are receptiveto adaptation , reuse and interpretablity : we posit that there is an existing distribution
h 0 whichmay have been obtained a priori by some means for the data population of interest, e.g., via kerneldensity estimation (KDE) [22] or mixture models [20] or some modern black box methods, suchas generative adversarial networks (GANs) [13, 1] or normalizing ﬂows [9].
Nonetheless, as moresamples come in and/or as the data population changes, it is possible that the true density may deviatefrom h 0 .
While h 0 is potentially difﬁcult to interpret, it is the deviation from the known h 0 that wewish to learn and interpret.
We will use a mixture distribution to represent this deviation, leading towhat we call a deviating mixture model for the underlying data population:
[Equation]
for x 2 R d , where F ( x, G ⇤ ) :
=
[Equation]
f
[Equation]
x | ✓
[Equation]
)) represents a mixture distribution for the densitycomponents deviating from h 0 .
Such deviating components are from a known family of densityfunction f .
The unknown parameters for this model are the mixing proportion  ⇤ 2 [0 , 1] , and the mixing measure G ⇤ =
P k
⇤ i =1 p
⇤ i  ✓ ⇤ i , where k ⇤  1 number of deviated components.
The choice ofmixture distribution F ( x, G ⇤ ) allows us to express complex deviation from h 0 , yet the overall modelremains amenable to the interpretation of its parameters:  ⇤ represents the amount of deviation fromthe existing candidate h 0 , while the mixing measure G ⇤ represents heterogeneous patterns of thedeviation.
The primary contribution of this paper is a rigorous investigation into the rather challengingquestions of identiﬁability and parameter learning rates that arise from a standard maximum likelihoodestimation procedure.
Relations to existing works.
This modeling framework owes its roots to several signiﬁcant bodiesof work in both statistics and machine learning literature.
In classical statistics, a dominant approachto address increased complexity of data populations is via hypothesis testing: one can test alternative(possibly composite) hypothesis represented by a class of distributions against the null hypothesisrepresented by h 0 .
Due to the constraint for obtaining simple and theoretically valid test statisticsin order to accept or reject the null hypothesis, the testing approaches were mostly restricted tosimple choices of distribution for the null and alternative hypotheses [6, 10, 7, 4, 8].
More similarto (1) is the class of contaminated mixture models for density estimation: in this framework, the dataare assumed to be sampled from a mixture of P 0 and Q where either P 0 or Q can be an unknowndistribution that needs to be estimated.
While this approach offers more ﬂexibility in terms ofmodeling, it does not always guarantee the identiﬁability of the mixing weight or mixture components
P 0 , Q [25, 23, 19].
Without identiﬁability, it is virtually impossible to interpret the model parametersfor the data domains.
To avoid the identiﬁability issue, several researchers added the semi-parametricor parametric structures on P 0 and Q , such as P 0 and Q are mixture distributions [2, 12].
However, tothe best of our knowledge, the convergence theory of these models remains poorly understood, exceptfor some simple settings (see also [3, 5]).
The main distinction between our modeling framework ofdeviating mixture models and the existing research on contaminated mixtures lies in our assumptionthat one of the mixture components, namely h 0 is known, allowing us to focus on the inferenceof the deviation from h 0 , for which a considerable learning theory for the parameters of interestcan be established and will be presented in this paper.
Finally, estimating parameters of mixturedistributions is an essential problem in mixture models.
The convergence properties have been studiedusing identiﬁability notions and Wasserstein distances [21, 18, 15].
Our technical approach requiresa generalization of the identiﬁability notion to take account of structural property of the existingcomponent h 0 , which helps to shed light on a considerably more complex convergence behavior ofthe deviated components.
Contributions.
The primary contribution of this paper is a rich theory of identiﬁability and ratesof convergence for parameters and density estimation that arise in the deviating mixture model (1),under various settings of the existing component h 0 , and that of the deviating components (viaf and G ⇤ ).
Because the convergence of density estimation in Hellinger distance under the MLEprocedure is well studied in [26], the bulk of our technical innovation lies in establishing a collectionof inverse bounds which relate the Hellinger distance of densities in model (1) in terms of that oftheir parameters.
To do that, we introduce a novel notion of distinguishability between h 0 and familyof density f .
The inverse bounds will be characterized under such distinguishability conditions (orthe lack thereof).
The novelty in our proof technique allows us to characterise different convergencesrates of parameters in the deviating mixture model under distinguishable settings.
It also gives riseto several new types of inverse bounds in partially distinguishable settings, where we may not haveidentiﬁability in our model.
To the best of our knowledge, it is the ﬁrst time such bounds is obtainedin mixture modeling literature.
Moreover, we will provide many examples to demonstrate the broadapplicability of our theory, including cases where the existing component h 0 is obtained by a blackbox method (e.g., deep learning model) and a more traditional method (e.g., via KDE’s or mixturemodels). By doing that, we are able to push the boundary of identiﬁability and learning theory ofmixture models towards modern machine learning models.
Organization.
The remainder part of this paper is structured as follows.
In Section 2, we review the
MLE method and the identiﬁability conditions, where the notion of distinguishability is presented.
In Section 3, the main results of inverse bounds and convergence rates for parameters estimation ofmodel (1) are shown.
In Section 4, multiple simulation experiments are carried out to support thetheory.
Finally, Section 5 is used to discuss and conclude.
Proofs of all the results in the main text aredeferred to the Supplementary Material.
Notation.
We denote by E k ( ⇥ )
= {
P ki [Equation] 1 p i f ( x | ✓ i ) : P ki
[Equation] [Equation] [Equation] 1  i  k }the family of mixtures with exactly k components and O K ( ⇥ ) = {
P
Ki = [Equation] p i f [Equation]
|
[Equation]
P
Ki = [Equation] 1 , p i  0 , ✓ i 2 ⇥ 8 1  i  K } the family of mixtures with no more than K components.
E k,c 0 ( ⇥ )
={
P ki =1 p i f ( x | ✓ i ) :
P ki =1 p i = 1 , p i  c 0 , ✓ i 2 ⇥ 8 1  i  k, k  K } is the family of mixtures 93 with exactly K components and mixing proportions being bounded below by c 0 , and
O K,c 0 ( ⇥ ) ={
P k 0 i =1 p i f ( x | ✓ i ) :
P k 0 i =1 p i = 1 , p i  c 0 , ✓ i 2 ⇥ 8 1  i  k 0 , k 0  K } .
k · k 2 is the usual l 2 normfor vectors in R d and matrices in R d ⇥ d .
We write g ( x ) & h ( x ) if g ( x ) > ch ( x ) for all x , where c isa constant does not depend on x (similar for g ( x ) .
h ( x ) ).
For any  2 R and B ⇢ R , denote by1 {  2 B } the function that takes value 1 if  2 B , and 0 otherwise.
For any two densities p and q , wedenote h ( p, q ) by the Hellinger distance and V ( p, q ) by the Total Variation distance between them.
Identiﬁability and distinguishability theory
The principal goal of the paper is to establish the efﬁciency of parameter learning for the deviatingmixture model (1) via the standard maximum likelihood estimation (MLE) method.
To achieve thisgoal, the parameters have to be identiﬁable to begin with.
Thus, our theory builds on and extends astandard notion of identiﬁability of families of density { f ( x | ✓ ) : ✓ 2 ⇥ } that has been considered in 104 previous work [21, 15].
Deﬁnition 2.1.
The family { f ( x | ✓ ) , ✓ 2 ⇥ } (or in short, f ) is identiﬁable in the order r , for some 106 r  1 , if f ( x | ✓ ) is differentiable up to the order r in ✓ and the following holds:
A1.
For any k  1 , given k different elements ✓ 1 , . . .
, ✓ k 2 ⇥ , if we have ↵ ( i ) ⌘ such that for
almost all x
[Equation]
then
↵ i ) ⌘ = 0 for all  i  k and | ⌘ | 
r Many commonly used families f for mixture modeling satisfy the ﬁrst order identiﬁability condition,including location-scale Gaussian distributions, e.g., f ( x | ✓ ) =
N ( x | µ,  2 ) where µ and  2 representthe mean (location) and variance (scale) parameters, and location-scale Student’s t-distributions.
In model (1), however, due to the presence of the existing component h 0 , the deviated mixturecomponents need to be distinguishable from h 0 .
This motivates a more general notion of identiﬁability,namely, distinguishability that we now deﬁne.
This condition speciﬁes a property jointly for both theexisting component h 0 and the family of density functions f that make up the deviated components.
Deﬁnition 2.2.
For any natural numbers k, r  1 , we say that the family of density functions{ f ( ·| ✓ ) , ✓ 2 ⇥ } with complexity level k (or in short, ( f, k ) ) is distinguishable up to the order r fromh
if the following holds:
A2.
For any k distinct components ✓ 1 , . . .
, ✓ k , if we have real coefﬁcients ↵ ( i ) ⌘ for 0  i  k such that
[Equation]
for almost surely x 2 X , then ↵ (0) =
↵ ( i ) ⌘ = 0 for 1  i  k and | ⌘ |  r .
We observe that the identiﬁable condition is a direct consequence of the corresponding distinguishablecondition.
A simple but non-trivial example of the distinguishability condition can be derived directlyfrom the deﬁnitions.
Example 2.3.
(a) When h 0 ( x ) =
P k 0 i =1 p 0 i f ( x | ✓ 0 i ) for some given weights ( p 01 , . .
. , p 0 k 0 ) and parameters ( ✓ 01 , . .
, ✓ 0 k 0 ) where k 0  1 , then ( f, k ) is distinguishable in the order r from h 0 as long ask < k 0 and the family of density f is identiﬁable in the order r .(b)
Given the choice of h 0 in (a), ( f, k ) is not distinguishable in the order r from h 0 when k  k 0 .
More signiﬁcantly, we can establish a broad class of h 0 and families f for which distinguishabilityholds.
This is exempliﬁed by the following theorem, where f represents a family of location orlocation-scale Gaussian kernels, and h 0 is subject to a relatively weak condition.
Theorem 2.4.
(a) Suppose that  log h 0 ( x ) & k x k  1 2 or  log h 0 ( x ) .
k x k  2 2 for all k x k 2 > x 0 , 134 for some x 0 > 0 ,  1 > 2 , and  2 < 2 .
Then, for f being family of location-scale Gaussian and anyk > 0 , ( f, k ) is distinguishable from h 0 up to the ﬁrst order, where the derivatives in Assumption A2are taken with respect to both location and scale parameters, and ( f, k ) is also distinguishable fromh
up to any order, where the derivatives in Assumption A2 are taken only with location parameters.(b) Suppose that h 0 is the pdf of a pushforward measure of N (0 , I d ) by a piecewise linear functionwith a ﬁnite and positive number of breakpoints.
Then, the same conclusions as in part (a) hold.
The proof of Theorem 2.4 is in Appendix C.1, where the main proof technique is carefully examinedthe tail densities of h 0 and f at inﬁnity.
Note that in part (a), h 0 can be a pdf of any distributionpossessing a lighter or heavier tail than Gaussian distributions, and in part (b), h 0 represents thepushforward of a Gaussian distribution by any piecewise linear function (recall that family ofpiecewise linear functions is dense in the Banach space of continuous functions with compactsupport).
In the sequel we shall demonstrate several examples of interest that are applicable to
Theorem 2.4 where h 0 may have been estimated by some popular "black box" methods.
Kernel based representation.
Suppose that h 0 was obtained from a m - sample Y 1 , . . .
, Y m 2 R dby a classical kernel density estimation (KDE) method [22] or a RKHS-based method [24], so that
[Equation]
where k  is a kernel function with bandwidth  .
Popular choices of kernels include the Gaussian kernel k  ( x, x
[Equation]
and the multivariate Student’s kernelk  ( x, x 0 ) =
[Equation]
The corresponding distinguishability guarantee is as follows.
Corollary 2.5.
Suppose h 0 is deﬁned by Eq.
(2) , where k  is Gaussian kernel and m > K , or k is the multivariate Student’s kernel.
Then, for f being family of location-scale Gaussian, ( f, K ) isdistinguishable from h 0 up to the ﬁrst order, where the derivatives in Assumption A2 are taken withrespect to both location and scale parameters, and ( f, K ) is also distinguishable from h
up to anyorder, where the derivatives in Assumption A2 are taken only with location parameters.
In application, it is common that the condition m > K is satisﬁed.
It is also matches with the scenariothat we consider in the paper, where h 0 is already trained using a big data set, and there is a smallnumber of deviated components.
Neural networks.
Deep neural networks represent a powerful, albeit black box, approximationdevice for constructing rich classes of distribution for generative models [13, 1].
Accordingly, h 0 isthe pdf function of a Gaussian distribution being push-forwarded by a map T , which is representedby a neural network (NN).
Suppose that the NN representing T has a positive and ﬁnite number oflayers L , and so
[Equation]
where W 1 , . .
, W L 2 R d ⇥ d are the weights and b 1 , . . .
, b L 2 R d are the biases.
The activationfunction a is chosen to be rectiﬁed linear unit (ReLU) function deﬁned by a ( x ) = max { x, 0 } , andis applied elementwise to any vector in R d .
The corresponding guarantee on the distinguishabilitycondition is as follows.
Corollary 2.6.
Suppose that h 0 is the pdf a pushforward measure of N (0 , I d ) by a map T deﬁned by
Eq.
Then, for f being family of location-scale Gaussian and any k > 0 , ( f, k ) is distinguishablefrom h 0 up to the ﬁrst order, where the derivatives in Assumption A2 are taken with respect to bothlocation and scale parameters, and ( f, k ) is also distinguishable from h 0 up to any order, where thederivatives in Assumption A2 are taken only with location parameters.
Convergence rates of density estimation
In this section, we ﬁrst establish the rate of density estimation for the deviating mixture modelsin Section 3.1.
We then describe a general procedure to obtain the convergence rate of parameter estimation based on that of density estimation via inverse bounds in Section 3.2.
Finally, we providecomprehensive inverse bounds under several settings of the deviating mixture models in Section 3.3.
3.1 MLE for deviating mixture model
Given n i.i.d.
sample X 1 , X 2 , . . .
, X n from p  ⇤ G ⇤ as in model (1), where G ⇤ has k ⇤ components,we want to estimate  ⇤ and G ⇤ from the data.
We refer to the problem as in exact-ﬁtted setting if k ⇤is known, and we refer to it as in over-ﬁtted setting if k ⇤ is unknown but is known to be bounded bysome number K .
We denote the MLE for exact-ﬁtted setting by
[Equation]
and for the over-ﬁtted setting, we replace E k ⇤ ( ⇥ ) in the equation above by O K ( ⇥ ) , where K  k ⇤ .
In order to state a rate of convergence for the density estimators p ˆ G n under the Hellinger distance h[26], we need a condition on the complexity of the function class
[Equation]
where for any G 2 O K ( ⇥ ) , we write ¯ p  G =
( p  G + p  ⇤ G ⇤ ) / 2 .
The deﬁnition of P k ( ⇥ , ✏ ) originates 189 from [26].
We measure the complexity of this class through the bracketing entropy integral
[Equation]
where N B ( ✏ , X, ⌘ ) denotes the ✏ -bracketing number of a metric space ( X, ⌘ ) and ⌫ is the Lebesguemeasure.
We require the following assumption.
A3.
Given a universal constant J > 0 , there exists N > 0 , possibly depending on ⇥ and k , such that for all n  N and all ✏ > (log n/n ) [Equation] [Equation]
[Equation]
Theorem 3.1.
Assume that Assumption A3 holds, and let k  1 .
There exists a constant C >depending only on ⇥ , k such that for all n  1 ,
[Equation]
Therefore, in order to get convergence rate for density functions based on MLE procedure, we onlyneed to check assumption A3.
This assumption holds true for a wide range class of parametric model[26].
For our model, we give an example that it holds when h 0 has an exponential tail (satisﬁed for
KDE’s and Neural networks above) and f is location-scale Gaussian distribution.
Proposition 3.2.
Suppose f is location-scale Gaussian family and ⇥ =
[  a, a ] d ⇥ ⌦ , where ⌦ isa subset of S ++ d
whose eigenvalues are bounded in [  ,  ] , a,  ,  > 0 , and h 0 is bounded with tail log h 0 ( x ) & k x k  2 for some  > 0 .
Then, the family of densities { p  G :  2 [0 , 1] , G 2 O k ( ⇥ ) } 199 satisﬁes assumption A3.
3.2 Parameter learning rates of deviated components
The core of this paper lies in establishing a collection of inverse bounds , provided that some distinguishability condition developed in Section 2 holds.
The inverse bounds basically say that a smalldistance between p  G and p  ⇤ G
⇤ under the total variation distance entails that (  , G ) and (  ⇤ , G ⇤ )are similar under appropriate distances, where (  ⇤ , G ⇤ ) is ﬁxed.
To this end, we employ Wassersteinmetrics [27] and their extensions.
Wasserstein distances.
Wasserstein distances are natural and useful for assessing the convergenceof latent mixing measures in mixture models [21, 16, 14].
Given two measures G =
P ki =1 p i  ✓ i and
G 0 =
P k 0 j =1 p
j  ✓ 0 j on a space ⇥ endowed with a metric ⇢ , the Wasserstein metric of order
r  1 is:
[Equation]
where the inﬁmum is taken over all joint distribution on [1 , . .
, k ] ⇥ [1 , . . .
, k 0 ] such that
P i q ij = 210 p [Equation] j ,
P
j q ij = p
i .
Note that if G n is a sequence of discrete measures that converges to G in a
Wasserstein distance, then for every atom of G , there is a subset of atoms of G n converges to it.
Therefore, the convergence in Wasserstein metrics implies convergence of parameters in mixturemodels.
In this paper, space ⇥ is often chosen to be a compact subset of R d and ⇢ is the usual l 2distance.
In the case of location-scale Gaussian mixtures, space ⇥ is a compact subset of R d ⇥
S ++ d , where S ++ d is the set of positive deﬁnite and symmetric matrices in R d ⇥ d , and for every( µ, ⌃ ) , ( µ 0 , ⌃ 0 ) 2 ⇥ , the distance ⇢ is deﬁned by ⇢ (( µ, ⌃ ) , ( µ 0 , ⌃ 0 ))
=
k µ  µ 0 k 2 + k ⌃  ⌃ 0 k 2 .
From inverse bounds to parameter learning rates.
Suppose that some distinguishablity conditionis satisﬁed, then we will establish an inverse bound providing a guarantee that a small distancebetween p  ⇤ G ⇤ and p  G entails a small distance between  and  ⇤ and between G and G ⇤ .
More 220 concretely, deﬁne a divergence between two measures  G and  ⇤ G ⇤ via
[Equation]
for all r  1 , and the inverse bounds will have the form
that V ( p  G , p  ⇤ ,G ⇤ ) & W r (  G,  ⇤ G ⇤ ) , forsome r that depends on the level of distinguishable level of the model.
Since total variational distanceis upper bounded by Hellinger distance, if Assumption A3.
holds, then combining the aforementionedinverse bound with Theorem 3.1 we immediately obtain
[Equation]
This further implies that the convergence rate of ˆ  n to  ⇤ is of order (log( n ) /n ) 1 / 2 and the convergence rate of W r ( ˆ G n , G ⇤ ) to 0 is of order (log( n ) /n )
1 / 2 r .
3.3 Inverse bounds in distinguishable setting
We shall establish inverse bounds provided a distinguishability condition for model (1) holds undereither exact-ﬁtted and over-ﬁtted settings regarding the true number of components k ⇤ .
Theorem 3.3.
Assume that k ⇤ is known and ( f, k ⇤ ) is distinguishable in the ﬁrst order from h 0 .
Then,for any G 2 E k ⇤ ( ⇥ ) , there exist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such 232 that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .(b)
When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G
⇤ , p  G )  C 2 W 1 (  G,  ⇤ G ⇤ ) .
We now present a proof sketch for Theorem 3.3.
It is a combination of the Taylor expansion aroundthe true parameters and the Fatou’s lemma; the proof technique for the remaining results also sharessimilar spirit as that of Theorem 3.3. Detailed proof of Theorem 3.3 is deferred to the Appendix. Proof sketch for part (b): Suppose that the {+bound is not correct, so there exists a sequence n 2 (0 , 1] and G n 2 E k ⇤ ( ⇥ ) such that V ( p  ⇤ G ⇤ , p  n G n ) /W 
+}
(  ⇤ G
⇤ ,  n G n ) ! 0 .
Because of 240 the compactness of the parameter space, by extracting a subsequence if necessary, we can assume n !
0 , G n W 1 !
G 0 .
{+If (  0 , G 0 ) 
+}
= (  ⇤ , G ⇤ ) , we have W 1 (  ⇤ G
⇤ ,  n G n ) !
W 1 (  ⇤ G
⇤ ,  0 G 0 ) 6 = 242 0 .
It indicates that V ( p  ⇤ G
⇤ , p  n G n ) !
0 , which leads to p
⇤ G
⇤ =
p  0 G 0 .
{+It contradicts to the 243 distinguishable condition when (  0 , G 0 ) 
+}
= (  ⇤ , G ⇤ ) ).
Otherwise, we have  n !
⇤ , G n !
G ⇤ , and can present G n =
P k
⇤ i =1 p ni  ✓ ni and G ⇤ =
P k
⇤ i =1 p
⇤ i  ✓ ⇤ i 245 such that p ni !
p ⇤ , ✓ ni !
✓ ⇤ i .
Because of these limits and by Taylor expansion, we can arrange the difference ( p  n G n ( x )  p
⇤ G ⇤ ( x )) /W
1 (  n G n ,  ⇤ G ⇤ ) in terms of a linear combination ofh 0 ( x ) , f ( x | ✓ ⇤ i ) , @@✓ f ( x | ✓ ⇤ i ) such that at least one coefﬁcient is different from 0.
By Fatou’s lemma,
[Equation]
dx , which equals to theabsolute integral of the linear combination above.
Hence, there exists a non-trivial linear combinationof h 0 ( x ) , f ( x | ✓ ⇤ i ) , @@✓ f ( x | ✓ ⇤ i ) that equals 0, which contradict to the distinguishability condition.
Therefore, we complete the proof.
In application, the true number of components k ⇤ might not be known and we often ﬁt the model (1)with G 2 O K ( ⇥ ) for some large K  k ⇤ .
The next result shows that similar bounds can also be 254 established in this case, where we require distinguishability of f and h 0 in a higher order.
Theorem 3.4.
Assume that k ⇤ is unknown and strictly upper bounded by a given K .
Assumeadditionally that ( f, K ) is distinguishable in second order from h 0 .
Then, for any G 2 O K ( ⇥ ) , thereexist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .(b)
When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G ⇤ , p  G )  C 2 W 2 (  G,  ⇤ G ⇤ ) .
Thanks to the distinguishability up to second order, no matter how large the number of over-ﬁttedcomponents K is, we always get the W 2 lower bound for the total variation distances.
Proof of thistheorem shares the same spirit with what of Theorem 3.3.
The difference here is when we overﬁt
G ⇤ with some ˆ G , there are some atoms of ˆ G that converges to the same atom of G ⇤ , which requiresus to do Taylor expansion up to second order and explain the higher order of Wasserstein distancehere.
Next, we relax the assumption of Theorem 3.4 by working on the setting where f is not secondorder identiﬁable.
This is an instance of the so-called weakly identiﬁable setting — One popularexample of weakly identiﬁable f is location-scale Gaussian distribution, which admits the partialdifferential equation (PDE) structure @ 2 f @ µ 2 ( x | µ, ⌃ ) = 2 @ f @ ⌃ ( x | µ, ⌃ ) , for all x 2 R d where f ( x | µ, ⌃ )stands for location-scale Gaussian density function with location µ and covariance ⌃ .
In order toillustrate the result of our bound for that weak identiﬁability setting of f , we speciﬁcally consider fto be location-scale Gaussian distribution.
In this case, the parameter space ⇥ is a compact subset of
R d ⇥ S ++ d , where S ++ d is the set of positive deﬁnite and symmetric matrices in R d ⇥ d equipped withthe usual Frobenius norm.
To put our result in context, we shall adopt a notion used in analyzing theconvergence rate of parameter estimation in location-scale Gaussian mixtures in [16].
For any k  1 ,let r ( k ) be the minimum value of r such that the following system of polynomial equations:
[Equation]
does not have any nontrivial solution for the unknown variables ( a j , b j , c j ) k +1 j =1 , where the ranges of 277 n 1 and n 2 in the second sum consist of all natural pairs satisfying the equation n 1 + 2 n 2 = ↵ .
Asolution to the above system is considered nontrivial if all of variables c
j are non-zeroes, while atleast one of the a j is non-zero.
Some examples of known values of r are r (1) = 4 and r (2) = 6 ,and r ( k )  7 for all k  3 .
Using this notion, we can characterize the convergence of parameters ofmodel (1) for the location-scale Gaussian family via the following theorem for inverse bounds.
Theorem 3.5.
Assume that G
⇤ 2 E k
⇤ ,c 0 ( ⇥ ) , and k ⇤ is unknown and strictly upper bounded by a 283 given K .
In addition, f is location-scale Gaussian distribution and ( f, K ) with varied location, ﬁxedvariance parameters is distinguishable in any order from h 0 .
Then, for any G 2 O K,c 0 ( ⇥ ) , there 285 exist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .
(b) When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G ⇤ , p  G )  C 2 W r
( K  k ⇤ )
(  G,  ⇤ G ⇤ ) .
The proof technique of this result involves doing Taylor expansion of both location and scaleparameter up to order r , then utilize the heat equation @ f @ ⌃ ( x | µ, ⌃ ) =
1 2 @ 2 f @ µ 2 ( x | µ, ⌃ ) to compressthis expression into linear combination of h 0 and derivatives of f ( x | µ, ⌃ ) with respect to µ only.
This allows us to use the condition in this theorem to imply a contradiction, and gives rise to Eq.
3.4 Inverse Bounds in Partially Distinguishable Setting
What happens if the distinguishability condition required by Def.
2.2 no longer holds generally?
Recall in Example 2.3 (b) that this situation is not uncommon, speciﬁcally when
[Equation]
where G 0 :
=
P k {+0 i =1 p 
+}
i  ✓ 0 i .
In some speciﬁc cases of this setting, in fact, we fail to attain distin- 296 guishability, and the model may not even be identiﬁable in the classical sense, i.e. p  G = p
⇤
G ⇤ 297 does not guarantee to have  G =
⇤ G
⇤ .
Since h 0 is the pdf of a mixture distribution — a popular choice for modeling complex forms of probability densities given its amenability to interpretationcompared to black box type models — it is of interest to study the implication of parameter estimationfor the deviated components in this setting, provided that the distinguisability condition may be atleast partially achieved in some suitable sense.
As we shall see, our theory demands a more reﬁnedanalysis.
To facilitate the presentation, denote A :=
1  i  k ⇤ : ✓ ⇤ i 2 { ✓ 01 , . .
, ✓ 0 k 0 } .
Also, set¯ k := |A| , which stands for the cardinal of the set A .
Our results will be divided into three separateregimes of ¯ k and  ⇤ : (i)  ⇤ = 0 , (ii) ¯ k < k 0 and  ⇤ 2 (0 , 1] , and (iii) ¯ k = k 0 and  ⇤ 2 (0 , 1] .
Weonly choose to present results of the second regime (ii) in the main text because of limited space andbecause of its representativeness as it shows all the intriguing behaviours of the model in this partiallydistinguishable setting.
The ﬁrst and third regime are deferred to Appendix A.
3.4.1 Regime B: ¯ k < k 0 and  ⇤ 2 (0 , 1]
First, we consider the exactly-speciﬁed setting of model (1), namely, k ⇤ is known.
When ¯ k < k 0 ,we can check that we still have dishtinguishability of h 0 and linear combinations of { f ( x | ✓ ⇤ i ) }
k
⇤ i =1 311 and its derivatives.
Therefore, as long as f is ﬁrst order identiﬁable, one can invoke the proof of
Theorem 3.3 to establish the same lower bound V ( p  G , p  ⇤ G ⇤ ) in terms of W r (  G,  ⇤ G ⇤ ) for some 313 r  1 .
Thus, our focus in this subsection is the settings when k ⇤ is unknown.
Over-ﬁtted setting with strongly identiﬁable f .
Moving to the over-ﬁtted settings of modelsetup (1), i.e., k ⇤ is unknown and strictly upper bounded by a given K , as long as K  k 0 ,( f, K ) is not distinguishable from h 0 .
Therefore, the results of Theorem 3.3 are not always applicableto the setting when K  k 0 .
Besides, in the over-ﬁtted setting, the identiﬁability of model (1) nolonger holds.
Indeed, for any  >  ⇤ , if we take
[Equation]
) then p  ⇤ G ⇤ =
p  G
⇤ (  ) .
We present this pathological behavior in the following result.
Theorem 3.6.
Assume that h 0 takes the form (7) and ¯ k < k 0 .
Besides that, K  k 0 and f is secondorder identiﬁable.
Then, for any G 2 O K ( ⇥ ) , there exist positive constants C 1 and C 2 dependingonly on  ⇤ , G ⇤ , h 0 , ⇥ such that the following hold:
(a) If K  k ⇤ + k 0  ¯ k  1 , then V ( p  ⇤ ,G ⇤ , p  ,G )  C 1 W 2 (  G,  ⇤ G ⇤ ) ,(b) If K  k ⇤ + k 0  ¯ k , then
[Equation]
(c) As a special case, if K = k ⇤ + k
¯ k , we have
[Equation]
for all  > 0 , where C 3 depends on  ⇤ , G ⇤ , h 0 , ⇥ ,  .
As we can see, the magnitude of  compared to  ⇤ will decide the solution of (  , G ) to the identiﬁableequation p  G
= p  ⇤ G
⇤ , therefore lead to different lower bounds such in part (b) of the theorem.
In particular, if    ⇤ , the solution is (  , G ) =
(  ⇤ , G ⇤ ) , and for any  >  ⇤ , the solution is
G = G ⇤ (  ) given in Eq.
Speciﬁcally, when  is strictly larger than  ⇤ by some amount  > 0 ,then the latter case is well separated from the former, and we have an exact-ﬁtted result when[-K = k 
 + k ⇤  k .-]
K = k {+
+}
+ k[-+
 -] ⇤  k .
[-⇤ i  ✓ ⇤ i having exactly k 
-]
[-(a) When K  k ⇤ + k 
-]
[-k ⇤ + k 
-]
[-G, G ⇤ (  )) , and when K > k ⇤ + k 
-]
[-If K = k 
-]
[-K > k 
-]
Experiments
We now would like to demonstrate the convergence rates in Section 3 via two synthetic experiments: one for distinguishable setting and one for partially distinguishable setting.
For the partiallydistinguishable one, the experiments are in Appendix B.
Distinguishable setting.
We conduct an experiment where the original data distribution comes froman uniform distribution on a curve (half circle) in R 2 convoluted with Gaussian noises (red curve andblue points in Fig. 1(a)), and train a Normalizing Flow neural network [11] (Masked Autoregressivearchitecture) with 5 layers to get a good density estimation h 0 for this dataset.
Then we assume thatthere are new data coming in, and the original distribution h 0 is deviated by a mixture of distributionsin the location Gaussian family f ( x | ✓ ) .
So the true generating density now is p
⇤ G ⇤ ( x ) =
(1   ⇤ )
h 0 ( x ) +  ⇤ X i = p ⇤ i f ( x | ✓ ⇤ i ) , (9) where  ⇤ = 0 . 5 , G ⇤ =
P 3 i =1 p
⇤ i  ✓ ⇤ i , where p ⇤ 1 = 0 .
3 , p ⇤ 2 = 0 .
3 , p ⇤ 3 = 0 .
4 , ✓ ⇤ 1 =
(  0 .
7 , 1 . 5) , ✓ ⇤ 2 =(0 .
1 , 2 . 0) , ✓ ⇤ 3 = (1 . 0 , 1 . 5) .
Samples from the deviated component are green points in Fig. 1(a).
It canbe seen from Proposition 2.4(a) that h 0 is distinguishable with family f .
For each n , we simulaten data points from true model (9), estimate ˆ  n , ˆ G n by the EM algorithm (it is possible because
Normalizing Flows provides exact density computation), and measure its convergence to the true
⇤ , G ⇤ .
We conduct 16 replications for each sample size.
The average error estimations with a 75%error bar can be seen in Fig.
The W 1 error in the exact-ﬁtted case is of order (log( n ) /n )
1 / 2 and W 2error in the over-ﬁtted case is of order (log( n ) /n )
Meanwhile, thanks to the distinguishability,the estimation errors in both cases of  are all of the order (log( n ) /n )
These simulation resultsare matched with the theoretical results found in Theorem 3.3 and Theorem 3.4.
From the result,we see that the deviating mixture model successfully learns the deviated components and reuses thepre-trained black box model h 0 , which helps to reduce computational costs.
[Figure]
Discussion
In this work, we have presented the deviating mixture model and studied its parameter learning ratesunder MLE procedure.
With a novel notion of distinguishability between distributions, we are able toprove inverse bounds for our model under several distinguishability settings, which allow us to deducethe parameter learning rates from the convergence rate of density functions.
The distinguishabilitycondition is shown to be satisﬁed for multiple families of distributions including those that comefrom black box models.
We now discuss practical implication of the theory.
The deviating mixture model is designed to capturethe deviated mixture components, and learning its parameters can reveal meaningful informationabout subpopulations in the data.
When there is distinguishability in the model, our theory impliesthat we can learn the deviated proportion with the parametric rate and deviated components with arate depending on the identiﬁable of f .
However, our theory does not support using deviating mixturemodel when the existing distribution h 0 itself is a mixture distributions in family f and possessesparameters similar to deviated part, as the learning rate can be slow, and the deviated proportionestimator may not converge to the true value. Asymptotically, when h 0 is estimated using a verycomplex model (eg.
a wide and deep neural network)
and somehow approximates a mixture of f ,and/or the signal from deviating components is low, then the provided learning rates in the paper,although are still the same with respect to sample size n , can be suffered from a large multiplicativeconstant that depends on h 0 ,  ⇤ , and G ⇤ .
We believe that this work is the ﬁrst attempt in the effort of understanding a broader class of mixturemodels combining with black box models, and its interpretability.
There is room for future workbuilding on top of this work.
From a theoretical viewpoint, one can consider showing minimax lowerbounds for the learning rates of the deviating mixture model, or show uniform inverse bounds for themodel when  ⇤ and G ⇤ are considered as signals that will change with samples.
From a modelingviewpoint, it is potential to explore mixtures of black box models and develop a new suitable notionof identiﬁability and inverse bounds so that the learning process is efﬁcient.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Section
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] It can be seen in the source code.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 1, 2, 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No] The experiments are run on CPU’s
only.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[No] We do not use any
existing assets.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, [Table]
if applicable?
[No] We do not use crowdsourcing/conducted research with human [Table]
subjects
Did you describe any potential participant risks, with links to Institutional Review [Table]
Board approvals, if applicable?
[No]
[Table]
c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[No]
[Table]