Beyond Black Box Densities: Parameter Learning for the Deviated Components
Abstract
As we collect additional samples from a data population for which a known density function estimate may have been previously obtained by a black box method, the increased complexity of the data set may result in the true density being deviated from the known estimate by a mixture distribution.
To learn about this phenomenon, we consider the deviating mixture model (1   ⇤ )
h
+  ⇤ (
[Equation]
i )) , where h 0 is a known density function, while the deviated proportion  ⇤ and latent mixing measure G ⇤ = [Equation]
ki [Equation] p ⇤ [Equation]
✓ ⇤ i associated with the mixture distribution are unknown.
Using a novel notion of distinguishability between the known density h 0 and the deviated mixture distribution, we establish rates of convergence for the maximum likelihood estimates of  ⇤ and G ⇤ under Wasserstein metrics.
Simulation studies are carried out to illustrate the theory.
Introduction
Most data-driven learning processes typically consist of an iteration of steps that involve model training and ﬁne-tuning, with more data in-take leading to further model re-training and reﬁnement.
As more samples become available and exhibit more complex patterns, the initial model may be obsolete, risks being discarded or absorbed into a richer class of models that adapt better to the increased complexity.
It takes considerable resources to train complex models on a rich data population.
Moreover, many successful models in modern real-world applications have become so complex that make them hard to properly evaluate and interpret; aside from the predictive performance they may as well be considered as black boxes.
Nonetheless, as data populations evolve and so must the learning models, several desiderata remain worthy: the ability to adapt to new complexity while retaining aspects of old "wise" model and the ability to interpret the changes.
In this paper we will investigate a class of complex models for density estimation that are receptive to adaptation , reuse and interpretablity : we posit that there is an existing distribution h 0 which may have been obtained a priori by some means for the data population of interest, e.g., via kernel density estimation (KDE) [22] or mixture models [20] or some modern black box methods, such as generative adversarial networks (GANs) [13, 1] or normalizing ﬂows [9].
Nonetheless, as more samples become available and/or as the data population changes, it is possible that the true density may deviate from h 0 .
While h 0 is potentially difﬁcult to explicate, it is the deviation from the known h 0 that we wish to learn and interpret.
We will use a mixture distribution to represent this deviation, leading to what we call a deviating mixture model for the underlying data population:
[Equation]
for x 2 R d , where F ( x, G ⇤ ) :
=
P k
⇤ i =1 p ⇤ i f
( x | ✓ ⇤ i )) represents a mixture distribution for the density components deviating from h 0 .
Such deviating components are from a known family of density function f .
The unknown parameters for this model are the mixing proportion  ⇤ 2 [0 , 1] , and the mixing measure G ⇤ =
P k
⇤ i =1 p
⇤ i  ✓ ⇤ i , where k ⇤  1 number of deviated components.
The choice of mixture distribution F ( x, G ⇤ ) allows us to express complex deviation from h 0 , yet the overall model remains amenable to the interpretation of its parameters:  ⇤ represents the amount of deviation from the existing candidate h 0 , while the mixing measure G ⇤ represents heterogeneous patterns of the deviation.
Because h 0 might be complex and trained with great computational resource to estimate the density of prior data population, it is reasonable to assume h 0 be known in the model (1).
The primary contribution of this paper is a rigorous investigation into the rather challenging questions of identiﬁability and parameter learning rates that arise from a standard maximum likelihood estimation procedure.
Relations to existing works.
This modeling framework owes its roots to several signiﬁcant bodies of work in both statistics and machine learning literature.
In classical statistics, a dominant approach to address the increased complexity of data populations is via hypothesis testing: one can test an alternative (possibly composite) hypothesis represented by a class of distributions against the null hypothesis represented by h 0 .
Due to the constraint for obtaining simple and theoretically valid test statistics in order to accept or reject the null hypothesis, the testing approaches were mostly restricted to simple choices of distribution for the null and alternative hypotheses [6, 10, 7, 4, 8].
More similar to (1) is the class of contaminated mixture models for density estimation: in this framework, the data are assumed to be sampled from a mixture of P 0 and Q where either P 0 or Q can be an unknown distribution that needs to be estimated.
While this approach offers more ﬂexibility in terms of modeling, it does not always guarantee the identiﬁability of the mixing weight or mixture components
P 0 , Q [25, 23, 19].
Without identiﬁability, it is virtually impossible to interpret the model parameters for the data domains.
To avoid the identiﬁability issue, several researchers added the semi-parametric or parametric structures on P 0 and Q , such as P 0 and Q are mixture distributions [2, 12].
However, to the best of our knowledge, the convergence theory of these models remains poorly understood, except for some simple settings (see also [3, 5]).
The main distinction between our modeling framework of deviating mixture models and the existing research on contaminated mixtures lies in our assumption that one of the mixture components, namely h 0 is known, allowing us to focus on the inference of the deviation from h 0 , for which a considerable learning theory for the parameters of interest can be established and will be presented in this paper.
Finally, estimating parameters of mixture distributions is an essential problem in mixture models.
The convergence properties have been studied using identiﬁability notions and Wasserstein distances [21, 18, 15].
Our technical approach requires a generalization of the identiﬁability notion to take account of structural property of the existing component h 0 , which helps to shed light on a considerably more complex convergence behavior of the deviated components.
Contributions.
The primary contribution of this paper is a rich theory of identiﬁability and rates of convergence for parameters and density estimation that arise in the deviating mixture model (1), under various settings of the existing component h 0 , and that of the deviating components (via f and G ⇤ ).
Because the convergence of density estimation in Hellinger distance under the MLE procedure is well studied in [26], the bulk of our technical innovation lies in establishing a collection of inverse bounds which relate the Hellinger distance of densities in model (1) in terms of that of their parameters.
To do that, we introduce a novel notion of distinguishability between h 0 and family of density f .
The inverse bounds will be characterized under such distinguishability conditions (or the lack thereof).
Our proof technique allows us to characterize different convergence rates of parameters in the deviating mixture model under distinguishable settings.
It also gives rise to several new types of inverse bounds in partially distinguishable settings, where we may not have identiﬁability in our model.
To the best of our knowledge, this is the ﬁrst work in which such bounds are obtained in mixture modeling literature.
Moreover, we will provide many examples to demonstrate the broad applicability of our theory, including cases where the existing component h 0 is obtained by a black box method (e.g., deep learning model) and a more traditional method (e.g., via KDE’s or mixture models).
By doing so, we are able to push the boundary of identiﬁability and learning theory of mixture models toward a larger class of modern machine learning models.
Organization.
The remainder part of this paper is structured as follows.
In Section 2, we review the
MLE method and the identiﬁability conditions, where the notion of distinguishability is presented.
In Section 3, the main results of inverse bounds and convergence rates for parameters estimation of model (1) are shown.
In Section 4, multiple simulation experiments are carried out to support the theory.
Finally, Section 5 is used to discuss and conclude.
Proofs of all the results in the main text are deferred to the Supplementary Material.
Notation.
We denote by E k ( ⇥ )
= {
[Equation]
[Equation] } the family of mixtures with exactly k components and O K ( ⇥ ) = {
P
Ki
[Equation]
[Equation]
i [Equation] ( x | ✓ [Equation]
P
Ki =1 p i = 1 , p i  0 , ✓ i 2 ⇥ 8 1  i  K } the family of mixtures with no more than K components.
E k,c 0 ( ⇥ )
= {
P ki =
[Equation]
p i f ( [Equation] | [Equation]
i
[Equation]
P ki =
[Equation]
p
[Equation] = [Equation] , p i  c 0 , ✓ i 2 ⇥ 8 1  i  k, k  K } is the family of mixtures with exactly K components and mixing proportions being bounded below by c 0 , and
O K,c 0 ( ⇥ ) = {
P
[Equation] = [Equation] p
[Equation]
P [Equation]  [Equation]
k [Equation]  K } . k · k 2 is the usual l 2 norm for vectors in R d and matrices in R d ⇥ d .
We write g ( x ) & h ( x ) if g ( x ) > ch ( x ) for all x , where c is a constant does not depend on x (similar for g ( x ) .
h ( x ) ).
For any  2 R and B ⇢ R , denote by 1 {  2 B } the function that takes value 1 if  2 B , and 0 otherwise.
For any two densities p and q , we denote h ( p, q ) by the Hellinger distance and V ( p, q ) by the Total Variation distance between them.
Identiﬁability and distinguishability theory
The principal goal of the paper is to establish the efﬁciency of parameter learning for the deviating mixture model (1) via the standard maximum likelihood estimation (MLE) method.
To achieve this goal, the parameters have to be identiﬁable to begin with.
Thus, our theory builds on and extends a standard notion of identiﬁability of families of density { f ( x | ✓ ) : ✓ 2 ⇥ } that has been considered in previous work [21, 15].
Deﬁnition 2.1.
The family { f ( x | ✓ ) , ✓ 2 ⇥ } (or in short, f ) is identiﬁable in the order r , for some r  1 , if f ( x | ✓ ) is differentiable up to the order r in ✓ and the following holds:
A1.
For any k  1 , given k different elements ✓ 1 , . . .
, ✓ k 2 ⇥ , if we have ↵ ( i ) ⌘ such that for almost all x
[Equation]
=then ↵ ( i ) ⌘ = 0 for all 1  i  k and | ⌘ |  r .
Many commonly used families f for mixture modeling satisfy the ﬁrst order identiﬁability condition, including location-scale Gaussian distributions, e.g., f ( x | ✓ ) =
N ( x | µ,  2 ) where µ and  2 represent the mean (location) and variance (scale) parameters, and location-scale Student’s t-distributions.
In model (1), however, due to the presence of the existing component h 0 , the deviated mixture components need to be distinguishable from h 0 .
This motivates a more general notion of identiﬁability, namely, distinguishability that we now deﬁne.
This condition speciﬁes a property jointly for both the existing component h 0 and the family of density functions f that make up the deviated components.
Deﬁnition 2.2.
For any natural numbers k, r  1 , we say that the family of density functions { f ( ·| ✓ ) , ✓ 2 ⇥ } with complexity level k (or in short, ( f, k ) ) is distinguishable up to the order r from h
if the following holds:
A2.
For any k distinct components ✓ 1 , . . .
, ✓ k , if we have real coefﬁcients ↵ ( i ) ⌘ for 0  i  k such that
[Equation]
for almost surely x 2 X , then ↵ (0) =
↵ ( i ) ⌘ = 0 for 1  i  k and | ⌘ |  r .
We observe that the identiﬁable condition is a direct consequence of the corresponding distinguishable condition.
A simple but non-trivial example of the distinguishability condition can be derived directly from the deﬁnitions.
Example 2.3.
(a) When h 0 ( x ) =
P k 0 i =1 p 0 i f ( x | ✓ 0 i ) for some given weights ( p 01 , . .
. , p 0 k 0 ) and parameters ( ✓ 01 , . .
, ✓ 0 k 0 ) where k 0  1 , then ( f, k ) is distinguishable in the order r from h 0 as long as k < k 0 and the family of density f is identiﬁable in the order r .
(b) Given the choice of h 0 in (a), ( f, k ) is not distinguishable in the order r from h 0 when k  k 0 .
More signiﬁcantly, we can establish a broad class of h 0 and families f for which distinguishability holds.
This is exempliﬁed by the following theorem, where f represents a family of location or location-scale Gaussian kernels, and h 0 is subject to a relatively weak condition.
Theorem 2.4.
(a) Suppose that  log h 0 ( x ) & k x k  1 2 or  log h 0 ( x ) .
k x k  2 2 for all k x k 2 > x 0 , for some x 0 > 0 ,  1 > 2 , and  2 < 2 .
Then, for f being family of location-scale Gaussian and any k > 0 , ( f, k ) is distinguishable from h 0 up to the ﬁrst order, where the derivatives in Assumption A are taken with respect to both location and scale parameters, and ( f, k ) is also distinguishable from h
up to any order, where the derivatives in Assumption A2 are taken only with location parameters. (b) Suppose that h 0 is the pdf of a pushforward measure of N (0 , I d ) by a piecewise linear function with a ﬁnite and positive number of breakpoints.
Then, the same conclusions as in part (a) hold.
The proof of Theorem 2.4 is in Appendix C.1, where the main proof technique is carefully examining the tail densities of h 0 and f at inﬁnity.
Note that in part (a), h 0 can be a pdf of any distribution possessing a lighter or heavier tail than Gaussian distributions, and in part (b), h 0 represents the pushforward of a Gaussian distribution by any piecewise linear function (recall that family of piecewise linear functions is dense in the Banach space of continuous functions with compact support).
In the sequel we shall demonstrate several examples of interest that are applicable to
Theorem 2.4 where h 0 may have been estimated by some popular "black box" methods.
Kernel based representation.
Suppose that h 0 was obtained from a m - sample Y 1 , . . .
, Y m 2 R d by a classical kernel density estimation (KDE) method [22] or a RKHS-based method [24], so that
[Equation]
where k  is a kernel function with bandwidth  .
Popular choices of kernels include the Gaussian kernel k  ( x, x 0 ) =
[Equation]
and the multivariate Student’s kernel k [Equation] [Equation] =
[Equation]
The corresponding distinguishability guarantee is as follows.
Corollary 2.5.
Suppose h 0 is deﬁned by Eq.
(2) , where k  is Gaussian kernel and m > K , or k  is the multivariate Student’s kernel.
Then, for f being family of location-scale Gaussian, ( f, K ) is distinguishable from h 0 up to the ﬁrst order, where the derivatives in Assumption A2 are taken with respect to both location and scale parameters, and ( f, K ) is also distinguishable from h
up to any order, where the derivatives in Assumption A2 are taken only with location parameters.
In application, it is common that the condition m > K is satisﬁed.
It is also matches with the scenario that we consider in the paper, where h 0 is already trained using a big data set, and there is a small number of deviated components.
Neural networks.
Deep neural networks represent a powerful, albeit black box, approximation device for constructing rich classes of distribution for generative models [13, 1].
Accordingly, h 0 is the pdf function of a Gaussian distribution being push-forwarded by a map T , which is represented by a neural network (NN).
Suppose that the NN representing T has a positive and ﬁnite number of layers L , and so
[Equation]
where W 1 , . .
, W L 2 R d ⇥ d are the weights and b 1 , . . .
, b L 2 R d are the biases.
The activation function a is chosen to be rectiﬁed linear unit (ReLU) function deﬁned by a ( x ) = max { x, 0 } , and is applied elementwise to any vector in R d .
The corresponding guarantee on the distinguishability condition is as follows.
Corollary 2.6.
Suppose that h 0 is the pdf a pushforward measure of N (0 , I d ) by a map T deﬁned by
Eq.
Then, for f being family of location-scale Gaussian and any k > 0 , ( f, k ) is distinguishable from h 0 up to the ﬁrst order, where the derivatives in Assumption A2 are taken with respect to both location and scale parameters, and ( f, k ) is also distinguishable from h {+
+} up to any order, where the derivatives in Assumption A2 are taken only with location parameters.
Convergence rates of density estimation
In this section, we ﬁrst establish the rate of density estimation for the deviating mixture models in Section 3.1.
We then describe a general procedure to obtain the convergence rate of parameter estimation based on that of density estimation via inverse bounds in Section 3.2.
Finally, we provide comprehensive inverse bounds under several settings of the deviating mixture models in Section 3.3.
3.1 MLE for deviating mixture model
Given n i.i.d.
sample X 1 , X 2 , . . .
, X n from p  ⇤ G ⇤ as in model (1), where G ⇤ has k ⇤ components, we want to estimate  ⇤ and G ⇤ from the data.
We refer to the problem as in exact-ﬁtted setting if k ⇤ is known, and we refer to it as in over-ﬁtted setting if k ⇤ is unknown but is known to be bounded by some number K .
We denote the MLE for exact-ﬁtted setting by
[Equation]
and for the over-ﬁtted setting, we replace E k ⇤ ( ⇥ ) in the equation above by O K ( ⇥ ) , where K  k ⇤ .
In order to state a rate of convergence for the density estimators p ˆ G n under the Hellinger distance h [26], we need a condition on the complexity of the function class
[Equation]
where for any G 2 O K ( ⇥ ) , we write ¯ p  G =
( p  G + p  ⇤ G ⇤ ) / 2 .
The deﬁnition of P k ( ⇥ , ✏ ) originates from [26].
We measure the complexity of this class through the bracketing entropy integral
[Equation]
where N B ( ✏ , X, ⌘ ) denotes the ✏ -bracketing number of a metric space ( X, ⌘ ) and ⌫ is the Lebesgue measure.
We require the following assumption.
A3.
Given a universal constant J > 0 , there exists N > 0 , possibly depending on ⇥ and k , such that for all n  N and all ✏ > (log n/n ) 1 / 2 ,
[Equation]
Theorem 3.1.
Assume that Assumption A3 holds, and let k  1 .
There exists a constant C >depending only on ⇥ , k such that for all n  1 ,
[Equation]
Therefore, in order to get convergence rate for density functions based on MLE procedure, we only need to check assumption A3.
This assumption holds true for a wide range class of parametric model [26].
For our model, we give an example that it holds when h 0 has an exponential tail (satisﬁed for
KDE’s and Neural networks above) and f is location-scale Gaussian distribution.
Proposition 3.2.
Suppose f is location-scale Gaussian family and ⇥ =
[  a, a ] d ⇥ ⌦ , where ⌦ is a subset of S ++ d whose eigenvalues are bounded in [  ,  ] , a,  ,  > 0 , and h 0 is bounded with tail  log h 0 ( x ) & k x k  2 for some  > 0 .
Then, the family of densities { p  G :  2 [0 , 1] , G 2 O k ( ⇥ ) } satisﬁes assumption A3.
3.2 Parameter learning rates of deviated components
The core of this paper lies in establishing a collection of inverse bounds , provided that some distinguishability condition developed in Section 2 holds.
The inverse bounds basically say that a small distance between p  G and p  ⇤ G ⇤ under the total variation distance entails that (  , G ) and (  ⇤ , G ⇤ ) are similar under appropriate distances, where (  ⇤ , G ⇤ ) is ﬁxed.
To this end, we employ Wasserstein metrics [27] and their extensions.
Wasserstein distances.
Wasserstein distances are natural and useful for assessing the convergence of latent mixing measures in mixture models [21, 16, 14].
Given two measures G =
P ki =1 p i  ✓ i and
G 0 =
P k [Equation] j =1 p
j  ✓ 0 j on a space ⇥ endowed with a metric ⇢ , the Wasserstein metric of order
r  1 is:
[Equation]
where the inﬁmum is taken over all joint distribution on [1 , . .
, k ] ⇥ [1 , . . .
, k 0 ] such that
P [Equation] p [Equation]
P
j q ij = p
i .
Note that if G n is a sequence of discrete measures that converges to G in a
Wasserstein distance, then for every atom of G , there is a subset of atoms of G n converges to it.
Therefore, the convergence in Wasserstein metrics implies convergence of parameters in mixture models.
In this paper, space ⇥ is often chosen to be a compact subset of R d and ⇢ is the usual ldistance.
In the case of location-scale Gaussian mixtures, space ⇥ is a compact subset of R d ⇥
S ++ d , where S ++ d is the set of positive deﬁnite and symmetric matrices in R d ⇥ d , and for every ( µ, ⌃ ) , ( µ 0 , ⌃ 0 ) 2 ⇥ , the distance ⇢ is deﬁned by ⇢ (( µ, ⌃ ) , ( µ 0 , ⌃ 0 ))
=
k µ  µ 0 k 2 + k ⌃  ⌃ 0 k 2 .
From inverse bounds to parameter learning rates.
Suppose that some distinguishablity condition is satisﬁed, then we will establish an inverse bound providing a guarantee that a small distance between p  ⇤ G ⇤ and p  G entails a small distance between  and  ⇤ and between G and G ⇤ .
More concretely, deﬁne a divergence between two measures  G and  ⇤ G ⇤ via
[Equation]
for all r  1 , and the inverse bounds will have the form that V ( p  G , p  ⇤ ,G ⇤ ) & W r (  G,  ⇤ G ⇤ ) , for some r that depends on the level of distinguishable level of the model.
Since total variational distance is upper bounded by Hellinger distance, if Assumption A3.
holds, then combining the aforementioned inverse bound with Theorem 3.1 we immediately obtain
[Equation]
This further implies that the convergence rate of ˆ  n to  ⇤ is of order (log( n ) /n ) 1 / 2 and the convergence rate of W r ( ˆ G n , G ⇤ ) to 0 is of order (log( n ) /n )
1 / 2 r .
3.3 Inverse bounds in distinguishable setting
We shall establish inverse bounds provided a distinguishability condition for model (1) holds under either exact-ﬁtted and over-ﬁtted settings regarding the true number of components k ⇤ .
Theorem 3.3.
Assume that k ⇤ is known and ( f, k ⇤ ) is distinguishable in the ﬁrst order from h 0 .
Then, for any G 2 E k ⇤ ( ⇥ ) , there exist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .
(b) When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G ⇤ , p  G )  C 2 W 1 (  G,  ⇤ G ⇤ ) .
We now present a proof sketch for Theorem 3.3.
It is a combination of the Taylor expansion around the true parameters and the Fatou’s lemma; the proof technique for the remaining results also shares similar spirit as that of Theorem 3.3.
Detailed proof of Theorem 3.3 is deferred to the Appendix.
Proof sketch for part (b): Suppose that the bound is not correct, so there exists a sequence  n 2 (0 , 1] and G n 2 E k ⇤ ( ⇥ ) such that V ( p  ⇤ G
⇤ , p  n G n ) /W
(  ⇤ G
⇤ ,  n G n ) ! 0 .
Because of the compactness of the parameter space, by extracting a subsequence if necessary, we can assume  n !
0 , G n W 1 !
G 0 .
If (  0 , G 0 )
= (  ⇤ , G ⇤ ) , we have W 1 (  ⇤ G
⇤ ,  n G n ) !
W 1 (  ⇤ G
⇤ ,  0 G 0 ) 6 = 0 .
It indicates that V ( p  ⇤ G
⇤ , p  n G n ) !
0 , which leads to p
⇤ G
⇤ =
p  0 G 0 .
It contradicts to the distinguishable condition when (  0 , G 0 )[-
-]
[Equation]
Otherwise, we have  n !
⇤ , G n !
G ⇤ , and can present G n =
[Equation]
and
[Equation]  [Equation] i such that p ni !
p ⇤ , ✓ ni !
✓ ⇤ i .
Because of these limits and by Taylor expansion, we can arrange the difference ( p  n G n ( x )  p
⇤ G ⇤ ( x )) /W
1 (  n G n ,  ⇤ G ⇤ ) in terms of a linear combination of h 0 ( x ) , f ( x | ✓ ⇤ i ) , @@✓ f ( x | ✓ ⇤ i ) such that at least one coefﬁcient is different from 0.
By Fatou’s lemma, 0 =
[Equation]
inf
[Equation]
dx , which equals to the absolute integral of the linear combination above.
Hence, there exists a non-trivial linear combination of h 0 ( x ) , f ( x | ✓ ⇤ i ) , @@✓ f ( x | ✓ ⇤ i ) that equals 0, which contradict to the distinguishability condition.
Therefore, we complete the proof.
In application, the true number of components k ⇤ might not be known and we often ﬁt the model (1) with G 2 O K ( ⇥ ) for some large K  k ⇤ .
The next result shows that similar bounds can also be established in this case, where we require distinguishability of f and h 0 in a higher order.
Theorem 3.4.
Assume that k ⇤ is unknown and strictly upper bounded by a given K .
Assume additionally that ( f, K ) is distinguishable in second order from h 0 .
Then, for any G 2 O K ( ⇥ ) , there exist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .
(b) When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G ⇤ , p  G )  C 2 W 2 (  G,  ⇤ G ⇤ ) .
Thanks to the distinguishability up to second order, no matter how large the number of over-ﬁtted components K is, we always get the W 2 lower bound for the total variation distances.
Proof of this theorem shares the same spirit with what of Theorem 3.3.
The difference here is when we overﬁt
G ⇤ with some ˆ G , there are some atoms of ˆ G that converges to the same atom of G ⇤ , which requires us to do Taylor expansion up to second order and explain the higher order of Wasserstein distance here.
Next, we relax the assumption of Theorem 3.4 by working on the setting where f is not second order identiﬁable.
This is an instance of the so-called weakly identiﬁable setting — One popular example of weakly identiﬁable f is location-scale Gaussian distribution, which admits the partial differential equation (PDE) structure @ 2 f @ µ 2 ( x | µ, ⌃ ) = 2 @ f @ ⌃ ( x | µ, ⌃ ) , for all x 2 R d where f ( x | µ, ⌃ ) stands for location-scale Gaussian density function with location µ and covariance ⌃ .
In order to illustrate the result of our bound for that weak identiﬁability setting of f , we speciﬁcally consider f to be location-scale Gaussian distribution.
In this case, the parameter space ⇥ is a compact subset of
R d ⇥ S ++ d , where S ++ d is the set of positive deﬁnite and symmetric matrices in R d ⇥ d equipped with the usual Frobenius norm.
To put our result in context, we shall adopt a notion used in analyzing the convergence rate of parameter estimation in location-scale Gaussian mixtures in [16].
For any k  1 , let r ( k ) be the minimum value of r such that the following system of polynomial equations:
[Equation]
does not have any nontrivial solution for the unknown variables ( a j , b j , c j ) k +1 j =1 , where the ranges of n 1 and n 2 in the second sum consist of all natural pairs satisfying the equation n 1 + 2 n 2 = ↵ .
A solution to the above system is considered nontrivial if all of variables c
j are non-zeroes, while at least one of the a j is non-zero.
Some examples of known values of r are r (1) = 4 and r (2) = 6 , and r ( k )  7 for all k  3 .
Using this notion, we can characterize the convergence of parameters of model (1) for the location-scale Gaussian family via the following theorem for inverse bounds.
Theorem 3.5.
Assume that G
⇤ 2 E k
⇤ ,c 0 ( ⇥ ) , and k ⇤ is unknown and strictly upper bounded by a given K .
In addition, f is location-scale Gaussian distribution and ( f, K ) with varied location, ﬁxed variance parameters is distinguishable in any order from h 0 .
Then, for any G 2 O K,c 0 ( ⇥ ) , there exist positive constant C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following holds:
(a) When  ⇤ = 0 , then V ( p  ⇤ G
⇤ , p  G )  C 1  .
(b) When  ⇤ 2 (0 , 1] , then V ( p  ⇤ G ⇤ , p  G )  C 2 W r
( K  k ⇤ )
(  G,  ⇤ G ⇤ ) .
The proof technique of this result involves doing Taylor expansion of both location and scale parameter up to order r , then utilize the heat equation @ f @ ⌃ ( x | µ, ⌃ ) =
1 2 @ 2 f @ µ 2 ( x | µ, ⌃ ) to compress this expression into linear combination of h 0 and derivatives of f ( x | µ, ⌃ ) with respect to µ only.
This allows us to use the condition in this theorem to imply a contradiction, and gives rise to Eq.
3.4 Inverse Bounds in Partially Distinguishable Setting
What happens if the distinguishability condition required by Def.
2.2 no longer holds generally?
Recall in Example 2.3 (b) that this situation is not uncommon, speciﬁcally when
[Equation]
where G 0 :
=
P k 0 i =1 p
i  ✓ 0 i .
In some speciﬁc cases of this setting, in fact, we fail to attain distin- guishability, and the model may not even be identiﬁable in the classical sense, i.e. p  G = p
⇤ G ⇤ does not guarantee to have  G =
⇤ G
⇤ .
Since h 0 is the pdf of a mixture distribution — a popular choice for modeling complex forms of probability densities given its amenability to interpretation compared to black box type models — it is of interest to study the implication of parameter estimation for the deviated components in this setting, provided that the distinguisability condition may be at least partially achieved in some suitable sense.
As we shall see, our theory demands a more reﬁned analysis.
To facilitate the presentation, denote A :=
1  i  k ⇤ : ✓ ⇤ i 2 { ✓ 01 , . .
, ✓ 0 k 0 } .
Also, set ¯ k := |A| , which stands for the cardinal of the set A .
Our results will be divided into three separate regimes of ¯ k and  ⇤ : (i)  ⇤ = 0 , (ii) ¯ k < k 0 and  ⇤ 2 (0 , 1] , and (iii) ¯ k = k 0 and  ⇤ 2 (0 , 1] .
We only choose to present results of the second regime (ii) in the main text because of limited space and because of its representativeness as it shows all the intriguing behaviours of the model in this partially distinguishable setting.
The ﬁrst and third regime are deferred to Appendix A.
3.4.1 Regime B: ¯ k < k 0 and  ⇤ 2 (0 , 1]
First, we consider the exactly-speciﬁed setting of model (1), namely, k ⇤ is known.
When ¯ k < k 0 , we can check that we still have dishtinguishability of h 0 and linear combinations of { f ( x | ✓ ⇤ i ) }
k
⇤ i =1 and its derivatives.
Therefore, as long as f is ﬁrst order identiﬁable, one can invoke the proof of
Theorem 3.3 to establish the same lower bound V ( p  G , p  ⇤ G ⇤ ) in terms of W r (  G,  ⇤ G ⇤ ) for some r  1 .
Thus, our focus in this subsection is the settings when k ⇤ is unknown.
Over-ﬁtted setting with strongly identiﬁable f .
Moving to the over-ﬁtted settings of model setup (1), i.e., k ⇤ is unknown and strictly upper bounded by a given K , as long as K  k 0 , ( f, K ) is not distinguishable from h 0 .
Therefore, the results of Theorem 3.3 are not always applicable to the setting when K  k 0 .
Besides, in the over-ﬁtted setting, the identiﬁability of model (1) no longer holds.
Indeed, for any  >  ⇤ , if we take
[Equation]
then p  ⇤ G ⇤ =
p  G
⇤ (  ) .
We present this pathological behavior in the following result.
Theorem 3.6.
Assume that h 0 takes the form (7) and ¯ k < k 0 .
Besides that, K  k 0 and f is second order identiﬁable.
Then, for any G 2 O K ( ⇥ ) , there exist positive constants C 1 and C 2 depending only on  ⇤ , G ⇤ , h 0 , ⇥ such that the following hold:
(a) If K  k ⇤ + k 0  ¯ k  1 , then V ( p  ⇤ ,G ⇤ , p  ,G )  C 1 W 2 (  G,  ⇤ G ⇤ ) , (b) If K  k ⇤ + k 0  ¯ k , then V ( p  ⇤ ,G ⇤ , p  ,G )  C 1 {    ⇤ } W 2 (  G,  ⇤ G
[Equation]
. (c) As a special case, if K = k ⇤ + k
¯ k , we have
[Equation]
for all  > 0 , where C 3 depends on  ⇤ , G ⇤ , h 0 , ⇥ ,  .
As we can see, the magnitude of  compared to  ⇤ will decide the solution of (  , G ) to the identiﬁable equation p  G = p
⇤ G
⇤ , therefore lead to different lower bounds such in part (b) of the theorem.
In particular, if    ⇤ , the solution is (  , G ) =
(  ⇤ , G ⇤ ) , and for any  >  ⇤ , the solution is
G = G ⇤ (  ) given in Eq.
Speciﬁcally, when  is strictly larger than  ⇤ by some amount  > 0 , then the latter case is well separated from the former, and we have an exact-ﬁtted result when
K = k
+ k ⇤  k .
Experiments
We now would like to demonstrate the convergence rates in Section 3 via two synthetic experiments: one for distinguishable setting and one for partially distinguishable setting.
For the partially distinguishable one, the experiments are in Appendix B.
Distinguishable setting.
We conduct an experiment where the original data distribution comes from an uniform distribution on a curve (half circle) in R 2 convoluted with Gaussian noises (red curve and blue points in Fig. 1(a)), and train a Normalizing Flow neural network [11] (Masked Autoregressive architecture) with 5 layers to get a good density estimation h 0 for this dataset.
Then we assume that there are new data coming in, and the original distribution h 0 is deviated by a mixture of distributions in the location Gaussian family f ( x | ✓ ) .
So the true generating density now is
[Equation]
where  ⇤ = 0 . 5 , G ⇤ =
P 3 i =1 p
⇤ i  ✓ ⇤ i , where p ⇤ 1 = 0 .
3 , p ⇤ 2 = 0 .
3 , p ⇤ 3 = 0 .
4 , ✓ ⇤ 1 =
(  0 .
7 , 1 . 5) , ✓ ⇤ 2 = (0 . 1 , 2 . 0) , ✓ ⇤ 3 =
Samples from the deviated component are green points in Fig. 1(a).
It can be seen from Proposition 2.4(a) that h 0 is distinguishable with family f .
For each n , we simulate n data points from true model (9), estimate ˆ  n , ˆ G n by the EM algorithm (it is possible because
Normalizing Flows provides exact density computation), and measure its convergence to the true  ⇤ , G ⇤ .
We conduct 16 replications for each sample size.
The average error estimations with a 75% error bar can be seen in Fig.
The W 1 error in the exact-ﬁtted case is of order (log( n ) /n )
1 / 2 and Werror in the over-ﬁtted case is of order (log( n ) /n )
Meanwhile, thanks to the distinguishability, the estimation errors in both cases of  are all of the order (log( n ) /n )
These simulation results are matched with the theoretical results found in Theorem 3.3 and Theorem 3.4.
From the result, we see that the deviating mixture model successfully learns the deviated components and reuses the pre-trained black box model h 0 , which helps to reduce computational costs.
[Figure]
Discussion
In this work, we have presented the deviating mixture model and studied its parameter learning rates under MLE procedure.
With a novel notion of distinguishability between distributions, we are able to prove inverse bounds for our model under several distinguishability settings, which allow us to deduce the parameter learning rates from the convergence rate of density functions.
The distinguishability condition is shown to be satisﬁed for multiple families of distributions including those that come from black box models.
We now discuss practical implication of the theory.
The deviating mixture model is designed to capture the deviated mixture components, and learning its parameters can reveal meaningful information about subpopulations in the data.
When there is distinguishability in the model, our theory implies that we can learn the deviated proportion with the parametric rate and deviated components with a rate depending on the identiﬁablity of f .
However, our theory does not support employing the deviating mixture model when the existing distribution h 0 itself is a mixture distributions in family f and possesses parameters similar to deviated part, as the learning rate can be slow, and the deviated proportion estimator may not converge to the true value.
Asymptotically, when h 0 is estimated using a very complex model (eg.
a wide and deep neural network)
and somehow approximates a mixture of f , and/or the signal from deviating components is low, then the provided learning rates in the paper, while still the same with respect to sample size n , may deteriorate from a large multiplicative constant that depends on h 0 ,  ⇤ , and G ⇤ .
We believe that this work is the ﬁrst attempt in the effort of understanding a broader class of mixture models combining with black box models, and interpreting the learned model parameters.
There is room for future work going forward.
From a theoretical viewpoint, one may be interested in establishing minimax lower bounds for the learning behavior of the deviating mixture model, or show uniform inverse bounds for the model when  ⇤ and G ⇤ are considered as signals that will change with samples.
From a modeling viewpoint, it is worthwhile to explore mixtures of black box models and develop a suitable notion of identiﬁability and inverse bounds so that the learning process is efﬁcient.
Acknowledgements
Nhat Ho acknowledges support from the NSF IFML 2019844 and the NSF AI Institute for Foundations of Machine Learning.
Long Nguyen is partially supported by NSF grant DMS-2015361.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section(b)
Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] It can be seen in the source code.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 1, 2, 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] The experiments are run on CPU’s only.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[No] We do not use any existing assets.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] We do not use crowdsourcing/conducted research with human subjects (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]