A Fourier Approach to Mixture Learning
Abstract
We revisit the problem of learning mixtures of spherical Gaussians.
Given samples from mixture 1 k
P kj =1 N ( µ j , I d ) , the goal is to estimate the means µ 1 , µ 2 , . . .
, µ k 2 R d up to a small error.
The hardness of this learning problem can be measured by the separation  deﬁned as the minimum distance between all pairs of means.
Regev and Vijayaraghavan [2017] showed that with  = ⌦ ( p log k ) separation, the means can be learned using poly( k, d ) samples, whereas superpolynomially many samples are required if  =
o ( p log k ) and d = ⌦ (log k ) .
This leaves open the low-dimensional regime where
d = o (log k ) .
In this work, we give an algorithm that efﬁciently learns the means in d =
O (log k/ log log k ) dimensions under separation d/ p log k
(modulo doubly logarithmic factors).
This separation is strictly smaller than p log k , and is also shown to be necessary.
Along with the results of Regev and Vijayaraghavan [2017], our work almost pins down the critical separation threshold at which efﬁcient parameter learning becomes possible for spherical Gaussian mixtures.
More generally, our algorithm runs in time poly( k ) · f ( d,  , ✏ ) , and is thus ﬁxed-parameter tractable in parameters d ,  and ✏ .
Our approach is based on estimating the Fourier transform of the mixture at carefully chosen frequencies, and both the algorithm and its analysis are simple and elementary.
Our positive results can be easily extended to learning mixtures of non-Gaussian distributions, under a mild condition on the Fourier spectrum of the distribution.
Introduction
Gaussian mixture models (GMMs) are one of the most well studied models for a population with different components.
A GMM deﬁnes a distribution over the d -dimensional Euclidean space as the weighted sum of normal distributions
P ki =1 w i · N ( µ i , ⌃ i ) , which are speciﬁed by following quantities: the number of components k 2 N , the component means µ i 2 R d , the component covariances ⌃ i 2 R d ⇥ d , which are positive deﬁnite matrices, and the weights w i  0 that sum up to 1 .
In this work, we consider the uniform spherical case, where the weights w i are uniform w i = 1 k and the covariance matrix ⌃ i
=
I
d is the identity matrix.
The central problem in this setup is to efﬁciently estimate the means µ 1 , . . . , µ k .
To avoid degenerate cases such as when some of the means are the same, it is common to parameterize the problem by the separation of the means  , which guarantees that k µ i  µ j k 2   for all i 6 = j .
More precisely, the problem is to estimate the means µ 1 , . . .
, µ k 2 R d up to an error ✏ with runtime that is poly( k, 1 ✏ , d ) with as small a separation  as possible.
There has been a long line of work on this problem which we survey in Section 1.3.
Recently, Regev and Vijayaraghavan [2017] showed that a separation  = ⌦ ( p log k ) is strictly necessary when the dimension d = ⌦ (log k ) .
Two natural questions arise immediately.
First, if  = p log k is sufﬁcient when d = ⌦ (log k ) .
Although the original work of Regev and Vijayaraghavan [2017] showed that it was information theoretically possible, an actual efﬁcient algorithm was only recently developed by Liu and Li [2022] (who show nearly tight results).
The second main question is determining the optimal separation in low dimensions when d = o (log k ) .
Previously, even in O (1) dimensions, the exact separation necessary was unknown.
In this paper, we settle the second question and give nearly optimal bounds on the separation necessary in low dimensions (see Figure 1 for more details).
1.1 Overview of Results
We begin with a few deﬁnitions.
A point set { x 1 , x 2 , . . .
, x k } is called  -separated if k x j  x j 0 k 2   for any j 6 = j 0 .
We say that a Gaussian mixture is  -separated (or has separation  ) if the means of its components are  -separated.
Two point sets { u 1 , . . .
, u k } and { v 1 , . . .
, v k } are ✏ -close if for some permutation  over [ k ] , k u j  v  ( j ) k 2  ✏ holds for every j .
Our main result is an algorithm that efﬁciently learns the parameters of a mixture of k spherical
Gaussians under separation  ⇡ d p log k · poly(log log k ) in d = O (log k/ log log k ) dimensions.
In the low-dimensional regime, this separation is strictly smaller than min { p log k, p d } , the smallest separation under which previous algorithms could provably learn the parameters in poly( k ) time.
Theorem 1.1 (Upper bound, informal) .
Let P be a uniform mixture of k spherical Gaussians in d = O ⇣ log k loglog k ⌘ dimensions with separation  = ⌦ ✓
[Equation]
p
[Equation]
◆ .
There is a poly( k ) -time algorithm that, given samples from P , outputs k vectors that are w.h.p.
✏ -close to the true means for ✏ =
O (  / p d ) .
See Theorem 2.1 and Remark 2.2 for a more formal statement of our algorithmic result, which holds for a wider range of separation  and accuracy parameter ✏ .
Our learning algorithm is provably correct for arbitrarily small  , ✏ > 0 (possibly with a longer runtime), whereas for most of the previous algorithms, there is a breakdown point
⇤ such that the algorithm is not known to work when  <  ⇤ .
Two exceptions are the algorithms of Moitra and Valiant [2010], Belkin and Sinha [2010], both of which allow an arbitrarily small separation but run in e ⌦ ( k ) time.
We also remark that the runtime of our algorithm scales as ˜ O
( k 3 ) · f ( d,  , ✏ ) , and is thus ﬁxed-parameter tractable in parameters d ,  and ✏ .
We complement Theorem 1.1 with an almost-matching lower bound, showing that the d/ p log k separation is necessary for efﬁcient parameter learning in low dimensions.
Theorem 1.2 (Lower bound, informal) .
For d = O ⇣ log k loglog k ⌘ and  =
o ⇣ d p log k ⌘ , there are two mixtures of k spherical Gaussians in R d such that: (1) both have separation  ; (2) their means are not (  / 2) -close; and (3) the total variation (TV) distance between them is k  !
See Theorem E.1 for a more formal version of the lower bound.
Theorem 1.1 and Theorem 1.2 together nearly settle the polynomial learnability of spherical Gaussian mixtures in the low-dimensional regime.
Up to a doubly logarithmic factor, the “critical separation” where efﬁcient learning becomes possible is d/ p log k .
To the best of our knowledge, this was previously unknown even for d = O (1) .
See Figure 1 below for a plot of our results in the context of prior work.
The green regions cover the parameters (  , d ) such that mixtures of k spherical Gaussians in d dimensions with separation  are separation Δ dimension " log
[Figure] [Figure]
/ log
[Figure]
log
[Figure]
Learnable with poly & samples
Not poly & -sample learnable
[Figure]
learnable (up to O (  ) error) using poly( k ) samples.
The red regions contain the parameters under which polynomial-sample learning is provably impossible.
The algorithm that underlies Theorem 1.1 can be easily extended beyond the spherical Gaussian case.
The following more general result states that for any distribution D whose the Fourier transform does not decay too fast, we can efﬁciently learn the parameters of a mixture of k translated copies of D .
In the following, let D µ denote the distribution of X + µ when X is drawn from D .
Theorem 1.3 (Learning more general mixtures, informal) .
Let P = 1 k
P kj =1 D µ j for  -separated µ 1 , . . .
, µ k 2 R d .
There is an algorithm that, given ✏ > 0 and samples from P , runs in time poly ✓
[Equation]
i
[Equation]
◆ for some  =  ( D , ✏ ) and M =
M ( k, d,  , ✏ ) , and outputs ˆ µ 1 , . . .
, ˆ µ k that are w.h.p.
✏ -close to the true parameters.
See Theorem F.1 and Corollary F.2 for a more formal statement of the runtime.
Theorem 1.3 applies to many well-known distribution families that are speciﬁed by either a “location parameter” or a “scale parameter”.
Table 1 gives a few examples of applying Theorem 1.3 to mixtures of single-parameter univariate distributions; see Appendix F.2 and Appendix F.3 for more details.
Limitations of our work.
The main limitation is that the positive results only apply to the regime that the dimension d is logarithmic in the number of clusters, and that all clusters are translated copies of the same distribution.
A concrete future direction would be to extend our results to learning mixtures of general Gaussians, even in one dimension.
1.2 Proof Overview
For simplicity, we focus on the one dimensional case that P = 1 k
P kj =1 N ( µ j , 1) for  -separated means µ 1 , . . .
, µ k 2 R .
We index the components such that | µ 1 |  | µ 2 |  · · ·  | µ k | , and focus
[Table]
on the following testing problem: Given ✏ > 0 and samples from P , determine whether µ 1 = 0 or µ 1  ✏ , assuming that one of them is true.
Note that this testing problem is not harder than estimating µ 1 , . . .
, µ k up to error ✏ / 3 —in the former case that µ 1 = 0 , one of the mean estimates would fall into [  ✏ / 3 , ✏ / 3] , whereas all of them must be outside (  2 ✏ / 3 , 2 ✏ / 3) in the latter case.
Conversely, as we will prove in Section 2, an algorithm for the testing version can be used for recovering the means as well.
Examine the Fourier spectrum.
We start by examining the Fourier transform of P , ( F P )( ⇠ ) :=
E X ⇠ P ⇥ e i ⇠ X ⇤ , more commonly known in the literature as the characteristic function .
Since the
Fourier transform of a Gaussian is still a Gaussian, and a translation in the time domain shifts the phase in the frequency domain, we have
[Equation]
We will focus on the quantity A µ ( ⇠ ) :=
P kj =
e iµ j ⇠ , which is the “total phase” over the k components of P .
Equation (1) essentially states that A µ ( ⇠ ) can be estimated by averaging e i ⇠ X over samples
X ⇠ P .
The key observation is that each term e iµ j ⇠ of A µ ( ⇠ ) behaves quite differently depending on the magnitude of µ j : If µ j = 0 , e iµ j ⇠ = 1 is a constant, whereas e iµ j ⇠ is a high-frequency wave when | µ j | is large.
This suggests that the two cases ( µ 1 = 0 and | µ 1 |  ✏ ) can be distinguished by estimating A µ ( ⇠ ) at different frequencies.
The cost of estimating A µ ( ⇠ ) , however, depends heavily on the frequency ⇠ – Equation (1) together with a simple concentration bound shows that O ( k 2 ) · e O ( ⇠ 2 ) samples are sufﬁcient for estimating A µ ( ⇠ ) up to a constant error.
Therefore, the crux of this approach is to ﬁnd the minimum M >
such that the two cases can be distinguished by estimating A µ ( ⇠ ) over ⇠ 2 [  M, M ] .
(This is known as the super-resolution problem, which we discuss in Section 1.3.)
The sample complexity of the testing problem can then be roughly bounded by O ( k 2 ) ·
e O ( M 2 ) .
In the following, we explore different ways of picking ⇠ from the range [  M, M ] .
Choosing ⇠ randomly.
Our overall strategy is to draw ⇠ randomly from some distribution D ⇠ over interval [  M, M ] and evaluate E ⇠ ⇠ D ⇠ [ A µ ( ⇠ )] .
We will argue that this expectation is very close to its ﬁrst term E ⇠ ⇠ D ⇠ ⇥ e iµ 1 ⇠ ⇤ , which takes different values depending on whether µ 1 = 0 or | µ 1 |  ✏ .
Then, D ⇠ needs to be chosen such that: (1) There is a gap in the value of E ⇠ ⇠ D ⇠ ⇥ e iµ 1 ⇠ ⇤ between the two cases; (2)
[Equation]
⇥ e iµ j ⇠ ⇤ is small enough for the gap in the ﬁrst term to be easily identiﬁed.
As a warmup, let D ⇠ be the uniform distribution over [0 , M ] .
A simple calculation shows that the gap in the ﬁrst term E ⇠ ⇠ D ⇠ ⇥ e iµ 1 ⇠ ⇤ between the two cases ( µ 1 = 0 or | µ 1 |  ✏ ) is lower bounded by ⌦ (min { M ✏ , 1 } ) , whereas the contribution from the j -th component satisﬁes
E
[Equation]
⇠ ⇥
[Equation]
⇠ ⇤ =
O ⇣
[Equation]
M
[Equation]
⌘ .
Furthermore, the  -separation between the means implies | µ j | = ⌦ (  j ) .
Thus,
[Equation]
h
[Equation]
i .
k
X j
[Equation]
M
[Equation]
µ
[Equation]
| .
M  k
X j =  j . log k M  .
The above is much smaller than min { M ✏ , 1 } if we set M & max ⇢ log k  ,
q log k  ✏  .
Unfortunately, even when  and ✏ are constants, we have O ( k 2 ) ·
e O ( M 2 ) =
k O (log k ) and the resulting sample complexity is already super-polynomial in k .
A better choice of D ⇠ .
It turns out that choosing D ⇠ to be a truncated Gaussian leads to a much lower sample complexity.
For some  ⌧ M , we draw ⇠ ⇠ N (0 ,  2 ) and then truncate it to [  M, M ] .
Without the truncation, the expectation of e iµ j ⇠ has a nice closed form:
[Equation]
which is exactly the Fourier weight of N (0 ,  2 ) at frequency µ j .
Note that this decreases very fast as | µ j | grows, compared to the previous rate of
[Equation]
when D ⇠ is uniform.
It again follows from a simple calculation that: (1) Depending on whether µ 1 = 0 or | µ 1 |  ✏ , the gap between E ⇠ ⇥ e iµ 1 ⇠ ⇤ is ⌦ (min {  2 ✏ 2 , 1 } ) ; (2) The total contribution from j = 2 , 3 , . . .
, k is upper bounded by
[Equation]
2 ) , where the second step applies | µ j | =
⌦ (  j ) and the last step holds assuming  = ⌦ (1 /  ) .
In addition, we need to deal with the error incurred by the truncation.
The Gaussian tail bounds imply that | ⇠ |  M happens with probability e  ⌦ ( M 2 /  2 ) .
Since | A µ ( ⇠ ) |  k for any ⇠ , the noise from the truncation is at most k · e  ⌦ ( M 2 /  2 ) in magnitude.
It remains to choose M,  > 0 that satisfy the two inequalities:
[Equation]
It sufﬁces to choose  .
1
[Equation]
q
[Equation]
For ✏ = ⌦ (  ) , the sample complexity O ( k 2 ) ·
e O ( M 2 ) reduces to k O (1 /  2 ) , which is polynomial in k for any ﬁxed  .
We note that in the above derivation, the assumption that each cluster of P is a Gaussian is only applied in Equation (1) (through the Fourier transform).
For any distribution D over R and P = 1 k
P kj =1 D µ j , the quantity A µ ( ⇠ ) =
P kj =1 e iµ j ⇠ can still be read off from the Fourier transform of P at frequency ⇠ , except that the extra factor e  ⇠ 2 / 2 becomes ( FD )( ⇠ ) , the Fourier transform of D at frequency ⇠ .
This observation leads to our algorithm for learning a mixture of multiple translated copies of D (Theorem 1.3).
Gaussian truncation.
For the spherical Gaussian case, however, the above only gives an algorithm with runtime k O (1 /  2 ) , which becomes super-polynomial as soon as  = o (1) , falling short of achieving the near-optimal separation of q
[Equation]
k in Theorem 1.1.
We further improve our algorithm for the spherical Gaussian case using a “Gaussian truncation” technique.
Intuitively, when deciding whether the mixture P contains a cluster with mean zero, a sample X ⇠ P is much more informative when | X | is small.
This motivates us to focus on samples with a small magnitude via a truncation.
We apply such a truncation in a soft way—weighting each sample X with e  X 2 / 2 .
This turns out to be sufﬁciently effective while keeping the entire analysis simple.
Note that the weighting effectively multiplies the mixture P with the standard Gaussian N (0 , 1) pointwise.
The result is still a (un-normalized) mixture of Gaussians: Up to a constant factor, the pointwise product isk
P kj =1 e  µ 2 j / 4 · N ( µ j / 2 , 1 / 2) .
Consequently, if we repeat the analysis from previous paragraphs to this weighted mixture, the noise coming from components with large | µ j | become even smaller.
This eventually allows us to learn the mixture efﬁciently at separation  = q loglog k log k .
We prove
Theorem 1.1 via a natural extension of this analysis to the d -dimensional case.
Lower bound.
Our proof of Theorem 1.2 follows an approach similar to those in the previous lower bound proofs of Moitra and Valiant [2010], Hardt and Price [2015], Regev and Vijayaraghavan [2017]: First, construct two sets of well-separated points { µ ( P ) 1 , . . .
, µ ( P ) k } and { µ ( Q ) 1 , . .
. , µ ( Q ) k } with (approximately) matching lower-order moments, i.e., 1 k
P kj = h µ ( P ) j i ⌦ t ⇡ 1 k
P kj = h µ ( Q ) j i ⌦ t for every small t .
Then, show that these matching moments imply that after a convolution with
Gaussian, the resulting mixtures are close in TV-distance and thus hard to distinguish.
In more detail, similar to the proof of Regev and Vijayaraghavan [2017], we start by choosing N arbitrary points from a small ` 2 ball, such that they are  -separated.
The main difference is that Regev and Vijayaraghavan [2017] pick N  k , and show that among all the
Nk  possible mixtures, there are at least two mixtures with similar moments via a pigeon-hole type argument.
In contrast, we work in the N ⌧ k regime, and obtain two point sets with matching lower-order moments by slightly perturbing these N points in opposite directions.
The existence of such a good perturbation is shown via a careful application of the Borsuk–Ulam Theorem, which is inspired by a similar application in Hardt and Price [2015].
1.3 Related Work
Learning Gaussian mixture models.
Most closely related to this paper is the line of work in the theoretical computer science literature on algorithms that provably learn mixtures of Gaussians.
The pioneering work of Dasgupta [1999] showed that an ⌦ ( p d ) separation between the means of different components is sufﬁcient for the samples to be easily clustered.
Several subsequent work (e.g., Vempala and Wang [2004], Arora and Kannan [2005], Achlioptas and McSherry [2005], Kannan et al.
[2005], Dasgupta and Schulman [2007], Brubaker and Vempala [2008]) further generalized this result and the separation condition is relaxed to
= ⌦ ((min
[Equation]
by Vempala and Wang [2004].
We refer the readers to Regev and Vijayaraghavan [2017] for a more detailed survey of these results.
Moitra and Valiant [2010] and Belkin and Sinha [2010] gave the ﬁrst algorithms for learning general mixtures of Gaussians (with different unknown covariances) under an arbitrarily small separation between different components.
Both algorithms are based on the method of moments, and run in time polynomial in d and the inverse of the minimum TV-distance between different components
when k
= O (1) .
For large k , however, the runtime is exponential in k .
Regev and Vijayaraghavan [2017] showed that poly( k, d ) samples are sufﬁcient to estimate the means of a mixture of spherical Gaussians, if the means are separated by
= ⌦ ( p log k ) .
Unfortunately, their algorithm involves an exhaustive search and is computationally inefﬁcient.
Subsequently, three concurrent work [Diakonikolas et al., 2018, Hopkins and Li, 2018, Kothari et al., 2018] developed algorithms based on the Sum-of-Squares hierarchy that run in time ( dk )
poly(1 /  ) and learn mixtures with separation  = ⌦ ( k  ) .
In particular, setting
[Figure]
gives a quasi-polynomial time algorithm that achieves the p log k separation in Regev and Vijayaraghavan [2017].
A very recent work of Liu and Li [2022] made further progress towards learning ⌦ ( p log k ) -separated mixtures efﬁciently, by giving a polynomial-time algorithm that succeeds under separation (log k ) 1 / 2+ c for any constant c > 0 .
Lower bounds.
On the lower bound side, Moitra and Valiant [2010] ﬁrst proved, by explicitly constructing a hard instance, that learning a mixture of k Gaussians with separation  = ⌦ (1 / p k ) requires e ⌦ ( k ) samples even in d = 1 dimension.
Hardt and Price [2015] focused on the regime where
k
= O (1) and the recovery error ✏ goes to zero, and showed that ⌦ ( ✏ 2  6 k ) samples are necessary.
Anderson et al.
[2014] proved a lower bound that strengthens the result of Moitra and Valiant [2010]:
Separation  = 1 / poly( k ) is insufﬁcient for the mixture to be learnable with polynomially many samples, even when the means of the Gaussians are drawn uniformly at random from [0 , 1] d for d
= O (log k/ log log k ) .
Regev and Vijayaraghavan [2017] proved that in d = ⌦ (log k ) dimensions, no polynomial-sample algorithm exists if  =
o ( p log k ) .
In particular, even when the means are chosen randomly, the resulting mixture is still hard to learn with high probability.
This lower bound, together with their positive result for the  = ⌦ ( p log k ) regime, shows that p log k is the critical separation threshold in high dimensions.
If we restrict our attention to statistical query (SQ) algorithms, Diakonikolas et al.
[2017] proved that any SQ algorithm needs d ⌦ ( k ) queries if d  poly( k ) and non-spherical components are allowed.
Density estimation.
In contrast to the parameter estimation setting that we focus on, the density estimation setting requires the learning algorithm to output (the representation of) a hypothesis distribution ˆ P that is close to the mixture P in TV-distance.
Ashtiani et al.
[2020] proved that the sample complexity of learning Gaussian mixtures up to a TV-distance of ✏ is ˜ ⇥ ( d 2 k/ ✏ 2 ) in general, and ˜ ⇥ ( dk/ ✏ 2 ) if all components are axis-aligned.
Unfortunately, their learning algorithms are not computationally efﬁcient and run in time exponential in k .
For the spherical case, Diakonikolas and Kane [2020] gave a more efﬁcient algorithm that runs in quasi-polynomial time poly( d ) · ( k/ ✏ ) O (log 2 k ) .
In the proper learning setting (i.e., ˆ P must be a Gaussian mixture), Suresh et al.
[2014] gave an algorithm that takes poly( dk/ ✏ ) samples and runs in time poly( d ) · ( k/ ✏ ) O ( k 2 ) .
Super-resolution, and mixture learning using Fourier analysis.
Finally, we acknowledge that the Fourier approach that we explore in this work is fairly natural, and similar ideas have appeared in prior work.
In particular, our approach is closely related to the super-resolution problem (Donoho [1992], Candès and Fernandez-Granda [2013, 2014])—to recover k unknown locations µ 1 , . . . , µ k
R d from (exact or noisy) observations of the form
P kj =1 e iµ j > ⇠ for k ⇠ k  M , where M is called the cutoff frequency, and the norm is typically ` 2 or ` 1 .
In comparison, our approach (outlined in
Section 1.2) differs in two aspects: (1) We focus on a simpler testing version of the problem—to decide whether one of the locations is near a given reference point; (2) The Gaussian truncation signiﬁcantly down-weights the observation coming from the points that are far from the reference point.
For the d = 1 case, Moitra [2015] showed that a cutoff frequency of M = O (1 /  ) sufﬁces if the points are  -separated.
This implies an algorithm that efﬁciently learns spherical Gaussian mixtures in one dimension under separation 1 / p log k , which is (nearly) recovered by our positive result.
For the general case, Huang and Kakade [2015] gave an algorithm that provably works for ( ` 2 ) cutoff frequency M =
O
(( p d log k
+ log k ) /  ) .
When applied to learning GMMs, however, their algorithm requires separation p d .
It is conceivable that the algorithm of Huang and Kakade [2015], equipped with appropriate modiﬁcations and a tighter analysis (e.g., the approach of Chen and Moitra [2021]), might give a guarantee similar to ours, but no such analysis is explicit in the literature to the best of our knowledge.
Moreover, our approach leads to an arguably simpler algorithm with an elementary analysis.
More recently, Chakraborty and Narayanan [2020] gave an algorithm that is similar to ours for learning mixtures of spherical Gaussians via deconvolving the mixture.
However, their algorithm is only shown to work when  = ⌦ ( p d ) .
Chen et al.
[2020] studied the problem of learning mixtures of linear regressions (MLRs), which can be reduced to estimating the minimum variance in a mixture of zero-mean Gaussians.
They solved this problem by estimating the Fourier moments – the moments of the Fourier transform, and gave the ﬁrst sub-exponential time algorithm for learning MLRs.
Chen and Moitra [2021] studied learning mixtures of Airy disks, a problem that is motivated by the physics of diffraction.
Their algorithm also proceeds by ﬁrst estimating the Fourier transform of the mixture, and then dividing it pointwise by the Fourier spectrum of the “base” distribution.
1.4 Organization of the Paper
In Section 2, we formally state our main algorithmic result as well as our main technical theorem, which addresses a testing version of the problem.
We then sketch how our upper bound follows from a simple reduction from parameter learning to testing.
In Section 3, we give a simple algorithm that solves this testing version by examining the Fourier transform of the mixture.
The pseudocode of our algorithms are presented in Appendix A. We defer a few technical proofs to
Appendices B through D. We prove our lower bound result (Theorem 1.2) in Appendix E. Finally, in Appendix F, we formally state and prove the guarantees for learning non-Gaussian mixtures and present a few applications of this result.
From Learning to Testing
We ﬁrst state our main positive result – the formal version of Theorem 1.1.
Theorem 2.1.
Let P = 1 k
P kj =1 N ( µ j , I d ) be a uniform mixture of k  2 spherical Gaussians with  -separated means in R d and ✏ < min {  / 100 ,  / ( p min { d, ln k } ) } .
There is an algorithm (Algorithm 1) that, given samples from P , outputs k vectors that are ✏ -close to µ 1 , . . . , µ k with high probability.
The runtime (and thus the sample complexity) of the algorithm is upper bounded by
[Equation]
where M 2 .
1  2 (min { d, log k } + log  ✏ ) · (min { d + log k, d log(2 + d  2 ) }
+ log  ✏ ) .
Remark 2.2 .
When d = O ⇣ log k loglog k ⌘ and  / ✏ = ⇥ ( p d ) , the runtime can be simpliﬁed into
[Equation]
◆◆◆ , which is poly( k ) if  = ⌦ ✓ d p log k · q log log k d ◆ .
When d = o (log k ) , this condition is strictly looser than both  = ⌦ ( p d ) and
= ⌦ ( p log k ) .
We prove Theorem 2.1 by reducing the parameter learning problem into the following testing version:
Given samples from mixture P , determine whether P contains a cluster with a mean that is close to a given guess µ ⇤ 2 R d .
Formally, we prove the following theorem:
Theorem 2.3.
Let P = 1 k
P kj =1 N ( µ j , I d ) be a uniform mixture of k  2 spherical Gaussians with  -separated means in R d .
Let ✏ < min {  / 100 ,  / ( p min { d, ln k } ) } and µ ⇤ 2 R d .
There is an algorithm (Algorithm 2) that, given samples from P , either “accepts” or “rejects”, such that it:
• Accepts with probability  2 / 3 if min j 2 [ k ] k µ j  µ ⇤ k
 ✏ / 2 . • Rejects with probability  2 / 3 if min j 2 [ k ] k µ j  µ ⇤ k 2  ✏ .
The runtime (and thus the sample complexity) of the algorithm is upper bounded by O ( k 2 (  / ✏ ) 4 ) ·
e O ( d + M 2 ) , where M 2 . 1  2 (min { d, log k } + log  ✏ ) · (min { d + log k, d log(2 + d  2 ) }
+ log  ✏ ) .
Assuming the above theorem, the proof of our main theorem is straightforward and is thus deferred to
Appendix C.
The proof proceeds by ﬁrst drawing a few samples X 1 , . . .
, X N from the mixture, and then running the tester (from Theorem 2.3) to decide whether each X i is close to one of the mean vectors of the mixture.
Then, a simple argument shows that the samples that the tester accepts can be easily clustered to recover the means.
Note that if a sample X i comes from the j -th component
N ( µ j , I d ) , there is a decent probability (e.g., ⌦ ( ✏ ) when d = 1 ) that X i is ✏ -close to µ j .
Thus, we can guarantee that X 1 , . . .
, X N contain good guesses for all the k means for moderately large N .
Solve the Testing Problem using Fourier Transform
Now we solve the testing problem and prove Theorem 2.3.
We assume without loss of generality that µ ⇤ = 0 , since we can reduce to this case by subtracting µ ⇤ from each sample from P .
We also re-index the unknown means µ 1 , . . . , µ k so that k µ j k 2 is non-decreasing in j .
The testing problem is then equivalent to deciding whether k µ 1 k 2  ✏ / 2 or k µ 1 k 2  ✏ .
The following simple lemma, which we prove in Appendix D, gives the Fourier transform of the mixture P when a “Gaussian truncation” of e k x k 22 / 2 is applied.
Lemma 3.1.
For P = 1 k
P kj =1 N ( µ j , I d ) and any ⇠ 2 R d ,
[Equation]
where we deﬁne A µ ( ⇠ ) :
=
P
[Equation]
We will show that A µ ( ⇠ ) behaves quite differently depending on whether k µ 1 k 2  ✏ / 2 or k µ 1 k 2  ✏ .
Thus, we can solve the testing problem by estimating A µ ( ⇠ ) for carefully chosen ⇠ .
Concretely, we will draw ⇠ from N (0 ,  2 I d ) and then truncate it to B (0 , M ) :
= { x 2 R d : k x k 2  M } , for parameters M,  > 0 to be chosen later.
Formally, we focus on the following expectation:
[Equation]
The key step of our proof of Theorem 2.3 is to show that T µ is close to e  (  2 / 2+1) k µ 1 k 22 / 4 , and is thus helpful for deciding whether k µ 1 k 2  ✏ / 2 or k µ 1 k 2  ✏ .
The following lemma, the proof of which is relegated to Appendix D, helps us to bound the difference between T µ and e  (  2 / 2+1) k µ 1 k 22 / 4 .
Lemma 3.2.
For any M,  > 0 that satisfy M 2 /  2  5 d ,
[Equation]
j k 22 / 4 , where the O ( x ) notation hides a complex number with modulus  x .
By Lemma 3.2, the difference T µ  e  (  2 / 2+1) k µ 1 k 22 / 4 is given by
[Equation]
Let S 1 :=
P
[Equation]
and S 2 :=
P
[Equation] [Equation]
4 denote the two summations above.
We have the following upper bounds on S 1 and S 2 , which we prove in Appendix D:
Claim 3.3 .
Assuming (  2 / 2 + 1)  2  100 min { ln k, d } , S 1  2 e  (
[Equation]
2+1)
[Equation]
64 · min { k, 2 d } .
Claim 3.4 .
We have S 2  (
[Equation]
d. Furthermore, S 2 
[Equation]
min
[Equation]
Now we put all the pieces together and prove Theorem 2.3.
Proof of Theorem 2.3.
By Lemma 3.1,
[Equation]
i
[Equation]
Since the term inside the expectation has modulus  k · e O ( d + M 2 ) , a Chernoff bound implies that we can estimate T µ up to any additive error  > 0 using O (( k/  ) 2 ) ·
e O ( d + M 2 ) samples from P .
By Lemma 3.2 and our deﬁnition of S 1 and S 2 , assuming M 2 /  2  5 d ,
[Equation]
In the rest of the proof, we will pick  and M carefully so that both S 1 and e  M 2 / (5  2 ) S 2 are upper bounded by  :=
(  2 / 2 + 1) ✏ 2 / 64 .
Assuming this, we are done: We can simply estimate T µ up to an additive error of  .
Let c
T µ denote the estimate.
We accept if and only if Re c
[Equation] [Equation]
i .
Indeed, suppose that k µ 1 k 2  ✏ / 2 , we have
Re
[Equation]
Similarly, Re c
[Equation]
+
[Equation] [Equation]
+ 3
[Equation]
k µ 1 k 2  ✏ .
If we set  such that (  2 / 2 + 1) ✏ 2  1 , we have e  (  2 / 2+1) ✏ 2 / 16  e  (  2 / 2+1) ✏ 2 / 4  18 (  2 / 2 + 1) ✏ 2 = 8  , which implies Re c
T µ > ✓ in the former case and Re c
T µ < ✓ in the latter case.
Therefore, our algorithm decides correctly.
Choice of parameters.
Claim 3.3 implies that if we set  2 / 2 + 1 = 512
[Equation]
ln
[Equation]
ln
[Equation]
we can ensure S 1  (  2 / 2 + 1) ✏ 2 / 64 =  .
Furthermore, this choice of  and the assumption ✏ < min {  / 100 ,  / ( p min { d, ln k } ) } guarantee the condition (  2 / 2 + 1) ✏ 2  1 that we need.
It remains to pick M such that e  M 2 / (5  2 ) · S 2 
= (  2 / 2+1) ✏ 2 / 64 .
We also need M 2 /  2  5 d to ensure that Lemma 3.2 can be applied.
It sufﬁces to let M 2  5  2 · ⇣ d + ln S
+ ln 64 (  2 / 2+1) ✏
[Equation]
⌘ .
Our choice of  guarantees ln
[Equation]
✏ , so it is in turn sufﬁcient to pick M such that
[Equation]
Applying Claim 3.4 shows that M can be chosen such that
[Equation]
Runtime.
The runtime of our algorithm is dominated by the number of samples drawn from
P — O (( k/  ) 2 ) · e O ( d + M 2 ) , where  = ⇥ ((  2 + 1) ✏ 2 ) .
Plugging our choice of  into  gives 1 /
= O
((  / ✏ ) 2 ) .
The runtime can thus be upper bounded by O ( k 2 (  / ✏ ) 4 ) · e O ( d + M 2 ) .
Acknowledgments
We would like to thank the anonymous reviewers of earlier versions of this paper for their suggestions on the presentation and for pointers to the literature.