Paper: On Gaussian Mixture Models (ICLR-2024)

Reviewer comment (Writing):
There are several mistakes and typos which make this theoretical paper hard to read.

Author response:
I am sorry for the problem with the title and the writing. I explained the circumstance in my general response to reviewers.

Reviewer comment (Presentation):
What is Corollary 0.1 a corollary to? Rename it to a Proposition?

Author response:
Will fix it.

Reviewer comment (Theory):
Sentence below equation (0.2): make it tighter, maybe $\Omega(\epsilon^\ell)$.

Author response:
Done.

Reviewer comment (Theory):
Theorem 0.3: first line: mean does not seem to be 0? There are some other places as well where the mean is stated to be 0.

Author response:
Will fix it, thanks.

Reviewer comment (Writing):
Equation (1.2): $\mathcal{M}_d$ should be $\mathcal{M}_m$?

Author response:
Fixed.

Reviewer comment (Theory):
Sentence below eq (1.3): The variance of $x^m$ is not $\sigma^{2m}$. You seem to be missing a constant?

Author response:
Do you mean that the variance of $1/n \sum (x_i)^m$ is a constant time $\sigma^{2m}/n?$

Reviewer comment (Writing):
Sentence above Lemma 2.1: imperial → empirical.

Author response:
Fixed.

Reviewer comment (Theory):
Lemma 2.1 last line: less than "or equal to" $2i-1$.

Author response:
Fixed.

Reviewer comment (Theory):
Eq (2.9): maybe write one/two steps more to explain it.

Author response:
Sure.

Reviewer comment (Theory):
Eq (3.2): left hand side should have a factor of $n$. Right hand side should have $j-1$ in the exponent, not $j$. Also say that $e_0 = 1$. Ditto for Definition 3.

Author response:
Done.

Reviewer comment (Theory):
Lemma 3.1: The statement is missing "with probability at least $1-\delta$".

Author response:
Fixed, Thanks.

Reviewer comment (Writing):
Eq (3.4): Is the $K$ used here defined before?

Author response:
Sorry, this is a typo. Should be $k$.

Reviewer comment (Writing):
Line below eq (3.5): $i\geq m$?

Author response:
There were two typos, so I fixed them. Thanks for reading the manuscript carefully.

Reviewer comment (Writing):
Proof 4: In the first line, $p$ should be $P$.

Author response:
Fixed.

Reviewer comment (Theory):
Eq (4.2): How to handle the case where the denominator is $0$?

Author response:
Beauzam made a comment in a remark following the theorem. When the denominator is 0, the theorem becomes void, essentially stating that $∣x−y∣<\infty$, which is a trivial assertion. This situation arises when we have a double root, implying that two of the means are equal. However, if two of the means are equal, they effectively constitute the same Gaussian component, given our assumption that variances are both equal to one. In essence, we have $N(\mu, 1)+ N(\mu, 1)= 2N(\mu, 1).$

Reviewer comment (Theory):
Display above eq (4.4): How is $|\hat P'(x)|$ bounded by $\Pi_{j\neq m} (\mu_m - \mu_j)$ here?

Author response:
Sure.

Reviewer comment (Novelty):
The proofs are mostly calculations. I believe most of the lemmas are not fundamentally novel or can be easily followed from previously known results. This paper appears to primarily reiterate known results and may be somewhat remote from making a substantial contribution.

Author response:
Thank you for taking the time to review my paper and for making reference to the work of Wu and Yang. I will compare my results with theirs and I believe this would clarify the contribution. 

In Theorem 2, Wu and Yang consider scenarios in which a mixture has $k_0$ separated clusters. To underscore the advantage of my own results, assume $k_0 =2$,  and we have 2 clusters separated by a difference of 1 + O(ε). For instance, the first cluster possesses means of {-mε, -(m-1)ε, ..., 0, ε, ..., mε}, while the second cluster contains {1-mε, 1-(m-1)ε, ..., 1, 1+ε, ..., 1+ mε}, with k = 4m. Consequently, each cluster consists of k/2 members.

Applying Wu and Yang's theorem 2, we observe that the RHS of equation (11) yields n^(1/(4k-6)). Therefore we need n > (1/ε)^(4k-6) to have the RHS of (11) proportional to ε. This implies that they achieve a saving of ε^(-4) compared to their bound in (8) in the worst-case scenario. In contrast, my Corollary 2 results in a saving of ε^(-k/2) in this case. There is a substantial distinction between saving ε^k and ε^4. This clearly underscores the advantage of my results. 

The method of moments leads to a system of equations. Given that we employ samples to approximate moments, it is necessary to consider perturbations in the coefficients of the system. Hence, it is unsurprising that there are similarities between my paper and Wu and Yang's work, including the presence of certain lemmas. However, the divergence lies in how we approach and resolve the system and evaluate the impact of perturbations. In my case, I employ Beauzamy's theorem from mathematical analysis to address this. This is a sophisticated theorem, and applying it provides the distinct advantage between our methods. 

Second part of comparison: It is well known that parameter estimation is a more challenging task than returning a distribution that closely matches the mixture in a certain norm.  In their work Wu and Yang return a distribution that is close to the target in Wasserstein distance. In my work, I directly estimate parameters. In Lemma 1, Wu and Yang explore the conditions under which one can transition from the Wasserstein distance to parameter estimation. To elaborate, let's denote the minimum distance between the means as "min," with all weights equal to 1/k. To transfer their bound on the Wasserstein distance to a bound on parameter estimation, they require more than (k/(min*ε))^(4k) samples. Hence, the inverse of the minimum distance to the power of 4k contributes to sample complexity. In contrast, my method only demands the inverse of the minimum distance to rise to the power of the number of consecutive gaps with a minimum distance, which in most cases is much smaller. Therefore they have to use excessive amounts of samples (unnecessary in almost all cases) to get the Wasserstein distance small enough to be able to apply their method in parameter estimation.  

The difference between our methods is particularly significant in practice. Consider situations where the distribution of spacing between means follows a Poisson distribution or is GUE-like in the context of the distribution of eigenvalues of large Hermitian matrices. While small gaps are still present, their probability is exceedingly low. The mere possibility of such cases drastically increases the sample complexity when applying their method, whereas my approach only amplifies the sample complexity by (1/min)^(4k * probability of small consecutive gaps), representing a substantial practical advantage.

Moving on to the third point of comparison, a crucial feature of Hardt and Price's (HP) work is their theorem 3.10, which demonstrates that sample complexity depends on the differences between two means. If the gap between means is not small, the sample complexity remains moderate.  Thus, any extension of their result should uphold this characteristic. I accomplish this by introducing the pair correlation function between the means. In their paper, Yihong Wu and Pengkun Yang also make reference to HP's work: "By carefully analyzing Pearson’s method of moments equations, HP showed that the optimal rate for two-component location-scale mixtures is Θ(n^(-1/12)); however, this approach is difficult to generalize to higher order mixtures." As they acknowledge, generalizing HP's approach is challenging, which is precisely the task I pursued in my paper. Initially, I had no idea how I should generalize the feature. However, the natural application of Beauzamy's theorem addressed this challenge.

In conclusion, our bound only matches in the worst case scenario, in any other cases my bound has a clear advantage. My method estimates parameters directly, which is a harder task. Finally, in my opinion, my paper is making a significant contribution. 

Overall your comments have been very useful to me. By comparing the paper you referenced with my results and providing the explanations outlined above, I believe my paper has been substantially improved.

Reviewer comment (Theory):
For example, aren't Lemma 2.1 and 2.2 the Hermite polynomial expansion?

Author response:
The method of moments leads to a system of equations. Given that we employ samples to approximate moments, it is necessary to consider perturbations in the coefficients of the system. Hence, it is unsurprising that there are similarities between my paper and Wu and Yang's work, including the presence of certain lemmas. However, the divergence lies in how we approach and resolve the system and evaluate the impact of perturbations. In my case, I employ Beauzamy's theorem from mathematical analysis to address this. This is a sophisticated theorem, and applying it provides the distinct advantage between our methods.

Reviewer comment (Novelty):
The authors may also want to cite the paper "Optimal estimation of Gaussian mixtures via denoised method of moments" by Yihong Wu, Pengkun Yang. Even though the problem definitions are not exactly the same, there is discernible overlap in the techniques employed to address the underlying challenges.

Author response:
Thank you for taking the time to review my paper and for making reference to the work of Wu and Yang. I will compare my results with theirs and I believe this would clarify the contribution. 

In Theorem 2, Wu and Yang consider scenarios in which a mixture has $k_0$ separated clusters. To underscore the advantage of my own results, assume $k_0 =2$,  and we have 2 clusters separated by a difference of 1 + O(ε). For instance, the first cluster possesses means of {-mε, -(m-1)ε, ..., 0, ε, ..., mε}, while the second cluster contains {1-mε, 1-(m-1)ε, ..., 1, 1+ε, ..., 1+ mε}, with k = 4m. Consequently, each cluster consists of k/2 members.

Applying Wu and Yang's theorem 2, we observe that the RHS of equation (11) yields n^(1/(4k-6)). Therefore we need n > (1/ε)^(4k-6) to have the RHS of (11) proportional to ε. This implies that they achieve a saving of ε^(-4) compared to their bound in (8) in the worst-case scenario. In contrast, my Corollary 2 results in a saving of ε^(-k/2) in this case. There is a substantial distinction between saving ε^k and ε^4. This clearly underscores the advantage of my results. 

The method of moments leads to a system of equations. Given that we employ samples to approximate moments, it is necessary to consider perturbations in the coefficients of the system. Hence, it is unsurprising that there are similarities between my paper and Wu and Yang's work, including the presence of certain lemmas. However, the divergence lies in how we approach and resolve the system and evaluate the impact of perturbations. In my case, I employ Beauzamy's theorem from mathematical analysis to address this. This is a sophisticated theorem, and applying it provides the distinct advantage between our methods. 

Second part of comparison: It is well known that parameter estimation is a more challenging task than returning a distribution that closely matches the mixture in a certain norm.  In their work Wu and Yang return a distribution that is close to the target in Wasserstein distance. In my work, I directly estimate parameters. In Lemma 1, Wu and Yang explore the conditions under which one can transition from the Wasserstein distance to parameter estimation. To elaborate, let's denote the minimum distance between the means as "min," with all weights equal to 1/k. To transfer their bound on the Wasserstein distance to a bound on parameter estimation, they require more than (k/(min*ε))^(4k) samples. Hence, the inverse of the minimum distance to the power of 4k contributes to sample complexity. In contrast, my method only demands the inverse of the minimum distance to rise to the power of the number of consecutive gaps with a minimum distance, which in most cases is much smaller. Therefore they have to use excessive amounts of samples (unnecessary in almost all cases) to get the Wasserstein distance small enough to be able to apply their method in parameter estimation.  

The difference between our methods is particularly significant in practice. Consider situations where the distribution of spacing between means follows a Poisson distribution or is GUE-like in the context of the distribution of eigenvalues of large Hermitian matrices. While small gaps are still present, their probability is exceedingly low. The mere possibility of such cases drastically increases the sample complexity when applying their method, whereas my approach only amplifies the sample complexity by (1/min)^(4k * probability of small consecutive gaps), representing a substantial practical advantage.

Moving on to the third point of comparison, a crucial feature of Hardt and Price's (HP) work is their theorem 3.10, which demonstrates that sample complexity depends on the differences between two means. If the gap between means is not small, the sample complexity remains moderate.  Thus, any extension of their result should uphold this characteristic. I accomplish this by introducing the pair correlation function between the means. In their paper, Yihong Wu and Pengkun Yang also make reference to HP's work: "By carefully analyzing Pearson’s method of moments equations, HP showed that the optimal rate for two-component location-scale mixtures is Θ(n^(-1/12)); however, this approach is difficult to generalize to higher order mixtures." As they acknowledge, generalizing HP's approach is challenging, which is precisely the task I pursued in my paper. Initially, I had no idea how I should generalize the feature. However, the natural application of Beauzamy's theorem addressed this challenge.

In conclusion, our bound only matches in the worst case scenario, in any other cases my bound has a clear advantage. My method estimates parameters directly, which is a harder task. Finally, in my opinion, my paper is making a significant contribution. 

Overall your comments have been very useful to me. By comparing the paper you referenced with my results and providing the explanations outlined above, I believe my paper has been substantially improved.

Reviewer comment (Presentation):
The title of the paper is missing.

Author response:
I am sorry for the problem with the title and the writing. I explained the circumstance in my general response to reviewers.

Reviewer comment (Writing):
The abstract does not summarize the paper well. In particular, the authors should highlight that the sample complexity is established in one-dimensional setting.

Author response:
I will add more to the abstract.

Reviewer comment (Writing):
There are some undefined notations, e.g. $\sigma_i$ in the contribution paragraph.

Author response:
Fixed.

Reviewer comment (Theory):
The corollaries are presented without any theorems introduced before, which makes no sense.

Author response:
Since the theorem is more abstract, I put corollaries that are easier to understand first. I can bring the theorem first then corollaries on the referee's recommendation.

Reviewer comment (Presentation):
In the main text, the reviewer suggests that the authors should only provide a proof sketch in a separate section and then leave the full proofs in the supplementary material. This would help improve the cohesiveness of the paper.

Author response:
I will apply your suggestion.

Reviewer comment (Theory):
The sample complexity of GMMs derived in the paper is limited to the one-dimensional setting. Moreover, the variance of each Gaussian distribution is assumed to be one, which is quite restricted.

Author response:
I already mentioned that d-dimensional easily transfers to d instances of one-dimensional cases. Regarding the equality of the variance, please note that when we consider a scenario where all variances are equal, it represents the most challenging case for our problem. Essentially, when the set of means is fixed, if all variances are equal, we encounter the worst sample complexity. Therefore, the sample complexity we derive serves as an upper bound applicable to a broader context where variances are not necessarily equal. It's also important to mention that Hardt and Price [1] have previously demonstrated that for 2-mixtures, when means are in close proximity, differences in variances would reduce the sample complexity.

My results can be particularly valuable to researchers and practitioners dealing with the parameter estimation of GMMs. Especially when there is some prior information available about the distribution of spacings between means. I have elaborated further on this in my responses to your questions and in the reply to reviewer EBv4.

As for assuming the mean of the mixture is zero, we can always center our data by subtracting the empirical mean from each data point. This adjustment allows us to work in a setting where the overall mean is zero.

I would also like to draw your attention to two recent papers [2,3] that delve into the spherical case, assuming an additional condition of means being separated by (log k). These papers explore even more specific instances of the problem I studied and have been published in respected ML journals. This underscores that this is an active area of research in the theoretical ML. Nevertheless, I acknowledge that one limitation of my work is the assumption of equal variances.

[2] Mingda Qiao. "A Fourier Approach to Mixture Learning." NeurIPS 2022.

[3] Allen Liu and Jerry Li. 2022. "Clustering mixtures with almost optimal separation in polynomial time."  (STOC 2022).

Reviewer comment (Novelty):
When discussing parameter estimation of GMMs, the authors should involve more relevant papers, not only those using the method of moments but also other methods like Maximum Likelihood, namely [1], [2], [3], [4] and [5].

Author response:
Thanks for mentioning these papers. I will cite them. Please note that in response to Reviewer EBv4 I extensively compared my result to the one in reference [4].

Reviewer comment (Experiments):
The paper lacks a simulation study to empirically verify the theoretical results. The reviewer believes that such numerical experiments would significantly strengthen the paper.

Author response:
I wrote Python code to evaluate my algorithm's performance. I generated samples from a mixture of 3 Gaussians (k=3), and the algorithm performed well.  If you mean that I find a data set of a mixture of spherical Gaussians and try the algorithm on it; this is an interesting suggestion, and I will look into it.

Reviewer comment (Presentation):
The authors should include a discussion paragraph at the end to conclude the paper, discuss the limitations of their current technique, and its ability to extend to more general settings of GMMs.

Author response:
Sure.

Reviewer comment (Theory):
In Corollary 0.1, it is unclear if the variance $\sigma^2$ refers to the whole mixture or an individual Gaussian distribution.

Author response:
it's the variance of the whole mixture, I will clarify this.

Reviewer comment (Writing):
In Section 0.1, the authors should define the notation $\Omega(\sigma)$, and not assume that readers will understand it.

Author response:
Sure.

Reviewer comment (Theory):
In Theorem 0.3, it is unclear whether the mean zero and variance one assumptions are for simplicity or are necessary for the proof to hold true.

Author response:
The mean zero is for simplicity, however the condition variance equal one is necessary. In a subsequent paper, I will delve into cases where variances may not necessarily equal one, but it's worth noting that this will involve more complex mathematics.

Reviewer comment (Theory):
What are the main challenges of deriving the sample complexity of GMMs in high-dimensional settings?

Author response:
High dimensional setting is more or less straightforward, there are many tricks developed by researchers to reduce the d-dimensional to 1-dimensional. For example see [1], page 8, section on Dimension Reduction.

Reviewer comment (Theory):
In equation (4.2), it is questioned whether the term $\hat{P}'(x)$ is always different from zero.

Author response:
If it's zero then the theorem is void. Basically says that $|x-y|< \infty$. Beauzam [2] addressed this in comments after the theorem.

Reviewer comment (Theory):
The 'Corollary' should be renamed as 'Proposition'.

Author response:
I will adrees this.

Reviewer comment (Theory):
In equation (0.1), the weights $\omega$ need to be non-negative.

Author response:
Sure.

Reviewer comment (Writing):
The results are described in a very confusing manner all over the paper. To quote: "... assume that we have gaps of length $\varepsilon$ between consecutive means in our mixture. However, these gaps are isolated, meaning that if $\mu_{n+1}-\mu_n=\varepsilon$, then the adjacent gaps are significantly larger:  $\mu_{n+2}-\mu_{n+1}>>1$, and  $\mu_{n}-\mu_{n-1}>>1$". What does this mean? On the one hand, you are saying consecutive gaps are small. But immediately after that you are saying certain gaps are significantly large.

Author response:
$\textbf{Response:}$ The source of confusion here arises from distinguishing between gaps of length $\epsilon$ between consecutive means and consecutive gaps of length $\epsilon$. When I refer to gaps between means, it's important to specify that I am discussing gaps between two consecutive means. For example, in a sequence like $\mu_1< \mu_2< \mu_3< \mu_4<  \mu_5$;  both $\mu_2- \mu_1$   and $\mu_3- \mu_1$   are gaps between means. To clarify, we should first establish that 'gap' refers to the spaces between consecutive means, such as between  $\mu_1$ and $\mu_2$ or $\mu_3$ and $ \mu_4$. Then, we can discuss consecutive gaps.

Consider the example when  $\mu_1=0,  \mu_2= \epsilon, \mu_3=1,  \mu_4=2 $  and $\mu_5= 2+ \epsilon $. In this example, we have two gaps of length $\epsilon$ between $\mu_1$ and $\mu_2$ as well as between $\mu_4$ and $\mu_5$. However, these gaps are not consecutive. If we had $\mu_1=0,  \mu_2=\epsilon, \mu_3=2\epsilon,  \mu_4=1$ and  $ \mu_5= 2$, then we would have two consecutive gaps of length $\epsilon$, one between $\mu_1$ and $\mu_2$ and another between $\mu_2$ and $\mu_3$. A significant portion of the results in my paper is focused on distinguishing between cases like these two. I hope this clarifies things.

Reviewer comment (Presentation):
Now I'll come to the *main weakness* of the paper. The presentation is extremely poor and unprofessional. For example the paper don't even have a title and section title for the starting section. One cannot simply expect reviewers to spend time on a paper where the authors themselves have not spend much time.

Author response:
$\textbf{Response:}$ I apologize for problems regarding the writting and the lack of title. I explained the circumstance in my general response to reviewers.

Reviewer comment (Theory):
For the mean zero case, depends on a new parameter of pair correlation of means, which limits applications of this result.

Author response:
As for assuming the mean of the mixture is zero, we can always center our data by subtracting the empirical mean from each data point. This adjustment allows us to work in a setting where the overall mean is zero.

My results can be particularly valuable to researchers and practitioners dealing with the parameter estimation of GMMs. Especially when there is some prior information available about the distribution of spacings between means. I have elaborated further on this in my responses to your questions and in the reply to reviewer EBv4.

Reviewer comment (Theory):
Besides theoretical interests, may the authors suggest applications of the results, considering their limitations (works only for the spherical case of equal variance, and depends on pair correlation of means)?

Author response:
In response to your question regarding the practical applications, I have addressed a portion of this in my response to referee EBv4. As previously mentioned, when we fix the set of means, the most challenging sample complexity arises when all variances are equal. Therefore, my findings indicate that the sample complexity in a general setting (where variances are not equal, necessarily) is lower than that in the spherical case. In real-world applications, assume we have some knowledge about the distribution of means, even if we do not know their specific locations. Assume we have reason to believe they follow a distribution such as the Spatial Poisson point process. This assumption suggests the existence of small gaps between means. Using alternative methods, a factor of $(1/\text{minimum gap})^k$ contributes to the sample complexity. In contrast, my method introduces a factor of $(1/\text{minimum gap})^E$, where E represents the expectation of consecutive gaps with minimum distance. In practice, E tends to be much smaller than k. Therefore, at minimum, using my approach we know how many samples carry enough information to detect parameters of the mixture, even if the algorithm does not work for the general case.