Paper: Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem (ICLR-2024)

Reviewer comment (Theory):
The result looks not an extension of the Gauss-Markov theorem since it only holds under expectation over $X$ when $X$ is an Gaussian ensemble or a diagonal ensemble. The Gauss-Markov theorem works for any $X$.

Author response:
We apologize for the lack of clarity here. What we refer to as the Extended Gauss-Markov Theorem is actually Theorem 2 (and we have now clearly labeled this in the text in the statement of the theorem). You are correct that the results about the test error in sections 2.2.1 and 2.2.2 only hold under specific distributional hypotheses. In contrast, Theorem 2 does not make any distributional assumptions, or indeed have any probabalistic component whatsoever- it is purely a statement about an optimization problem which holds for basically arbitrary X (technically X must be full rank). However, if we assume that errors epsilon_i are uncorrelated, zero mean, and have finite variance, then we have the probabilistic interpretation that LL^T is the covariance matrix and LX-I is the bias operator. So while this interpretation does impose some distributional assumptions, it is not any more than found in the classical Gauss-Markov theorem (Note that, in the original submission  we required Gaussian error distributions, but as a consequence of this discussion have realized this is not actually necessary).

Reviewer comment (Experiments):
It looks more interesting to compare your experiment results with other norms (outside the class of Schatten norms) such as between the nuclear norm and Lasso (norm-$1$).

Author response:
We have done so and included results in the appendix (section A.10). The summary is that the spectral and nuclear models are competitive with lasso even when the ground truth data has sparse structure.

It is also worth noting here that the Lasso is not manifestly a “Linear estimator” in the sense we define it in Section 2, even allowing for the use of norms outside the Schatten family. As a reminder, in order to qualify as a Linear estimator in our sense, the coefficients must take the form LY, where L depends only on X. It is not at all obvious that this is the case for Lasso, although we do not have a proof to the contrary.

*Edit:*Actually,it is easy to prove this about Lasso. Indeed, in the case of $d=1$ (i.e. one predictor variable),the Lasso with regularization $\alpha$ has the well-known solution $w=sign(\langle X,Y\rangle)max(\langle X,Y\rangle/|X|^2-\alpha,0)$,which is clearly not a linear function of $Y$.

Reviewer comment (Writing):
Too many typos. Please check and correct them.

Author response:
We have corrected a large number of small typos, see the general response to reviewers as well as the individual responses to the other reviewers.

Reviewer comment (Experiments):
How do your results in comparison with other norms which don't belong to Schatten class of norms such as Lasso (norm-$1$)?

Author response:
See above

Reviewer comment (Writing):
$L \in \mathbb{R}^{k \times N} \rightarrow L \in \mathbb{R}^{d \times N}$

Author response:
Corrected

Reviewer comment (Theory):
p.2, line 21 from the top: $\mbox{var}\_{\epsilon}=L^T L$ should be changed to $\mbox{var}\_{\epsilon}=\sigma^2 L L^T$. To keep the later, you may change the definition of $\hat{\beta}$ to $L^T(X)Y$ throughout your paper.

Author response:
We have corrected the variance from L^TL to LL^T in both the main text and supplement. As far as the presence of sigma^2, we have in fact assumed the noise variance sigma^2=1  (see second paragraph of section 2.1), which is why we omitted it from the expression.

Reviewer comment (Theory):
In the definition 1, you aim to minimize the variance subject to a constraint on bias by $C$. But, in the later (Theorem 2, Figure 1, etc.), it seems to me that you don't mention $C$ again, but only mention $\alpha$. At least you should mention what is $\alpha$ as a function of $C$ or vice versa in Theorem 2.

Author response:
We have included an explicit description of the relationship between alpha and C in the supplement (A.8), see also our response to reviewer ACbc.

Reviewer comment (Presentation):
Right below Figure 2: Figure 3.1 $\rightarrow$ Figure 3.

Author response:
Corrected

Reviewer comment (Presentation):
Similarly, Figure 3.2.1 $\rightarrow$ Figure 5.

Author response:
Corrected

Reviewer comment (Writing):
Clarity and overall quality of presentation. The paper does not appear to be carefully edited and revised, with multiple typographical errors in the first paragraph of the introduction alone (examples: "somewht" in line 3, lack of period at end of sentence in line 4, reverted quotation marks on line 5, etc.). The graphs, equations, and tables in latter parts of the paper can also benefit from detailed revisions. These issues surrounding clarity and presentation are not constrained to the first paragraph and can be found throughout the paper and also the supplementary material.

Author response:
Thank you for pointing this out. We have corrected the specific typos mentioned here, as well as a large number of other small typos, see the general response to reviewers as well as the individual responses to the other reviewers.

Reviewer comment (Theory):
Discussion of main theorem. While the problem setting itself is interesting, the discussions of the main theorem (Theorem 2) leave an impression that it can be further discussed. For example, what are the values of alpha? Although it is shown in the supplementary that alpha is a consequence of solving Equation 1 using Lagrange multipliers, it is also unclear how large (or small) the value of alpha is and how it impacts the interpretations of the main result.

Author response:
We have added a discussion of the explicit relation between alpha and C in the appendix (section A.8), with a pointer following the main theorem.

Reviewer comment (Theory):
Considering that one of the paper's main claims is a "flat minima" phenomenon, the paper would benefit from stronger theoretical results (apart from simulation-based arguments) surrounding this claim.

Author response:
We appreciate this concern. While the review period is unfortunately likely too short for us to develop substantial new theoretical results, we can at least make an interesting theoretical observation about our results which bears on the "flat minima" phenomenon, and  was not mentioned in the main paper.  Namely, starting from our integral-based expressions for the test error (Prop. 2.1 and 2.4), we can obtain explicit expressions for the curvature of the error curve simply by differentiating under the integral. The resulting analytic expressions are complicated and not particularly illuminating, which is why we did not include them in the paper, and opted instead to explore the properties of the minima through simulations.

Reviewer comment (Theory):
Possible typo in main theorem. In the main theorem's statement for the nuclear norm, the result relies on $\max(\Sigma,\alpha)$: should $\Sigma$ be replaced with something like the maximum eigenvalue?

Author response:
The result is correct as stated, but we see how the notation could potentially be confusing here. We have clarified that the max(...) refers to an elementwise maximum of the singular values. (Note that we have also changed the notation from \Sigma to \sigma here).

Reviewer comment (Novelty):
I cannot assess novelty reliably because my expertise is mainly in optimisation rather than estimation (however, I understand all parts of the main paper well). In fact, rather than extending Gauss-Markov theorem, the paper generalizes ridge regression (please, consider changing the title accordingly). It is well-known that ridge regression has non-zero bias but a smaller variance than pseudoinverse.

Author response:
We appreciate and understand the suggestion to change the title. However, we believe that the characterization of the result in terms of generalizing the GM theorem is accurate and that changing the title would come at the expense of clarity. As we explain in Section 2: “The classic Gauss-Markov theorem tells us that if we want to impose exactly zero bias, then the minimal variance estimator is the ordinary least squares estimator. But what if we want to relax this, to allow non-zero but bounded amount of bias?”. From a presentational standpoint, moreover, we believe that the emphasis of the GM theorem viewpoint is clearer than a framing in terms of generalizing Ridge regression, for the reason that there are many potential ways one could think to generalize Ridge regression, while the GM theorem provides a very specific and principled avenue of generalization.   

As per your observation that ridge regression has non-zero bias but smaller variance than OLS, this is certainly true and widely known. But our Theorem 2 in fact implies a significantly stronger (and, we think, more interesting) statement, namely that ridge regression attains the best possible tradeoff between bias and variance, assuming that “bias” is being measured using the Frobenius norm of LX-I (cf. middle panel of figure 1). This particular fact, while easy to prove on its own, does not appear to be well known (at least, we could not find a result to exactly this effect in the literature).

Reviewer comment (Novelty):
The main novelty seems to be that of flatter minima of test error, rather than the generalization of ridge regression.

Author response:
We appreciate and understand the suggestion to change the title. However, we believe that the characterization of the result in terms of generalizing the GM theorem is accurate and that changing the title would come at the expense of clarity. As we explain in Section 2: “The classic Gauss-Markov theorem tells us that if we want to impose exactly zero bias, then the minimal variance estimator is the ordinary least squares estimator. But what if we want to relax this, to allow non-zero but bounded amount of bias?”. From a presentational standpoint, moreover, we believe that the emphasis of the GM theorem viewpoint is clearer than a framing in terms of generalizing Ridge regression, for the reason that there are many potential ways one could think to generalize Ridge regression, while the GM theorem provides a very specific and principled avenue of generalization.   

As per your observation that ridge regression has non-zero bias but smaller variance than OLS, this is certainly true and widely known. But our Theorem 2 in fact implies a significantly stronger (and, we think, more interesting) statement, namely that ridge regression attains the best possible tradeoff between bias and variance, assuming that “bias” is being measured using the Frobenius norm of LX-I (cf. middle panel of figure 1). This particular fact, while easy to prove on its own, does not appear to be well known (at least, we could not find a result to exactly this effect in the literature).

Reviewer comment (Experiments):
The experiments show that in estimating $\alpha$ by x-correlation, the nuclear norm most often wins, however, this might be due to the experimental methodology. For example, if there were more than 9 values of $\alpha$, the deeper minima of the ridge regression might have been hit much more often.

Author response:
You are certainly correct in observing that the number of alpha values will influence the relative performance of the models in cross-validation. We acknowledged this point, both in section 2.2.3, and in the Discussion: “It should be noted that the particulars of the cross-validation results depend on the number of α values used in the grid search, among other factors; and as noted before, the effect of the curvature at the minimum decreases as more values of α are used.”  That said, we have also included additional results in the Appendix (section A.9) showing the effect of adding more values of alpha. Not surprisingly, the Ridge model often benefits from extra values of alpha in these experiments.

Reviewer comment (Experiments):
A major weakness is that the experiments are done only on synthetic data. The applications of linear regression are abundant, so it should be possible to find many suitable real datasets for this.

Author response:
We appreciate this concern. When writing the paper, we considered including such experiments, but opted not to for the following reason. Our rationale in designing the experiments was to (1) understand the factors that can influence the performance of the three considered models, and (2) to illustrate the practical consequences of the “flat minima” phenomenon vis a vis cross-validated accuracy. We thus opted for exclusively synthetic data because it is much easier to control the underlying properties of such data compared with real datasets (as per (1)), and such data are already sufficient to illustrate (2). 

That said, there is no issue in principle or in practice with applying either the Spectral or Nuclear regressions to real datasets. In order to address your concern, we have done exactly this, and included the results in the Appendix (section A.11). To summarize, we consider the well-known Diabetes dataset and California housing dataset, both available from sklearn. For each dataset, we generated random splits with a training set size of 300, and the remaining samples used for testing. Otherwise, we followed the same methodology as in the synthetic Gaussian simulations. We find results that are qualitatively similar to the table5 from the Gaussian data. Namely, the Nuclear has a small advantage in terms of average MSE, and a fairly substantial one in terms of the probability of winning on a given split.

Reviewer comment (Writing):
In the 1st formula in section 2.1, symbol $L(X)$ is used here but then never more. Change to $L$.

Author response:
Corrected

Reviewer comment (Writing):
In Theorem 2, the letter $\Sigma$ is usually used to denote the diagonal matrix with singular values. For vector of singular values, better use $\sigma$ or $s$.

Author response:
Changed to \sigma

Reviewer comment (Writing):
There are many small mistakes/typos in the text. For example, references to figures in sections 2.2.4 and 3 refer to non-existent figures (e.g., figure 2.2.4 in section 2.2.4).

Author response:
Corrected. We have also corrected a large number of other small typos, see the general response to reviewers as well as the individual responses to the other reviewers.

Reviewer comment (Writing):
In the 5th line of section 2.2.1, can the formula for MSE be simplified (i.e., calculate the mean value in closed form)? It is confusing that the MSE has quite a different form in sections 2.2.1 and 2.2.2, which lets the reader wonder if this difference is substantial or just due to little care for text clarity. This might deserve a comment.

Author response:
Thank you for pointing this out. We have clarified this point in the text- we note that the expression for MSE can be algebraically simplified, but that we have defined it as we did in order to make clear that it is an average over the test distribution. We have also taken the opportunity to further explain why there is a difference between the MSE expressions between the spherical Gaussian and diagonal cases in section 2.2.2. We note there that the difference arises from the fact that the test observations (i.e. rows of Xtest) are not independent in the diagonal case.

Reviewer comment (Presentation):
Most displayed equations are unnumbered, which is not friendly for reviewers (given that lines are not numbered in the ICLR style).

Author response:
We have inserted numbers in all displayed equations in both the main text and supplement

Reviewer comment (Writing):
The asterisk symbol in Proposition 2.1 and in the first displayed formula in section 2.2.2 has not been defined.

Author response:
This symbol denotes the multiplication of two numbers- however we have removed it for the sake of clarity

Reviewer comment (Theory):
The integrals for $Err(\alpha)$ in sections 2.2.1 and 2.2.2 are almost the same, up to the integrating measures. It is unclear if they are correct or if there are typos in them.

Author response:
You are correct that these two integrals are nearly the same. The basic reason for this is that both formulas are derived in a similar fashion of exploiting rotational symmetry of the matrix ensemble to reduce the MSE to a sum over the spectrum of Xtr. Besides the integrating measures, the only difference is that the integral for the Diagonal case has an extra factor of x, which intuitively corresponds to the fact that errors in estimating the coefficient for a direction of high variance contributes more to the overall error than for a direction of low variance. We have clarified this point in the main text.