Paper: Fantastic Generalization Measures are Nowhere to be Found (ICLR-2024)

Reviewer comment (Theory):
Certain points in the introduction were not explained properly in later sections: a) The term "vacuous" was never formally defined. b) The connection between the tightness of the generalization bound (equation 1) and the notion of estimability is not discussed in depth (why are they equivalent?). Although the argument may not be hard, providing it would offer important context for the main theorems.

Author response:
> Certain points in the intro were not explained properly in later sections. a) The term "vacuous" was never formally defined, b) the connection between tightness of generalization bound (eq 1) and the notion of estimability is also not discussed in depth (why are they equivalent? I know the argument is not hard, but this is provides important contexts for the main theorems).

a) A generalization bound for binary classification with 0-1 loss is called vacuous if it exceeds the numerical value 1, which is the trivial upper bound of the population risk. We will add this in a footnote on the first page.

b) Let’s focus on algorithm-independent estimability. The existence of a tight generalization bound implies estimability, directly by Def. 3. In the other direction, estimability (with parameters $\epsilon,\delta$) implies the existence of a $2 \epsilon$- tight generalization bound, with the same probability parameter $\delta$. This is because estimability implies that there exists some estimator $E$, which might for some samples underestimate the risk by $\epsilon$, and for other samples overestimate the risk by $\epsilon$. But then the generalization bound $E + \epsilon$ upper bounds the risk with error at most $2 \epsilon$ with probability at least $1-\delta$. The exact same argument applies also for algorithm-dependent estimability.

Reviewer comment (Theory):
The theorem statements are quite complex, and except for Theorem 2, the authors did not offer discussions that help with parsing the statements.

Author response:
> The theorem statements are pretty mouthful themselves and except for Theorem 2, the authors did not offer discussions that helps with parsing the theorem statements.

The statement Theorem 1 makes essentially the same qualitative statement w.r.t. non-estimability as the less general(\*) Theorem 2, but it applies to all overparametrized settings rather than classes of finite VC dimension. 
Since these Theorems are so similar (Thm. 2 is essentially a Corollary of Thm. 1), we only discuss Thm. 2 in detail. We decided to focus on Theorem 2 since it is stated in terms of VC-dimension, which is a well-known quality for readers familiar with learning theory, whereas our notion of overparametrization is non-standard.

(\*) As we argued in the global discussion concerning overparametrization, finite VC dimension implies overparametrized.

Let us also add some more details to the discussion of Theorem 3:

$H_0$ is to be understood as the class that capture's the learner's "bias". For example, we have an algorithm that performs well on $H_0$, say with accuracy $\epsilon=0.01$ and confidence $\delta=0.01$.
The intuition behind $H_1$ is the following: first note that $H_1$ cannot be arbitrary: If $H_1$ is very small, for example, a set of three functions, $H=H_0 \cup H_1$ would be learnable with comparable parameters $\alpha \approx 0.01$ and $\beta \approx 0.01$. That would render item 2 in the theorem to be useless since $\alpha-\epsilon \approx 0$ would be very small. If $H_0$ is very small, for example, a set of three functions, and $H_1$ would consist of all other functions not in $H_0$, we could easily distinguish between samples from $H_0$ and samples from $H_1$ so the total variation is close to 1 and $\gamma \approx 0$. That would render item 2 in the theorem to be ineffective since $\eta \approx 0$ would be very small.
Finally, an intuitive way to understand Theorem 3 is a scenario where it is possible for an algorithm to learn $H_0$ or $H_1$ separately, but not both (which is consistent with the above statements). Theorem 3 shows that any algorithm that performs well in such a case (learns well one of $H_0$ or $H_1$) would not be estimable over the union $H=H_0 \cup H_1$.

Does this help with parsing Theorem 3?

Regarding Thms. 4 and 5:

Since Theorems 4 and 5 are formulated directly in terms of estimability, we believe that they can be parsed on a technical level without problems once Def. 4 is understood, and we address their implications in subsection 6.1. We are of course happy to answer further questions regarding them (or any of the other Theorems) – perhaps we missed a detail that makes them hard to parse for first-time readers. If so, please let us know what this detail might be.

Nevertheless, a few more remarks reg. Thm. 5:

Since Theorem 4 provides only weak numerical values when applied to parities (a particularly interesting/easy to grasp subset of linear functions), we show non-estimability of (parities functions, linearly biased ERMs) directly, which leads to significant numerical values. This implies that no tight bound (exceeding these numbers $(\epsilon, \delta, n)$) can exist in this setting.
We will incorporate a version of the above explanations in the forthcoming updated manuscript, which should improve the overall presentation.

Reviewer comment (Theory):
The definition of over-parameterization (Definition 2) is not standard and appears to be phrased to make the proofs simpler. While it does intuitively fit the idea of over-parameterization, the author should add: a) detailed discussions on why this definition is consistent with the standard setting in the literature, and b) examples.

Author response:
> The definition of over-parameterizaton (Definition 2) in this paper is not standard and my impression is that they are phrased to make the proofs simpler. While Definition 2 does intuitively fit the idea of over-parameterization, I feel strongly that the author should add: a) detailed discussions on why this definition is consistent with the standard setting in the literature, b) examples.

Please see the global discussion regarding overparametrization.

Reviewer comment (Theory):
In Theorem 3, the condition on TV distance is unmotivated and seems to only exist to make the problem easier.

Author response:
> Similar to the previous point, in Theorem 3, the condition on TV distance is unmotivated and seems to only exist to make the problem easier.

Theorem 3 gives a sufficient condition for non-estimability. Our proof essentially boils down to the fact that samples drawn from two different distributions – with labelings that disagree over a significant fraction of the space – “look the same” or “are easily confusable” for samples of size $n$, with reasonably large probability. If then the algorithm can learn over one of the distributions, but fails to learn over the other one (both of these items are part of the assumptions of the Theorem), any estimator must be loose.
In the above context, the TV distance between these two distributions is a natural measure for the “confusability” of (the samples from) these two distributions.

As mentioned, this technique only yields a sufficient condition, so in theory it might be that it cannot yield meaningful numerical values of non-estimability. This is why we show in Theorem 4 that this is not the case (showcased for linear functions, together with ERMs with linear bias): we get essentially the strongest possible result already for $q=11$: that is, no estimator can have confidence much higher than a random guess, if it is supposed to do better than the trivial error level (see the discussion under Thm. 4).

Reviewer comment (Theory):
In the proof of Theorem 1, the authors did not show the existence of Bayes-like Random ERM.

Author response:
Regarding (6) and Definition 7 (Bayes-like Random ERM), we missed mentioning in our paper that all spaces we consider are finite. With this in mind, the definition of Bayes-like Random ERM  is a well-defined random variable over a finite sample space. Does this answer your query?

Reviewer comment (Theory):
How is over-parameterization used in the argument between Definition 3 and Theorem 1?

Author response:
> I am talking about the paragraph between the statements of Definition 3 and Theorem 1. This is in the middle of page 6.

Thank you for the pointer. After careful re-reading, we polished the paragraph further and added more details, which now hopefully makes it easier to parse:

Note that given an algorithm-independent and distribution-independent bound $C$, it is not hard to construct a \emph{single} (algorithm, distribution) pair that makes it vacuous. To illustrate this, consider the ERM defined as $\mathcal A_C(S):=\arg\min_{h\in \mathcal H: \: L_S(h)=0} C(h,S)$. Assume that $C$ is a valid generalization measure, i.e., (1) holds almost surely for every $\mathcal H$ realizable distribution $\mathcal D$ with labeling function $h_\mathcal D$. Then, by construction, $L_{\mathcal D(\mathcal A_C(S))} < L_S(\mathcal A_C(S)) + C(\mathcal A_C(S),S)\leq C(h_\mathcal D,S)$ almost surely over $S\sim \mathcal D^n$. 

Now assume that the setting is overparametrized (say with large $\alpha$ and $\beta$ for a fixed $n$), i.e., no algorithm can learn (with error $\alpha$) the ground truth labeling w.p. at least $1- \beta$ jointly over the uniform choice of the distributions, and $n$ samples from the chosen distribution. This implies that for every algorithm $\mathcal A$, there exists at least one distribution $\mathcal D’$ such that $\mathcal A$ cannot learn w.p. at least $\beta$ over $S \sim \mathcal D’^n$. Hence there exists a realizable distribution $\mathcal D'$ with deterministic labeling function $h_{\mathcal D'}$ such that w.h.p. $L_{\mathcal D'}(\mathcal A_C(S))> \alpha$ which implies by the above inequality $C(h_{\mathcal D'},S)>\alpha$ w.h.p. But then it holds w.h.p. that $C(h_{\mathcal D'},S)\gg L_{\mathcal D'}(h_{D'}) - L_S(h_{D'})= 0$ and hence the bound is w.h.p. not $\alpha$-tight (in the sense of Eq. 2) for the pair $(\mathcal A_{h_{\mathcal D'}},\mathcal D')$, where $\mathcal A_{h_{\mathcal D'}}$ is the constant algorithm that always outputs $h_{\mathcal D'}$.

In particular, we removed the use of the no-free-lunch theorem and made the connection to the overparametrized setting more direct. Does this clarify things? If not, which step causes confusion?

Reviewer comment (Theory):
While the authors correctly argue that "for a bound to be nonvacuous in the overparametrized regime, it must be dependent on the sampling distribution", they overstate the extent to which this invalidates existing literature. The bounds that are genuinely in the overparametrized setting are rare, with parameter counting bounds ([6]) being a genuine example. Many existing bounds—such as norm-based uniform convergence bounds ([7,12]), margin bounds ([7]), and more modern data-dependent results ([8,9,10,11])—do not fall under the applicability of the theorems presented, as they incorporate some data dependency. The results in [3,4] and bounds using rank or sparsity clearly use the sampling distribution and should be discussed.

Author response:
Thank you for your detailed feedback, which includes much appreciated comments regarding the fine details in the appendix.

Before addressing your questions in detail, we would like to first clarify a few high-level points.


It seems to us that some of your questions stem from a disconnect of our definition of ‘distribution-dependent’ and the one you have in mind. For example in your third point, you mention:
> There are plenty of papers that actively take the sampling distribution into account in their analysis and are not even cited here. The most obvious examples would be [3] and the NTK literature [4].

However, the bounds in these works only take into account the training set and the output hypothesis, but not the properties of the distributions themselves. 
By ‘distribution dependent’ bound we mean (throughout the paper) that the bound takes into account some property of the underlying data distribution (e.g., boundedness, information about the moments, smoothness…). However, note that receiving a training sample does not qualify a bound for being distribution dependent.

Relatedly, you are asking:
> In the current version of the paper, on page 13, margin bounds are presented as part of examples of bounds that are "distribution independent", which is quite disputable. 

All bounds we mention in Appendix B do not assume anything about the algorithm being considered or the underlying distribution. Hence they fall within the setting of Theorem 1. If you disagree, can you lay out your argument based on one paper from that list? For example, why is this disputable over the margin-based bounds? They are just a function of the training set and the output hypothesis.

Does this clarify some of your questions regarding distribution dependence?

Further, you are asking:
> Which bounds in the existing literature are really "in the overparametrized setting"?

We are not sure what you mean when you call a bound being in the overparametrized setting – overparametrization is a quantifier of a learning task, not of a generalization bound. 

Since ‘overparametrized’ means (roughly speaking) not having enough training samples to learn over all distributions, and none of these bounds specifically exclude this regime, all our results apply to all of the distribution-independent bounds that we cite, once one is situated in this learning setting. 

Does this answer your question? If not, could you please expand on this point?

Reviewer comment (Theory):
In the current version of the paper, page 13, margin bounds are presented as examples of "distribution independent" bounds, which is disputable. Similarly, the paper offers a blanket dismissal of so-called "Rademacher bounds", despite the fact that many actively distribution-dependent bounds such as [3,4] rely on Rademacher arguments in parts of their proofs. This stance risks unfairly dismissing legitimate research. The tone of the main paper and appendices B, C, and D should be moderated.

Author response:
## **Margin Bounds**

You raise the question of whether our negative results in Theorems 1 and 2 apply to margin bounds. We assert that they do indeed apply. We will explain this both via a concrete example, and then in a more general discussion.

### Example: A Margin Bound That Is Not Uniformly Tight

To follow up on the specific example you mentioned in [your comment](https://openreview.net/forum?id=NkmJotfL42&noteId=92pbVP25Si) above, let the domain $\mathcal{X} = \\{x \in \mathbb{R}^d: ~ \\|x\\|_2 = 1\\}$ be the unit sphere in $\mathbb{R}^d$. Consider a half-space classifier, and a margin bound of the form $f(|W|, \gamma)$ as you described, such that $f$ (an upper bound on the difference between the population and empirical loss) is small only if the margin $\gamma$ is large. We will show that $f$ is not uniformly tight. 

Indeed, consider a population distribution $\mathcal{D}$ that has (say) a uniform marginal on the domain $\mathcal{X}$, with labels that are generated by a specific half space $h^*$ (e.g., the half space that passes through the origin and is perpendicular to the $x_0$ axis). Consider an algorithm $A$ the has a bias towards $h^*$, in the sense that it will output $h^*$ whenever the training set is consistent with $h^*$. 

In this case, given samples from $\mathcal{D}$, with probability $1$, $A$ will output $h^*$. Hence, the population loss and the empirical loss are the same (they are both exactly 0). Because the marginal of the population distribution on the domain is uniform, the population margin is 0, and therefore the empirical margin $\gamma$ will be close to $0$ (even for a training set that is small compared to the dimension $d$). Thus, $f(|W|, \gamma)$ is large, but the generalization gap is 0. Hence, the bound $f(|W|, \gamma)$ is not tight for the pair $(\mathcal{D}, A)$.

Note that our results in the paper are stronger, in the sense that they show that there exist many (distribution, algorithm) pairs for which the bound is not tight, not just one.


### General Case: Why Margin Bounds Are Not Uniformly Tight

More generally, in margin bounds the difference between the empirical and population loss is bounded by a quantity $f = f(|W|, \gamma)$. This quantity $f$ is strictly a function of the selected hypothesis and of the training sample. As such, a bound of this kind satisfies the algorithm- and distribution-independence requirements of Theorem 1. If the bound is applied to a neural network that has a lot more edges than the size of the training set, then this is an overparamterized setting (as we explain in detail in _**General Answer 1: Our Definition of Overparametrization**_), and therefore Theorem 1 implies that the bound $f$ will not be uniformly tight. This is a dry logical consequence of Theorem 1. 

We agree with the observation that the quantities $|W|$ and $\gamma$ depend in nontrivial ways on the population distribution and on the training algorithm, and that therefore the quantity $f(|W|, \gamma)$ depends indirectly on the on the population distribution and on the training algorithm. This is not a problem, because this is always true: a bound that depends on the selected hypothesis and on the training sample must always depend indirectly on the training algorithm and on the population distribution. That does not detract from the applicability of Theorem 1.

Reviewer comment (Writing):
I believe the notion of overparametrised setting should be explained more carefully and clarified. Indeed, in the introduction, authors precise that such a setting 'roughly means that the number of parameters in the networks is much larger than the number of examples in the training set' while Definition 2 does not even make intervene the notion of parametrised space. The link with neural nets should be explicited.

Author response:
> I believe the notion of overparametrised setting should be explained more carefully and clarified. 

Please see our general discussion **General Answer 1: Our Definition of Overparametrization** on the top of the comment section.

Reviewer comment (Writing):
page 6 'analouge of Theorem 1' -> analogue

Author response:
> page 6 'analouge of Theorem 1' -> analogue

Thanks! corrected.

Reviewer comment (Theory):
About the implications of Theorems 4 & 5, can you be more specific about the links with practical neural nets? I read Appendix D and I am wondering about the recommendation given at the end of the section: how is it possible for theoreticians to either 'exclude many distributions over linear functions' if one doesn't know which one to focus on? Second, how realistic is it, in terms of computational time, to verify that 'the neural network architecture with SGD cannot learn any large linear subspace of functions'?

Author response:
> About the implications of Theorems 4 & 5, can you be more specific about the links with practical neural nets? 

What we wrote in Appendix D is just the immediate **logical** implication of our work from Theorem 4: 

* Exclude in advance many distributions over linear functions.

* Show that the neural network architecture with SGD cannot learn any large linear subspace
of functions.

If either one is true, then the bound that was derived is necessarily vacuous in some scenarios for neural networks, even if the bound was specifically tailored for SGD with a specific architecture.

> How is it possible for theoreticians to either 'exclude many distributions over linear functions' if one doesn't know which one to focus on?

This is exactly our point. If one is willing to take a more **intuitive** leap or implication  (not a **logical** one), we argue and advocate that a more bottom-up approach is preferred. We think it is better to study very specific combinations of distributions with specific classes of algorithms while keeping in mind that no bound can work for all distributions. Essentially, let go of generality (distribution-independent bound) and aim towards tighter analysis of specific cases.
 
Also, perhaps it's better first to uncover and study properties of distributions (such as natural images, natural language, etc.) and only then try and derive bounds tailored for distributions with said properties.

> Second, how realistic is it, in terms of computational time, to verify that 'the neural network architecture with SGD cannot learn any large linear subspace of functions'?

Probably, one cannot verify this empirically, maybe only provide a mathematical proof. Since this is so hard, it again softly implies the abovementioned direction.

Reviewer comment (Theory):
From a broader perspective, I believe that some claims in the paper are more a matter of personal interpretation than scientific facts and should be reformulated. For instance, author claims in section 5.1 that 'Most published generalization bounds do not restrict the set of distributions or algorithms the bound should apply for. Hence, they should work in all scenarios.' This is not true, having a bound holding for any distribution does not imply that the bound will be tight on all situations (and this is precisely what your work is proving). On the contrary, there may be several interests to derive a generalisation bound, a practical one is deriving generalisation-driven learning algorithms who may have a practical performance tighter than the associated bound as in Dziugaite & Roy 2017, or again simply propose a certain measure of complexity as in Neyshabur et al. 2017 to check if, in a few concrete learning problems, this complexity measure is tailored to explain generalisation. To me, the sentence at the end of Appendix D: 'At the very least, for a generalization bound for a neural network architecture trained with SGD to be meaningful, it must satisfy either of the following items...' is misleading as you assimilate the notion of 'meaningful' to 'precisely explaining the tightness on a given situation'. From my understanding of your work, I would say that existing generalisation bounds are too generic to provide such a precise understanding and are only an intermediary step which has to be completed by a study of the intrications between the data distribution and a learning problem.

Author response:
> For instance, author claims in section 5.1 that 'Most published generalization bounds do not restrict the set of distributions or algorithms the bound should apply for. Hence, they should work in all scenarios.' This is not true, having a bound holding for any distribution does not imply that the bound will be tight on all situations (and this is precisely what your work is proving). 

We think the confusion stems from ‘should work’ which you interpret as ’should be tight’. Is that correct? 
We meant ‘should work’ = ‘should hold’. So we suggest changing our text as follows:

Most published generalization bounds do not restrict the set of distributions or algorithms the bound should apply for. Hence, they should upper bound the risk in all scenarios. Because these bounds do not distinguish between distributions that an algorithm performs well on and distributions that it does not, such bounds have no other option but to declare the largest true error possible (and become vacuous).

Does this clear things up in this example? 


> On the contrary, there may be several interests to derive a generalisation bound, a practical one is deriving generalisation-driven learning algorithms who may have a practical performance tighter than the associated bound as in Dziugaite & Roy 2017, or again simply propose a certain measure of complexity as in Neyshabur et al. 2017 to check if, in a few concrete learning problems, this complexity measure is tailored to explain generalisation. 

We agree with your perspective! This fits with the implications of Theorem 4 we discussed above!



> To me, the sentence at the end of Appendix D: 'At the very least, for a generalization bound for a neural network architecture trained with SGD to be meaningful, it must satisfy either of the following items...' is misleading as you assimilate the notion of 'meaningful' to 'precisely explaining the tightness on a given situation'. 

We agree with your perspective. Suggested replacement: 

"A generalization bound for a neural network architecture trained with SGD can be at most an intermediary step to our understanding of neural network generalization if it doesn't satisfy one of the following items: ...".

Reviewer comment (Theory):
The analysis of the bound of Dziugaite & Roy (2017) in section 2.3 would gain to be reworked. Indeed, the sentence 'the bound in Dziugaite & Roy (2017) relies on implicit assumptions about the algorithm and the population distribution' is simply not true as it relies on the McAllester's bound which does assume anything on the data distribution. However, the efficiency of their algorithm on MNIST is effectively not due to the PAC-Bayesian bound but to more subtle assumptions satisfied by the specific learning problem of interest and is not easily extendable to other learning problems.

Author response:
> Similarly, the analysis of the bound of Dziugaite & Roy (2017) in section 2.3 would gain to be reworked. Indeed, the sentence 'the bound in Dziugaite & Roy (2017) relies on implicit assumptions about the algorithm and the population distribution' is simply not true as it relies on the McAllester's bound which does assume anything on the data distribution. 

Here we disagree :) The fact that the McAllester bound does not assume anything on the data distribution in its derivation, does not mean one cannot make it distribution-dependent and algorithm-dependent. This is true since if you have some prior knowledge of your algorithm and the underlying distribution, you can use this prior knowledge in the prior $P$ in the term $KL(Q||P)$ in the bound. We believe that Dziugaite & Roy (2017)  achieved a non-vacuous bound for MNIST because for the prior they chose, they implicitly assumed that for the encountered distribution, the algorithm finds a flat minima not far from the initialization.

> However, the efficiency of their algorithm on MNIST is effectively not due to the PAC-Bayesian bound but to more subtle assumptions satisfied by the specific learning problem of interest and is not easily extendable to other learning problems. 

We think that what you write here is consistent with our paragraph above. It is indeed not easily extendable to other learning problems. For other networks and datasets, different priors might prove more beneficial (an example of distribution-dependent).

Reviewer comment (Theory):
The main issue I found in the paper is that in many places the discussions are not precise. As an example the claim about cross-validation can be misleading: The authors claim that cross-validation approaches do not lead to algorithm design principles. It is not completely correct. For instance, the well-known algorithm of the one-inclusion graph [Haussler et al 1988] is based on the cross-validation analysis.

Author response:
Thank you for the reference [Haussler et al 1988]! We were not familiar with it. Can you refer us to the location of your claim about cross-validation? After searching the paper, the word ‘validation’ does not appear there. Regardless, your point makes sense. To avoid being imprecise, we will change the term cross-validation (which is indeed an umbrella term for several methods) to using a validation set (the holdout method - the simplest form of cross-validation), which is the staple for tuning the hyperparameters of neural networks. 

Do you find the above now a precise statement?

Reviewer comment (Theory):
Definition of tightness can be improved: In many cases, we are interested in correlation of complexity measure and the actual generalization gap. It is not clear to me that the definition of tightness is the best possible.

Author response:
Interesting comment and question! 

But what is the right correlation measure to use? There are many possible choices, and it is likely there is no consensus on which are the most beneficial.  Since the tightness of generalization bounds is a well-defined mathematical question it makes sense to us to focus on that at this point. Do you agree?

We lay it as a conjecture that if such a good correlation measure exists, then there exists a good generalization measure. Intuitively this might hold because if one can prove that a correlation measure is really good then we could artificially adjust it to be a generalization bound that works well (albeit without proof because maybe the analysis is too hard). Then, since our work shows that a good generalization measure does not exist, a good correlation measure should not exist as well. So we also conjecture that good correlation measures should not exist as well.

Reviewer comment (Presentation):
In Page 14, there is this paragraph which clearly is not about Negrea et al. (2020). This paragraph is unrelated to this paper.

“Negrea et al. (2020): The paper studies convex optimization, so the results can hold only for a single neuron. Nevertheless, although it gives matching lower and upper bounds, the bounds match only asymptotically when n is very large so the scenario is far away from the overparametrized regime (which is the focus of interest for neural networks).”

Author response:
True, thanks for noticing this! We intended to cite "Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization".