Paper: If there is no underfitting, there is no Cold Posterior Effect (ICLR-2024)

Reviewer comment (Theory):
I find the claims in this work somewhat misleading. The main contribution is the fact that if the posterior is not underfitting, then there is no cold posterior effect. But this statement is almost trivial as (almost) all that is to prove is already assumed in the statement. Underfitting, as defined by the authors, means “a model having training and testing losses much higher than they could be i.e. there exists another model in the model class having simultaneously lower training and testing losses.” Taking the opposite, no underfitting basically means that there is no model with better train and test loss, hence if the Bayesian posterior is not underfitting then of course there is no cold posterior effect. Am I missing something here? The authors then show that posteriors with smaller temperature having smaller training loss, which is not surprising since the effect of the prior is essentially down-weighted and the model concentrates more on the likelihood, which exactly guarantees a better fit. This, not very surprising fact does all the heavy-lifting here: If one now assumes CPE, then yes, there is a model with smaller temperature and better test loss, which also has smaller training loss but again, not much has been shown here. I also find all the mathematical notation more confusing than helpful here, it makes this relatively simple fact seem more complicated than it really is. I think my issues are somewhat addressed by Theorem 4 which seems to more strongly show the claims made **before** presenting it. I would really appreciate if the authors could clear up my confusion as to what is assumed when words like “underfitting” are used, and what is proved rigorously.

Author response:
Thanks for your feedback. Let us try to clarify the confusion. 

> Taking the opposite, no underfitting basically means that there is no model with better train and test loss, hence if the Bayesian posterior is not underfitting then of course there is no cold posterior effect. Am I missing something here?



This kind of reasoning can be tricky, we will try to explain here why “no underfitting implies no CPE” cannot be directly proved by simply inverting the definition (without using Theorem 2).

It’s correct that no underfitting means there is no model with **both** better training **and** better testing loss **simultaneously**. Hence, the following must hold if there is no underfitting:
1. If another model has better train loss, it must have worse test loss.
2. If another model has better test loss, it must have worse train loss.

The situation, where increasing $\lambda>1$ gives a better testing loss (there is CPE), itself contradicts neither situation 1) nor 2). It is because it could be the case where the tempered posterior has worse train loss (situation 2). Therefore, no underfitting and CPE could co-exist. 

However, situation 2) cannot happen as Theorem 2 shows that the train loss is also enhanced by increasing $\lambda>1$. If you do not use Theorem 2 and try to show the implication from the definition, you do not reach any contradiction.

Reviewer comment (Theory):
I think one important theoretical work in the literature regarding the origins of CPE in conjunction with data augmentation has not been treated in this work. While you cite [1] for re-scaling the prior, you don’t discuss their results regarding data augmentation and how the correlations of errors, arising from assigning the same label to augmented samples, influences the resulting posterior. It is not obvious to me at first glance how those results can be casted within the under-fitting hypothesis of this work, especially the correlated nature of the data. Could you elaborate on the connection to your work?

Author response:
> I think one important theoretical work in the literature regarding the origins of CPE in conjunction with data augmentation has not been treated in this work. While you cite [1] for re-scaling the prior, you don’t discuss their results regarding data augmentation and how the correlations of errors, arising from assigning the same label to augmented samples, influences the resulting posterior. It is not obvious to me at first glance how those results can be casted within the under-fitting hypothesis of this work, especially the correlated nature of the data. Could you elaborate on the connection to your work?

We also thank the reviewer for bringing the connection to this paper up. We have given new thoughts to this work and we have realized our work aligns with the findings of [1] in relation to the degree of model invariances with respect to the data augmentation. For example, if the model is completely invariant to the data augmentation, we would have that $\hat{L}_{DA}(D,\theta)=\hat{L}(D,\theta)$.. And, in consequence, $p^{\lambda=1}_DA=p^{\lambda=1}$  and the inequality of Eq. (10) will be an equality. I.e., DA would not cause a stronger CPE. If there is no CPE under the standard Bayesian posterior (i.e, $\lambda=1$ is optimal), using DA over a completely invariant model would not induce any CPE (i.e., $\lambda=1$  would be optimal as well). We plan to elaborate on this and include this connection in the final version of the paper.

Reviewer comment (Theory):
In Proposition 3, how does the underlying data distribution $\nu$ enter the picture here? The CPE is clearly a function of the data distribution $\nu$ as it talks about generalisation loss, but the inequality seems to be independent of $\nu$ except for involving training samples that are drawn from it. Could you elaborate how you are able to make this connection?

Author response:
> In Proposition 3, how does the underlying data distribution enter the picture here?

Proposition 3 is a consequence of Proposition 6 in Appendix A.3 and the definition of CPE. To shed some light here, Proposition 6 says essentially that the gradient of the Bayes loss has an equivalent form as the difference between the Gibbs loss of the updated posterior (Eq. (6)) and the tempered posterior; where the “data-generating distribution” appears in the updated posterior. Next, the definition of CPE says the gradient of the Bayes loss is negative $< 0$, i.e, the Gibbs loss of the updated posterior is lower than the Gibbs loss of the tempered posterior. Hence, the Gibbs loss of the posterior cannot reach the minimum, which is the message of Proposition 3. Therefore, we don’t see the data-generating distribution in the conclusion of Proposition 3 since the related quantity is thrown away in some sense in the middle of derivation.

Reviewer comment (Writing):
The argument is a bit unclear from my perspective. The authors argue that the problem is under-fitting which comes from misspecification. So is the problem the under-fitting or the misspecification which causes the under-fitting.

Author response:
Thanks for your feedback. Let us try to clarify the confusion. 

> The argument is a bit unclear from my perspective. The authors argue that the problem is under-fitting which comes from misspecification. So is the problem the under-fitting or the misspecification which causes the under-fitting

> The paper seems to be stepping on previous results and works, claiming that the misspecification of the prior or the likelihood lead to underfitting, and therefore underfitting is the problem that causes CPE. Well if CPE is present when under-fitting is present then the problem falls on either the prior or the likelihood.

> My main question is what is the argument that the authors are making? Reading the paper it looks like you are arguing that misspecification is causing the CPE by causing underfitting. It is well known that any misspecification in Bayesian setups leads to lower performance.

> I think I have understood the math and the experiments of the paper but the main argument does not seem very novel. Can the authors better explain?

To start, we would like to clarify that the main takeaway message of our work is the following: **The presence of CPE implies that our Bayesian setup is misspecified and underfitting at the same time.**

This is a simple, clean but actually **novel** way to interpret CPE.
1. Connections to previous work: The connection with underfitting has not been made before when trying to understand CPE. Most of the current literature focuses on the misspecification problem itself, but **not on misspecification in the context of underfitting**. We find it quite important that the ML community is fully aware of this point.
2. “it is well known that any misspecification in Bayesian setups leads to lower performance.”: We totally agree. But this “lower performance” can be due to **underfitting or overfitting**, i.e., you can have a misspecified model that is severely overfitting your training data, as we show in Figure 1 (b). Understanding which of the two is present when we observe CPE is quite relevant. Reviewer wHkC states that this question “ hold[s] profound implications.” 

Having said that, we agree that we could do better in characterizing the relationship between misspecification and underfitting and will do so in the final version.

Reviewer comment (Novelty):
The paper seems to be stepping on previous results and works, claiming that the misspecification of the prior or the likelihood lead to underfitting, and therefore underfitting is the problem that causes CPE. Well if CPE is present when under-fitting is present then the problem falls on either the prior or the likelihood.

Author response:
Thanks for your feedback. Let us try to clarify the confusion. 

> The argument is a bit unclear from my perspective. The authors argue that the problem is under-fitting which comes from misspecification. So is the problem the under-fitting or the misspecification which causes the under-fitting

> The paper seems to be stepping on previous results and works, claiming that the misspecification of the prior or the likelihood lead to underfitting, and therefore underfitting is the problem that causes CPE. Well if CPE is present when under-fitting is present then the problem falls on either the prior or the likelihood.

> My main question is what is the argument that the authors are making? Reading the paper it looks like you are arguing that misspecification is causing the CPE by causing underfitting. It is well known that any misspecification in Bayesian setups leads to lower performance.

> I think I have understood the math and the experiments of the paper but the main argument does not seem very novel. Can the authors better explain?

To start, we would like to clarify that the main takeaway message of our work is the following: **The presence of CPE implies that our Bayesian setup is misspecified and underfitting at the same time.**

This is a simple, clean but actually **novel** way to interpret CPE.
1. Connections to previous work: The connection with underfitting has not been made before when trying to understand CPE. Most of the current literature focuses on the misspecification problem itself, but **not on misspecification in the context of underfitting**. We find it quite important that the ML community is fully aware of this point.
2. “it is well known that any misspecification in Bayesian setups leads to lower performance.”: We totally agree. But this “lower performance” can be due to **underfitting or overfitting**, i.e., you can have a misspecified model that is severely overfitting your training data, as we show in Figure 1 (b). Understanding which of the two is present when we observe CPE is quite relevant. Reviewer wHkC states that this question “ hold[s] profound implications.” 

Having said that, we agree that we could do better in characterizing the relationship between misspecification and underfitting and will do so in the final version.

Reviewer comment (Novelty):
My main question is what is the argument that the authors are making? Reading the paper it looks like you are arguing that misspecification is causing the CPE by causing underfitting. It is well known that any misspecification in Bayesian setups leads to lower performance.

Author response:
Thanks for your feedback. Let us try to clarify the confusion. 

> The argument is a bit unclear from my perspective. The authors argue that the problem is under-fitting which comes from misspecification. So is the problem the under-fitting or the misspecification which causes the under-fitting

> The paper seems to be stepping on previous results and works, claiming that the misspecification of the prior or the likelihood lead to underfitting, and therefore underfitting is the problem that causes CPE. Well if CPE is present when under-fitting is present then the problem falls on either the prior or the likelihood.

> My main question is what is the argument that the authors are making? Reading the paper it looks like you are arguing that misspecification is causing the CPE by causing underfitting. It is well known that any misspecification in Bayesian setups leads to lower performance.

> I think I have understood the math and the experiments of the paper but the main argument does not seem very novel. Can the authors better explain?

To start, we would like to clarify that the main takeaway message of our work is the following: **The presence of CPE implies that our Bayesian setup is misspecified and underfitting at the same time.**

This is a simple, clean but actually **novel** way to interpret CPE.
1. Connections to previous work: The connection with underfitting has not been made before when trying to understand CPE. Most of the current literature focuses on the misspecification problem itself, but **not on misspecification in the context of underfitting**. We find it quite important that the ML community is fully aware of this point.
2. “it is well known that any misspecification in Bayesian setups leads to lower performance.”: We totally agree. But this “lower performance” can be due to **underfitting or overfitting**, i.e., you can have a misspecified model that is severely overfitting your training data, as we show in Figure 1 (b). Understanding which of the two is present when we observe CPE is quite relevant. Reviewer wHkC states that this question “ hold[s] profound implications.” 

Having said that, we agree that we could do better in characterizing the relationship between misspecification and underfitting and will do so in the final version.

Reviewer comment (Presentation):
One tip that I can give on the presentation is to add the results of the big models in the main text and the toy experiments in the Supplement. Or have a gradual increase of difficulty in the presentation of the experiments. For instance you start with linear regression on synthetic data, then you have a ResNet experiment on CIFAR10 and then on Imagenet.

Author response:
>One tip that I can give on the presentation is to add the results of the big models in the main text and the toy experiments in the Supplement. Or have a gradual increase of difficulty in the presentation of the experiments. For instance you start with linear regression on synthetic data, then you have a ResNet experiment on CIFAR10 and then on Imagenet.

Thanks for this tip! We will consider your advice in the next version.

Reviewer comment (Novelty):
According to the definition of CPE, as lambda increases, the test loss should decrease. Additionally, as $\lambda$ increases, the train loss always decreases irrespective of CPE. Up to this point, arguments in Theorem 2, Proposition 3 are accurate but rather self-evident.

Author response:
> According to the definition of CPE, as $\lambda$ increases, the test loss should decrease. Additionally, as  $\lambda$  increases, the train loss always decreases irrespective of CPE. Up to this point, arguments in Theorem 2, Proposition 3 are accurate but rather self-evident.

As we have answered in the overall rebuttal at the top of the page, we do agree that the connection is, in some sense, straightforward.  However, it's noteworthy that this link has not been recognized in the literature. The primary contribution of our study is to make the community aware of this connection. Because it is essential for comprehending and addressing the challenges associated with CPE.

Having said that, our formalization also provides new insights into CPE. Theorem 4 provides an interesting characterization not done before. Reviewer B5C9 is very positive about it. In our opinion, the connection with Data Augmentation does also provide new insights.

Reviewer comment (Theory):
The authors have not demonstrated that the distribution, whose existence was proven, becomes a posterior distribution (which should be definable from a new likelihood or a new prior). Hence, according to the definition “underfitting” defined by authors, Insight 1 may not be correct.

Author response:
> However, they have not demonstrated that the distribution, whose existence was proven, becomes a posterior distribution (which should be definable from a new likelihood or a new prior). Hence, according to the definition “underfitting” defined by authors, Insight 1 may not be correct.


Let us first clarify that, in our work, we refer to a “posterior distribution” to **any distribution over the parameter space which depends on the training data**. We refer to the “Bayesian posterior distribution” to a “posterior distribution” which is the result of the strict application of the Bayes’ rule. I.e.,  $p^\lambda(\theta|D)$ is a posterior distribution, while $p^{\lambda=1}(\theta|D)$ is the Bayesian posterior distribution. We will make this distinction crystal clear in our paper. 

Therefore, Insight 1 is correct as we said “there exists **another posterior**, rather than **another Bayesian posterior**, that has lower training and testing loss at the same time.”

As you say,  it would be great to know if this tempered posterior can be attained by some other Bayesian posterior coming from another likelihood/prior. We think this indeed is an interesting question. Although we think it is out of the scope to include this in our paper, where the focus is to highlight the connection between CPE and underfitting that has been overlooked so far, there are references showing positive answers, which we provide in the next section.

Reviewer comment (Theory):
After demonstrating that, at least one of the following two pieces of evidence should be provided:  
1) The original likelihood (or prior) is misspecified, compared to the new likelihood (or prior).  
2) An inverse proposition also holds (i.e., If there is underfitting, there exists CPE).

Author response:
>  I believe at least one of the following two pieces of evidence should be provided. 
The original likelihood (or prior) is misspecified, compared to new likelihood (or prior).
An inverse proposition also holds.(i.e., If there is underfitting, there exists CPE).

The first point is highly related to the previous question; it is clear that if CPE is present, prior and/or likelihood are misspecified, otherwise the Bayesian posterior would be optimal and CPE would not be present. As far as we know, at least in certain cases, the tempered posterior can be derived as the Bayesian posterior with a proper likelihood and prior (Wenzel et al., 2020 Wilson, Izmailov, 2020, Nabarro et al., 2022). Please refer to those for more details.


Regarding the inverse proposition; whether it holds is also a good question. However, it is not true in general; consider the extreme case where the prior distribution is a delta distribution centered at a model that does not explain either train or test data, let’s say $\theta_0$. In this setting, the Bayesian posterior is
$$
p(\theta|D) \propto P(D|\theta)p(\theta),
$$
which is also a delta distribution centered at $\theta_0$. The same happens with any tempered posterior. Even if the model $\theta_0$ is underfitting (there exist models with better training and testing loss), CPE does not appear in this context.

Reviewer comment (Theory):
In the current manuscript, the main claim has not been theoretically clarified.

Author response:
> In the current manuscript, I believe that their main claim has not been theoretically clarified.

With the above clarifications, our claim “If CPE is present, there exist alternative **posteriors** with better train and test loss” theoretically holds. As said before, what we don’t show is that “If CPE is present, there exist alternative **Bayesian posteriors** with better train and test loss” theoretically holds. We are fully aware this would be a stronger result. But we do consider this paper as a first step in this direction. This work can help to bring attention to this relevant open questions.

Reviewer comment (Presentation):
It seems that Figure 2 and Figure 3 do not connect well with the main claims of the paper. The common implication in the results appears to merely suggest that "CPE exists in image data analysis."

Author response:
> It seems that Figure 2 and Figure 3 do not connect well with the main claims of the paper. The common implication in the results appears to merely suggest that "CPE exists in image data analysis."

We will try to summarize the figures here and will update them in the final version of the paper.

All plots in the figures are used to support our main claim: **if we mitigate underfitting, which can be due to several different reasons (below), we will observe less CPE**. In particular, we demonstrate:
1. The effect of the priors in Figures 2a compared to 2c (as well as 3a compared to 3c): When a sharp prior underfits the data, using wider priors reduces the underfitting, and thus has less CPE.
2. The effect of likelihood misspecification in Figures 2b compared to 2c (as well as 3b compared to 3c): When a softmax likelihood underfits the data (a curated dataset in our example), using shaper (less misspecified in this context) likelihoods reduces underfitting, thus has less CPE.
3. The effect of informative data augmentation in Figure 2d compared to 2c (as well as 3d compared to 3c): When removing DA, there is less information in the dataset, making the model easier to fit the data, i.e., less underfitting, and thus has a weaker CPE.
4. The effect of the model size (compare Figure 2 to Figure 3): When the models are bigger and more capable, it’s easier for the model to fit the data, thus there will be less underfitting and less CPE.

Reviewer comment (Novelty):
In page 3, the authors state:  
“In the context of Bayesian inference, we argue that the Bayesian posterior is underfitting if there exists another posterior distribution with lower empirical Gibbs and Bayes losses at the same time.”  
Is this a commonly accepted concept or a newly proposed one? If it's the former, it seems appropriate to provide references.

Author response:
> In the context of Bayesian inference, we argue that the Bayesian posterior is underfitting if there exists another posterior distribution with lower empirical Gibbs and Bayes losses at the same time.”Is this a commonly accepted concept or a newly proposed one? If it's the former, it seems appropriate to provide references.

We tried to find definitions of underfitting in the context of Bayesian methods, but to the best of our knowledge, we were not able to find a commonly adopted definition. The definition we use here aligns with what is intuitively understood as underfitting in non-Bayesian settings.

Reviewer comment (Evaluation):
I am trying to understand what the next steps for research on this topic might be. Did you observe any situations where underfitting could not explain the CPE?

Author response:
If there is CPE, then your Bayesian posterior is underfitting. But, it might be the case that the posterior is underfitting and you do not have CPE. We haven’t said anything about the reverse implication. But, we do think this implication does not really hold in general. In the response to Reviewer wHkC we provide a counterexample.  

For us, a relevant question is: is there any problem in working with tempered posteriors? What is preventing us from tuning this hyperparameter when using a Bayesian approach? This lambda hyper-parameter could be tuned using, for example, an independent validation dataset. Providing theoretical arguments in favor of this approach would be valuable. This work would support this, because if we do not do it, we know we are not squeezing out all the power of (generalized) Bayesian methods.