Paper: On Bias-Variance Alignment in Deep Models (ICLR-2024)

Reviewer comment (Writing):
**Title** – The use of the word "alignment" may not be appropriate. The intended meaning seems to be about the relationship between two variables (e.g., linear or proportional), which "aligned" does not quite capture. Additionally, the term is overloaded with other connotations, such as in AI safety.

Author response:
Thank you for your careful review and insightful comment. We appreciate your feedback regarding the use of the term "bias-variance alignment" in the title of our paper.

We acknowledge that the word "alignment" has multiple meanings in different contexts, and we understand your concern that it might not be the most precise term to convey the relationship between the bias and the variance in our work. However, we carefully considered several alternative terms, including "linear relationship," "proportional relationship," and "correlation," but ultimately decided to retain "bias-variance alignment" for the following reasons:

1. **Clarity and Conciseness:** The term "bias-variance alignment" is concise and easily understood, effectively conveying the idea that bias and variance are related in a specific way, namely that they are not independent of each other but rather exhibit a relationship.

2. **Contrast with Bias-Variance Trade-off:** The term "bias-variance alignment" serves to contrast with the more commonly used term "bias-variance trade-off." While the trade-off suggests that bias and variance are inversely related, our work highlights that they can also be aligned, implying a more nuanced relationship.

3. **Contextual Understanding:** We believe that readers familiar with machine learning will understand the intended meaning of "bias-variance alignment" without confusion with other uses of the term "alignment" in AI safety or other areas.

We understand that there is always room for debate about the choice of words in scientific writing, and we are open to considering alternative phrasing in future work. However, we believe that the term "bias-variance alignment" effectively conveys the essence of our findings and provides a clear distinction from the traditional bias-variance trade-off perspective.

Thank you again for your valuable feedback. We appreciate your thoughtful consideration of our work.

Reviewer comment (Novelty):
**Related work** – The related work section is highly incomplete, lacking references to previous works in the ensemble or deep ensemble literature. Many previous works (both recent and older) have explored overlapping aspects of decomposing ensembles into bias and variance, studied the role of overparameterization on variance, and observed that increasing ensemble variance can also increase predictive bias. The authors should perform an extensive literature review to position their work relative to these and other relevant studies.

Author response:
Thank you for these valuable references. As part of our rebuttal revision, we have included a comprehensive "Further Related Work" section (Appendix C) that cites and discusses the relevant papers you mentioned. For each of these papers, we have provided an in-depth discussion to highlight their contributions.

In the final version of our manuscript, we plan to integrate a condensed version of this related work discussion into the main body, ensuring that the reader receives a concise yet thorough overview of the existing literature.

It is important to note, however, that none of these works uncovers our key finding of the per-sample bias-variance alignment. Your insights have been instrumental in refining our work, and we appreciate your thoughtful review.

The discussion in the "Further Related Work" section is as follows:

"Gupta et al. (2022) investigate the theoretical underpinnings of ensemble methods for classification tasks, extending the bias-variance decomposition to derive generalized laws of total expectation and variance for nonsymmetric losses. Their work sheds light on the mechanisms by which ensembles reduce variance and potentially bias, providing valuable insights for improving the performance of ensemble classifiers.  Ortega et al. (2022) provide a comprehensive theoretical framework for understanding the relationship between diversity and generalization error in neural network ensembles. They  analyze  the  impact  of  diversity  on  ensemble  performance  for  various  loss  functions  and model combination strategies, offering valuable insights for designing effective ensemble learning algorithms. Brown et al. (2005) focus on managing diversity in regression ensembles. They introduce a control mechanism through the error function, demonstrating its effectiveness in improving ensemble performance over traditional methods.  This work provides insights into systematic control of the bias-variance-covariance trade-off in regression ensembles. Abe et al. (2023) challenge conventional wisdom on predictive diversity in deep neural network ensembles.  While diversity benefits small models, the authors find that encouraging diversity harms high-capacity deep ensembles used for classification. Their experiments show that diversity-encouraging regularizers hinder performance, suggesting that the best strategy for deep ensembles may involve using more accurate but less diverse component models. In contrast to traditional ensemble methods, deep ensembles of neural networks offer the potential for direct optimization of overall performance. However, Jeffares et al. (2023) reveal that jointly minimizing ensemble loss induces base learners to collude, inflating artificial diversity. This pseudo-diversity fails to generalize, highlighting limitations in direct joint optimization and its impact on generalization gaps."

Reviewer comment (Presentation):
**Calibration** – While the relationship between calibration and bias-variance is interesting, this section could be less detailed to allow more space for in-depth empirical evaluation of the main claims. The section could be restructured for clearer presentation, with some technical details moved to the appendix to focus on the connection to bias-variance alignment and upper-bounded variance.

Author response:
Thank you for your thoughtful feedback on the calibration section of our paper. We appreciate your suggestion to streamline the presentation of technical details. In response to your comments, we would like to highlight that the submitted version has already placed most of the technical intricacies, including all proofs, in the appendix.

In the "Calibration Implies Bias-Variance Alignment" section, we have deliberately focused on presenting the main theorem (Theorem 4.3) and its two corollaries (Corollary 4.4 and Corollary 4.5). The main theorem (Theorem 4.3) plays a pivotal role in characterizing the bias-variance gap and establishing its relationship with the calibration error. Corollary 4.4 is a key result that demonstrates the bias-variance alignment under the calibration assumption, showing that if an ensemble is perfectly calibrated and exhibits high confidence, then the bias equals the variance. This corollary is crucial in substantiating our main claim regarding the bias-variance alignment.

Furthermore, Corollary 4.5 delves deeper into the subject, providing additional insights and demonstrating that, under the calibration assumption, bias-variance alignment occurs on an entrywise basis. These three technical results are the cornerstone of our work, and we believe their inclusion in the main body is essential for clarity and coherence.

We acknowledge your suggestion to enhance clarity further and will carefully consider your feedback while revising the section. However, preserving these core results in the main body is integral to ensuring that our empirical claims are firmly grounded in the underlying theoretical framework. We hope this clarifies the rationale behind the current structure of the calibration section.

Reviewer comment (Theory):
**Neural collapse** – The Neural collapse section contributes little to the paper. The assumptions are restrictive, the results do not seem particularly useful, and the section is too short to provide adequate detail and background. It should be moved to the appendix to free space for expanding the empirical evaluation.

Author response:
We appreciate your feedback on the section discussing neural collapse in our paper. While we understand your perspective on the limitations of the neural collapse section, it's important to note that other reviewers have expressed positive views on this section.

For instance, Reviewer ehNn highlighted the strength of our neural collapse section in their review, stating, "The authors found two quite simple theoretical scenarios that can replicate the bias-variance alignment. Moreover, these scenarios are quite distinct: the calibration part requires no specific assumptions and provides rigorous upper bounds, while the neural collapse part uses a very specific model which admits an exact and explicit solution (i.e. not just an upper/lower bound)." This positive assessment underscores the value of the distinct theoretical scenarios we present, showcasing the diversity of our approach.

Additionally, in the "Strengths" section of Reviewer 18mF's review, it is noted that "Assumption 5.1 on model prediction under neural collapse is very interesting. Using this to show the bound the ratio of variance and bias is novel." This acknowledgment reinforces the novelty and interest surrounding Assumption 5.1 and its role in establishing bounds on the ratio of variance and bias under neural collapse.

While we understand your suggestion to consider moving this section to the appendix to make room for expanding the empirical evaluation, the positive feedback from other reviewers indicates that the neural collapse section has its merits. We will carefully consider your input and aim to strike a balance between addressing your concerns and preserving the valuable contributions of the neural collapse analysis in our paper.

Reviewer comment (Writing):
The two claims on page 2 are empirical but are presented as if they are theoretical truths, which led to misunderstanding. This should be clarified in the text.

Author response:
Thank you for your insightful feedback. We would like to highlight that these claims are indeed supported by our underlying theory, specifically under the calibration assumption, as demonstrated in Section 4, with Corollary 4.4 providing further corroboration. Moreover, we would like to draw your attention to Table 1, where we have taken care to clearly and explicitly indicate which results are theoretical and which are empirical. 

Nevertheless, we acknowledge that additional clarification in the main text can help clarify the nature of our contributions. In response to your suggestion, we have revised the introduction of the two claims on page 2 to underscore their empirical nature. The modified sentence now reads:

"Our key **empirical** observations can be summarized with the following two statements, which we call the Bias-Variance Alignment and the Upper Bounded Variance."

We hope that our revision addresses your concern effectively, and we hope that it contributes to a clearer understanding of our manuscript.

Reviewer comment (Presentation):
Many plots are very small, especially the text within them, making them difficult to read.

Author response:
We appreciate your feedback regarding the size of the plots and the text within them. We will address this issue by remaking the figures and making them larger and easier to read. We will also increase the size of the text within the figures so that it is more legible. We will make these changes in the final version of the manuscript.

Reviewer comment (Writing):
**Question** – Is the \( C_{h_{\theta}} \) in eqn (2) the same as the one in eqn (3)? This should be made more clear.

Author response:
Yes, the $C_{h_{\theta}}$ in Equation (2) is the same constant as in Equation (3). We have added a sentence to the manuscript to make this more clear.

Reviewer comment (Theory):
Assuming that the model predictions are well calibrated seems is a very strong assumption, thus using this to bound the bias variance gap makes the result less interesting.

Author response:
Thank you for your feedback. We'd like to address a potential misunderstanding regarding our approach. We want to clarify that we do not make the assumption that model predictions are inherently well-calibrated. Rather, our methodology involves bounding the bias-variance gap by considering the calibration error of an ensemble. This approach does not rely on assuming good calibration of individual models; instead, we establish a connection between the bias-variance gap and the level of calibration. If you have additional questions or concerns, please feel free to reach out.

Reviewer comment (Presentation):
The colors for negative and positive classes in figures 12,13 are swapped compared to the rest of the plots in the papers. Though completely stylistic, it would be nice to have a consistent color scheme to make paper reading more comfortable.

Author response:
Thank you for bringing this to our attention! We acknowledge the inconsistency in the color scheme for figures 12 and 13 compared to the rest of the plots in the paper. We intend to revise these figures to ensure a consistent color scheme throughout the entire presentation. Your observation is much appreciated.

Reviewer comment (Theory):
It took me some time to verify that variable $F$ appearing in eq. (59) indeed has $F(2(K-1),2)$ distribution. Since this fact is central for the following computation and, to my understanding, motivates the choice of Gumbel distribution for $v$ in assumption 5.1, it would be better to discuss and explain this explicitly in appendix.

Author response:
Thank you. In the revision rebuttal, we have added an explanation that this is because $e^{u_i}\sim \mathrm{Exp}(\lambda) = \frac{1}{2\lambda}\chi_2^2$.

Reviewer comment (Presentation):
Histograms for negative classes have disappeared in the middle row of Figure 12.

Author response:
Thank you for pointing that out. We suspect that the absence of histograms for negative classes in the middle row of Figure 12 might be attributed to a rendering issue with the vector graphics, particularly due to the numerous bins in the negative class histogram. We have opted for vector graphics to enhance the professional appearance of the figure. We are committed to addressing this concern and plan to recreate the figure, aiming to resolve the rendering issue. Your feedback is valuable, and we appreciate your attention to detail.

Reviewer comment (Presentation):
Does figure 13 contain only correctly classified samples or all of them? In the latter case, it would be interesting to know why the approximately vertical shape of wrongly classified points (e.g. a "blue tail" on figure 1.b) is not present in the synthetic data.

Author response:
Thank you for your question regarding Figure 13, and we appreciate the opportunity to provide clarification. Figure 13 indeed includes all samples from the dataset, not just the correctly classified ones. The absence of the approximately vertical shape, particularly the "blue tail" observed in Figure 1.b, is attributed to the fact that all the samples in Figure 13 are correctly classified.


It's important to note that Figure 13 illustrates the bias-variance alignment on a synthetic dataset generated in accordance with the principles of the neural collapse theory. In this idealized setting, where the data conforms perfectly to the theoretical framework, we observe a full alignment between bias and variance.


We appreciate your keen observation, and in response to your question, we have included a sentence in the paper to explicitly state that Figure 13 contains all samples and that they are correctly classified.

Reviewer comment (Theory):
*Choice of Gumbel distribution*. Overall, the validation of assumption 5.1 presented in sec. G.2 seems to support the hypothesis that we may write $\psi_\tau(X)=R(sw^{\mathrm{ETF}_Y+v})$ with random vector $v$ whose entries are i.i.d., zero mean, and have unimodal shape. For example, Gaussian distribution satisfies these criteria. So is there some additional evidence for choosing Gumbel distribution, besides it enabling an analytical solution?

Author response:
Indeed, the choice of the Gumbel distribution was primarily motivated by its ability to facilitate an analytical solution, despite the complexity of the mathematical expression. In contrast, opting for a Gaussian distribution introduces challenges, as even the expectation of the softmax of a Gaussian random vector lacks a tractable form. Researchers often have to resort to the mean-field approximation, as illustrated in the following reference:

Lu, Z., Ie, E., & Sha, F. (2020). Mean-field approximation to gaussian-softmax integral with application to uncertainty estimation. arXiv preprint arXiv:2006.07584.

Reviewer comment (Reproducibility):
After eq. (10) it is mentioned that the paper considers (except sec. E.2) MSE loss on top of classifier probabilities. However, I did not find any mention that this is also applied to the training of neural networks used for empirical results - that would be quite non-standard from a practical point of view (CE loss is usually used in practice). Could you please provide details on how the networks were trained?

Author response:
Thank you for your question. We appreciate your attention to detail. We want to clarify that, in practice, all of the neural networks utilized in our study were trained with the Cross-Entropy (CE) loss, adhering to the standard practice in the field. To address this, we have included a clarification at the end of page 3 to explicitly highlight the use of CE loss in the training of our networks. If you have any further questions or require additional details, please feel free to ask.

Reviewer comment (Theory):
If, in the previous question, CE loss was used, I would expect that scale parameter $s$ of ETF complex grows with training time in the terminal phase of training (it is required for reduction of train loss to extremely small values). Then, by training the network for a very long time, one can hope to cover a wide range of $c$ values from Corollary 5.3, and thus experimentally check whether theoretical prediction that $\log \mathrm{Bias}^2 / \log \mathrm{Var}$ changes from $0.557$ to $2$. It would be interesting to see whether the theory can work on such a fine level of detail. However, this also contradicts \textit{bias-variance alignment} statement (1) since it requires $\log \mathrm{Bias}^2 / \log \mathrm{Var} \to 1$ as $\log \mathrm{Var}\to-\infty$.

Author response:
Thank you for your thoughtful question and insightful observations regarding the evolution of the ETF in the context of neural collapse theory. We appreciate the opportunity to discuss these complex issues with you.

The evolution of the ETF with training time, as you rightly pointed out, is an intriguing aspect of our work. We acknowledge that delving into the intricacies of this evolution may necessitate a dedicated exploration in a subsequent paper, accompanied by extensive numerical results to validate theoretical predictions.

In our current paper, our primary focus is on examining the alignment between bias and variance as we terminate neural network training. The question of how this alignment evolves with prolonged training is indeed a pertinent one, and we agree that it could be an interesting avenue for future research.

Regarding the latter part of your question, we agree with the reviewer that the behavior of the ratio $\log \mathrm{Bias}^2 / \log \mathrm{Var}$ is contingent on the range covered by the parameter $c$ from Corollary 5.3. 

It is indeed plausible that, with prolonged training, the parameter $c$ might converge to a specific value, resulting in bias-variance alignment. Alternatively, bias-variance alignment could occur when training is halted, possibly when the model has achieved a satisfactory level of performance. We also acknowledge the possibility you raised, where the ratio evolves from $0.557$ to $2, but such a transition might necessitate an impractically long training duration, and practitioners may have terminated training due to the associated increase in validation error.

We appreciate your valuable feedback and are open to further discussion on these intriguing aspects of our work. If you have any additional questions or suggestions, please feel free to share them.