Paper: On the Variance of Neural Network Training with respect to Test Sets and Distributions (ICLR-2024)

Reviewer comment (Theory):
In the proof of Lemma 1, the exponent should be inside the expectation. Additionally, it's probably helpful to justify the penultimate step (interchange of expectation) by saying that Fubini's Theorem applies.

Author response:
We have updated the draft to add extra parentheses inside of the expectation brackets, such that the square is moved inside. We also noted as suggested that Fubini’s theorem is what allows the interchange of expectations.

Reviewer comment (Experiments):
Is the test set split fixed? In other words, when the authors write "we split the test-set into two halves of 5000 examples each", does this mean there is a single fixed split of the test set? If so, this seems to not make sense in the context of the proposed formalism ($S \sim \mathcal{D}^n$), which considers randomly sampled test-sets.

Author response:
The reviewer asked whether our test-set split is fixed. Yes, in Figure 2 we use a single fixed split of the test-set, into two halves.

To say why this is the case, we’ll first give a summary of the main experiment, which proceeds as follows: 

We first generated a fixed split of the CIFAR-10 test-set into two halves of 5,000 examples each. Then we trained 60,000 models on the CIFAR-10 training set, and evaluated them all on both test splits.

We found that the best-performing model on split 1 reached significantly higher performance than average - almost 95%, vs. an average of 94.2%. But this best model on split 1 didn’t do any better than average on split 2. In fact, we measured that the correlation between performance on the two splits is close to zero.

Resampling the split does not change this result - the resulting correlation will always be near the same value.

We also note that the mathematical analysis which proceeds in section 3.3 does not depend on using splits at all. Instead, the estimator given in Theorem 2 directly computes the average pairwise correlation.

We hope that this clarifies how and why splits are used in our analysis, and shows that using a fixed split is not a problem.

Reviewer comment (Presentation):
The two main itemized contributions contain way too many points to be in a list. They are paragraph sized. I do not think something of that size should be presented in a list. I believe they either need to be broken up into smaller components, or discussed in plain text instead of a list.

Author response:
We thank the reviewer for pointing out several problems with the presentation, which prompted us to make the following improvements to the draft:

* We have converted the list of contributions in the introduction into two paragraphs.

Reviewer comment (Writing):
What is $n$ in theorem 3, 4, and 5? I suppose it should be the size of the dataset, but it would be nice to include this right by the theorems in order to avoid any possible confusion.

Author response:
We thank the reviewer for pointing out several problems with the presentation, which prompted us to make the following improvements to the draft:

* We have clarified in the statements of Theorem 3-5 that $n$ refers to the size of the test-set.

Reviewer comment (Experiments):
I think it would be interesting to see the difference in Theorem 3 when varying the number of models in the ensemble. For instance, the bias in the estimate may be small, and a two model ensemble may give good results, or it may actually require quite a large ensemble in order to be close to correct. Can you add this experiment? I believe it can be done by only randomly choosing and expanding the ensemble of the models which are already trained.

Author response:
**Applying Theorem 4 to ensembles of two models**

In the paper, we focused on the variance in performance between models that are members of large ensembles of >1,000 independently trained networks. The reviewer made the interesting suggestion that we should look at smaller ensembles of two networks.

We found this suggestion interesting and have added a new theorem to the paper which derives the expected difference in accuracy between two models.

This can be found as Theorem 6 in the appendix. We prove that for binary classification, the expected squared difference between the test-set accuracies of two independently trained models is equal to the average error divided by the size of the test set.

We proved this under the assumptions of theorem 4, which are shown to approximately hold by the main experiments of the paper.

We hope that this theorem will provide practitioners with a guideline for how much variance to expect to see between pairs of trained models. In particular, if two models are trained with different hyperparameters, and their accuracy differs by significantly more than the prediction of Theorem 6, then there may be a genuine difference between the quality of the two hyperparameters. Otherwise, there may be no statistically significant difference.

Reviewer comment (Presentation):
Many of the claims listed as main contributions are in the appendix. Even in the conclusion, the things in the appendix as main contributions which have been demonstrated even though there has been nothing said about them up until this point in the text. I do not think this is fair to the reader and it should be reorganized or rewritten such that this does not happen.

Author response:
We thank the reviewer for pointing out several problems with the presentation, which prompted us to make the following improvements to the draft:

* Regarding the claims listed in the conclusion (regarding connections between variance and other phenomena), we have made it more explicit that two of them are presented only in the appendix. First, in the beginning of Section 4 we refer to the two additional experiments in the appendix. Second, in the conclusion we have added explicit pointers to the section that each claim is presented in. We regret that space issues precluded them being put directly in the main text.

Reviewer comment (Theory):
In section 3.4 it says: “That is, if we let S ′ be the subset of test images which are classified by 30-40% of independently trained networks as “dog”, then approximately 30-40% of the images in S ′ really will be dogs.” I do not think that is how calibration is phrased in most calibration works. In fact, all networks may correctly classify the image as a dog (meaning that the highest predicted probability is the dog class), but the predicted probability (confidence) should equal the empirical distribution of the positive class in $S^\prime$, then the model is calibrated.

Author response:
**Regarding the definition of calibration**

We agree with the reviewer that our notion of calibration, called “ensemble-calibration” and referencing [1], is different from the standard version of calibration defined for single models.

Indeed, the standard single-model calibration described typically *does not hold*, as neural networks tend to be overconfident [2]. On the other hand, the ensemble-calibration we describe *does* approximately hold, as observed in [1] and implicitly in [3].

This is good because ensemble-calibration is mathematically what is needed to prove Theorem 3-5, whereas standard calibration is not useful.

We hope this has clarified our usage of the notion of calibration.

Reviewer comment (Presentation):
In figure 2, why is (odd examples) in parentheses? I cannot figure out what this means since there are only two splits and they seem to be uniformly random.

Author response:
**Regarding the caption of odd vs even examples in Figure 2**

We have added the following sentence to section 3.1, which describes the reason for this: “CIFAR-10 is already shuffled, so for convenience we simply use the odd and even-indexed examples as the two halves.” We have confirmed that if we use a newly selected random split instead, it makes no difference for the results.

Reviewer comment (Evaluation):
Another concern is how well does the conclusion itself generalize to other settings. In Figure 1, it seems the variance becomes smaller as training goes on. But I think it is caused by the learning rate schedule: the authors say that they "always linearly ramp the learning rate down to zero by the end of training", so it is expected that the accuracy has less variation in the end.

Author response:
**Clarification of the learning rate schedule**

> In Figure 1, it seems the variance becomes smaller as training goes on. But I think it is caused by the learning rate schedule: the authors say that they "always linearly ramp the learning rate down to zero by the end of training", so it is expected that the accuracy has less variation in the end.

We have updated the draft (in section 2) to clarify that this means that, even for the four-epoch duration, we ramp the learning rate down to zero by the end of the four epochs. Thus, none of our trainings, no matter how short, end with a nonzero learning rate. We hope that we have correctly understood the reviewer’s concern.

We note that if the learning rate were not ramped all the way down, then indeed the variation would be greatly increased for the shorter trainings. But this isn’t what we did.

Reviewer comment (Experiments):
How does the conclusion generalize to other training setting or other network architecture (like Transformers)?

Author response:
**Generalization to transformers**

In Section 4.1, we confirm that for a canonical transformer-encoder finetuning scenario (BERT-base finetuned on MRPC), our results do empirically generalize, with test-distribution variance being small relative to test-set variance. For BERT-Large finetuning our results do not generalize, but this seems to be explained by this being a widely recognized case of pathological training instability [1, 2].

Thus, we have at least preliminary evidence that our results generalize to this quite different other setting. We agree with the reviewer that, for example, it is an interesting direction for future work to extend our study to transformers trained from scratch. (e.g. LLM pretrainings)


[1] Mosbach, Marius, Maksym Andriushchenko, and Dietrich Klakow. "On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines." arXiv preprint arXiv:2006.04884 (2020).

[2] Dodge, Jesse, et al. "Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping." arXiv preprint arXiv:2002.06305 (2020).