Paper: $\nu$-ensembles: Improving deep ensemble calibration in the small data regime (ICLR-2024)

Reviewer comment (Novelty):
The paper lacks the related works of other calibration method such as train time calibration loss, and post hoc calibration which is very important in this domain.

Author response:
Thank you for your detailed and useful review!

**Addressing the weaknesses:**

We would firstly like to point out that our method is a diversity promoting method for ensembles and not a regularization method or a new loss function for single networks. Thus in principle it can be combined with a number of calibration improving methods for single networks such as the focal loss, augmentation, and label smoothing. Indeed in the original submission we included standard augmentation flips + crops which improve \nu-ensembles.

We read the mentioned related works in detail. To the best of our understanding the primary contributor to improved calibration for single networks is temperature scaling. For example, in the most recent Focal Loss variant (dual focal loss) [1], temperature scaling results in large improvements over standard evaluation ~10-20%, however the dual focal loss only gives small additional improvements ~1%. We thus implemented temperature scaling which combined with nu-ensembles is in most cases the best calibrated ensemble.

Reviewer comment (Evaluation):
From my experience, the ECE measurement could be very unstable when classification accuracy is low. For experiments in table 1 for CIFAR100, the accuracy is very low, and the results may not reliable.

Author response:
Thank you for your detailed and useful review!

**Addressing the reviewer questions:**

We provide additional experiments with multiple seeds in the Appendix. The results remain largely unchanged.
We note also the following: 1) when the training data is small our method outperforms in calibration also in other metrics such as the TACE, the Brier and the NLL with especially the TACE considered to be a more robust variant of the ECE. 2) One can observe in Figure 2 that we continue getting relatively robust gains not only in the small training data regime but also in modest to large training set sizes 10000 to 40000 where the accuracy is within an acceptable range. Please note that using multiple seeds for all the experiments is prohibitive even though the initial setup is with a small training set.

Reviewer comment (Experiments):
The experiments lack the comparison with SOTA methods such as Focal Loss Calibration and Adaptive Label Smoothing.

Author response:
We would like to thank the reviewer for his/hers detailed and careful review. We will respond in detail in the upcoming days. For the moment, could you please kindly provide us with references on the "Focal Loss Calibration" and "Adaptive Label Smoothing" methods as well as the medical datasets you are referring to? (Ideally if any are available using tensorflow datasets - torchvision, that would be very convenient).

R2: Thank you for your detailed and useful review!

**Addressing the weaknesses:**

We would firstly like to point out that our method is a diversity promoting method for ensembles and not a regularization method or a new loss function for single networks. Thus in principle it can be combined with a number of calibration improving methods for single networks such as the focal loss, augmentation, and label smoothing. Indeed in the original submission we included standard augmentation flips + crops which improve \nu-ensembles.

We read the mentioned related works in detail. To the best of our understanding the primary contributor to improved calibration for single networks is temperature scaling. For example, in the most recent Focal Loss variant (dual focal loss) [1], temperature scaling results in large improvements over standard evaluation ~10-20%, however the dual focal loss only gives small additional improvements ~1%. We thus implemented temperature scaling which combined with nu-ensembles is in most cases the best calibrated ensemble.

Reviewer comment (Experiments):
**Insufficient experimental setup for proposed method.** The authors claim that for small-scale datasets their method preserves the performance boost of standard ensembling but results in better calibration, while maintaining the same level of efficiency (as opposed to joint training methods that are compared to). However, this comparison seems incomplete - firstly, my understanding is that the compared-to ensembling approaches do not make use of the additional unlabeled data (at least standard ensembling does not). In Table 1 (the main table in the paper) the results are with respect to a training size of 1000 data points, but the unlabeled data size and validation size are 5000 points each. As a result, this comparison seems unfair - one should at least consider some other pseudo-labeling scheme for the unlabeled data, since it makes up the majority of the data being considered.

Author response:
Please note that the agree to disagree method uses unlabeled data, however, it has both lower accuracy and calibration compared to our approach. We discuss the use of pseudo-labelling in the related works section. We do not dispute that pseudo labelling could improve standard ensembles, perhaps significantly. However, this is natural, as our method is extremely simple and makes minimal assumptions about the unlabeled data, namely only that it has been generated i.i.d. from the source distribution. Pseudolabelling, as for example implemented in [1], requires multiple rounds of pseudolabelling where a small percentage of the unlabelled data is labelled at a time. To obtain the final best performance a final distilled networked needs to be trained. This makes the implementation quite involved and includes many hyperparameters. By contrast, we propose a method that is almost as simple as temperature scaling or data augmentation with a single hyperparameter. 

We emphasize that our approach is not a regularization method of individual networks but a diversity promoting method of ensembles. Thus in principle it can be combined with any of the methods proposed by the reviewer (mixup, focal loss, temperature scaling). In particular we already implement standard data augmentation (flips + crops) which benefit nu-ensembles. Finally we include in the revised version experiments on temperature scaling + nu-ensembles which have the best ECE and other calibration metrics compared to the alternatives in most cases.

Reviewer comment (Experiments):
The experimental setup should also include comparisons to training ensembles with some kind of data augmentation (label smoothing, Mixup, etc.) since these methods are not only known to improve feature learning diversity but also regularize predicted confidences. Furthermore, training with these methods is going to be even more efficient than the proposed approach, and I expect would perform better. The reasons for expecting this are two-fold: firstly, the proposed approach intuitively regularizes confidence by having the ensemble uncertainty be high on the unlabeled data (essentially these points should be predicted uniformly randomly based on how the ensembles are trained), but Mixup and label smoothing are approaches that can do this as well. Additionally, and more importantly, the authors themselves note that their approach does not work (and can even hurt) for larger dataset sizes, but the aforementioned data augmentations are known to improve calibration even in that regime.

Author response:
We emphasize that our approach is not a regularization method of individual networks but a diversity promoting method of ensembles. Thus in principle it can be combined with any of the methods proposed by the reviewer (mixup, focal loss, temperature scaling). In particular we already implement standard data augmentation (flips + crops) which benefit nu-ensembles. Finally we include in the revised version experiments on temperature scaling + nu-ensembles which have the best ECE and other calibration metrics compared to the alternatives in most cases.

Reviewer comment (Theory):
**Theoretical approach needs greater clarity.** The theory here needs significantly more clarification. For example, the authors define $\hat{\rho}$ to be a uniform combination of point masses on different weights (and even here the notation should be made more precise, $\delta$ is not defined a priori) and then claim that $\hat{V}$ is the empirical variance of $\hat{\rho}$, but that is not what it represents from Equation (2), which is the variance of the predictive distribution of the ensemble when evaluated with respect to the true labels of data points in $U$. Furthermore, for the predictive distribution the authors use $p(y \mid x, f)$ in Equation (2) and it should be clarified at this point that $y$ corresponds to the true label of $x$ (which the authors mention later).

Author response:
To the best of our knowledge $\hat{\mathbb{V}}$ is defined as the variance of the predictive distribution in our text. If there is some point where we also erroneously define it as the variance of $\hat{\rho}$ we would be happy to correct this.

To the best of our knowledge it is fairly common to overload the notation for y and similar objects such that they represent both variables and variables instantiated with specific values [2]. We have moved the proof to the Appendix where we make every proof step in much more detail.

Reviewer comment (Theory):
The proof of Proposition 1 is very hard to make sense of. It is unclear what the indicator variable of $y$ not being in the random labels means. Aren't the random labels supposed to be exhaustive? Even ignoring this, how does the second term become zero when passing from the first line to the second line?

Author response:
We would like to clarify that for each data point the pseudolabels we create are exhaustive up to the number of ensemble members. Thus if we have 3 ensemble members and 10 classes we create K=3 unique labels. The more ensemble members we have, the more unique labels we have and the larger our variance term becomes. This makes both intuitive sense and is the cause of the variable K appearing in Proposition 1.

If the true label y is not in the K randomized labels then all $p(y|\boldsymbol{x},\boldsymbol{w}_i)=0$. Intuitively we've assumed that each ensemble member perfectly fits it's randomized label. Then since all ensemble members do not fit the true label they will all have $p(y|\boldsymbol{x},\boldsymbol{w}_i)=0$.

We would be happy to provide any additional clarifications.

Reviewer comment (Reproducibility):
The method "Sample y randomly without replacement", however, when the number of ensemble is larger than the number of classes, it is unclear to me how the method should be applied.

Author response:
When the number of ensemble members is larger than the number of classes we can sample the labels randomly with replacement. We provide a theoretical analysis of this setting in proposition 2. Our analysis shows that the variance of the ensemble should continue to grow (albeit slowly). We note that in practice a setting beyond 5-10 ensemble members is to the best of our knowledge rare.

Reviewer comment (Experiments):
Since the method assumes having access to a validation dataset, a baseline worth considering would be temperature scaling.

Author response:
We include experiments using temperature scaling in the revision. The combination of nu-ensembles + temperature scaling is the best in terms of ECE and other calibration methods in most experiments. We also note that nu-ensembles are almost as simple to implement as temperature scaling, and can also be used in settings where applying standard temperature scaling is difficult. These cases are for example when using machine learning libraries that make modifying models difficult and/or output only the softmax activations instead of the logit preactivations (for example this is the case with some scikit-learn classification models).

Reviewer comment (Evaluation):
Why the method becomes less effective when we have access to more data?

Author response:
We make very simple assumptions about our unlabeled data. When we have few training data new unlabeled data probably cover parts of the input space that the training data doesn't cover. We push our predictors to be uncertain on these parts of the input space. As the training dataset becomes larger and larger, training samples start "hitting" parts of the input space covered by unlabeled samples. It no longer makes sense to assume that we know nothing about this part of the input space. Our regularization parameter \beta captures this tradeoff and becomes close to 0 when we encounter this problem. We also include Figure 1 in the Appendix, which uses a toy example to capture these tradeoffs.

Reviewer comment (Evaluation):
The method is targeted solely at the small data regime, gains in calibration go to zero as the amount of labelled data increases.

Author response:
We include in the Appendix additional experiments with multiple seeds for the Standard and $\nu$-ensembles case. The conclusions are the same as the main. We understand that the reviewer views the use of our method in the small data regime as a weakness, however at the same time the small data regime is quite widespread in practice and the subject of considerable research in both ML conferences and journals. PAC-Bayes bounds are in general loose except for specific instantiations that have been optimized numerically to be tight. At the same time they are known to correlate well with out-of-sample performance [1], and also they have provided usefull objectives for improving deep neural networks [2].

Our method is extremely simple and doesn't incorporate any prior information about the unlabeled data apart from the fact that they have been generated i.i.d from the source distribution. Thus, we claim that it works for small to medium training set sizes. The converse (as the reviewer proposes) is also true, the unlabeled set should also not be too large, as in that case we need more complicated assumptions about how to label the unlabeled data. We created Figure 1 in the appendix that illustrates these tradeoffs. For a small training set and small unlabeled set we can both fit the training set while also forcing diversity out-of-sample. For, very large unlabeled sets we underfit the training data. But this is natural as our assumptions are very simple. Conversely for very large training sets the unlabeled data are not useful (or might even hurt the final predictor). The PAC-Bayes bound captures this tradeoff with a complexity term.

We want to note that the above is not a significant problem for our algorithm in the following sense: It suffices to do a hyperparameter search over \beta using a validation set to eliminate the issue. If we include \beta = 0 in our grid, the hyperparameter search can default to standard ensembles in these pathological cases by choosing this \beta=0 value. In fact this is what happens when our ECE gains become 0 as the training set increases in Figure 2 of the main.

Reviewer comment (Experiments):
The method introduces a new $\beta$ hyperparameter which must be tuned.

Author response:
We want to note that the above is not a significant problem for our algorithm in the following sense: It suffices to do a hyperparameter search over \beta using a validation set to eliminate the issue. If we include \beta = 0 in our grid, the hyperparameter search can default to standard ensembles in these pathological cases by choosing this \beta=0 value. In fact this is what happens when our ECE gains become 0 as the training set increases in Figure 2 of the main.

Reviewer comment (Evaluation):
The experiments are presented without error bars and it is unclear if they come from a single run or are averaged over multiple seeds, standard practice, especially when considering the relatively small datasets considered in this paper, is to run experiments with multiple random seeds and present averages and standard deviations of the metrics of interest (or better yet other forms of statistical test of the significance of the results).

Author response:
We include in the Appendix additional experiments with multiple seeds for the Standard and $\nu$-ensembles case. The conclusions are the same as the main.

Reviewer comment (Theory):
The primary theoretical motivation for the method is equation 1, which is a PAC bound on the generalization performance, it is difficult to get a sense of how tight this bound is and to what extent there is a competition between the various terms in the bound.

Author response:
PAC-Bayes bounds are in general loose except for specific instantiations that have been optimized numerically to be tight. At the same time they are known to correlate well with out-of-sample performance [1], and also they have provided usefull objectives for improving deep neural networks [2].

Reviewer comment (Experiments):
While the experimental results do not show big drops in accuracy, there is concern that given vastly more unlabelled data the method would lead to overfitting the random labels and thereby harm test set accuracy (as is a well known phenomenon in the noisy label literature). More formally, one could imagine that vast amounts of unlabelled data would promote the diversity term in the RHS of equation 1, but given results in the noisy labels literature, it is hard to believe that this would not come at a corresponding cost in the first term on the RHS of equation 1. The reviewer requests comments on this concern. Experimentally, they would be interested in seeing an experiment on ImageNet, for example, where the labelled set is of size 50k and the unlabelled set is 950k examples, a standard resnet50 or similar capacity model is used with 4 ensemble members (as per other papers in the literature) and a comparison to standard ensembles in terms of accuracy and calibration is given. This is seen as a significant concern, as usually with methods that make use of an unsupervised dataset, the expectation is that as the unlabelled dataset grows, the gains from using it grow too. The reviewer fears this will not be the case for this method, which would limit it to the small dataset, small number of classes and small unlabelled dataset regime. They recognise that the $\beta$ hyperparameter can to a certain extent control this trade-off, so if further experiments are conducted to address this concern, they request that the results over the $\beta$ hyperparameter range be reported.

Author response:
Our method is extremely simple and doesn't incorporate any prior information about the unlabeled data apart from the fact that they have been generated i.i.d from the source distribution. Thus, we claim that it works for small to medium training set sizes. The converse (as the reviewer proposes) is also true, the unlabeled set should also not be too large, as in that case we need more complicated assumptions about how to label the unlabeled data. We created Figure 1 in the appendix that illustrates these tradeoffs. For a small training set and small unlabeled set we can both fit the training set while also forcing diversity out-of-sample. For, very large unlabeled sets we underfit the training data. But this is natural as our assumptions are very simple. Conversely for very large training sets the unlabeled data are not useful (or might even hurt the final predictor). The PAC-Bayes bound captures this tradeoff with a complexity term.

We want to note that the above is not a significant problem for our algorithm in the following sense: It suffices to do a hyperparameter search over \beta using a validation set to eliminate the issue. If we include \beta = 0 in our grid, the hyperparameter search can default to standard ensembles in these pathological cases by choosing this \beta=0 value. In fact this is what happens when our ECE gains become 0 as the training set increases in Figure 2 of the main.

Reviewer comment (Experiments):
One major weakness of the paper is that their method only improves calibration error not accuracy but they have not compared to any other calibration technique like temperature sampling.

Author response:
Please note that in the revised version we include experiments with temperature scaling as well as additional datasets (SVHN and STL10) in the Appendix. In most cases nu-ensembles + temperature scaling results in the best calibration error, results for svhn and stl10 match those of CIFAR-10, CIFAR-100. Additionally, we would like to make two other points 1) our method increases the diversity of a deep ensemble and is not a regularization method. Thus in principle it can be combined with any other regularization method for individual networks (such as temperature scaling, mixup augmentation etc). 2) our method is applicable also in cases where it is difficult to modify the training objective or have access to the logits, such as for example when using standard libraries such as scikit-learn. This is in contrast to many calibration promoting approaches such as tempering.

Reviewer comment (Novelty):
The other issue is that the method appears very similar to the Agree to disagree work mentioned in the paper where they also use unlabeled data to maximize diversity and the idea seem incremental. Can the authors please explain in detail how exactly Agree to disagree maximizes diversity on the unlabeled set?

Author response:
**Our method has significant differences and improvements over the Agree-to-Disagree method. We describe these in the main text but we do a recap here**

The agree-to-disagree method trains an ensemble one member at a time. At step one they train a network to fit the labeled training set. At step two they train a second network that fits the labeled training set and which makes different predictions than the first network on an unlabeled set. The exact regularizer pushes the logits on unlabeled data to be different between the two networks. The method proceeds in this manner, adding one network to the ensemble at a time. By contrast nu-ensembles create K pseudolabeled datasets and then can train K ensemble members in parallel using the standard deep ensemble algorithm. Our method is much faster, simpler to implement and can be trivially parallelized. Also, agree to disagree was evaluated only on distribution shift datasets in the original paper. In our experiments we significantly outperform agree to disagree ensembles which undefit the data. Finally, the agree to disagree paper has only a very limited theoretical justification for their approach. By contrast we include a very detailed analysis based on a PAC-Bayes bound.

|metric | Agree to Disagree    |$\nu$-ensembles |
|-------| -------- | ------- |
|embarrassingly parallel| no | yes |
|best test performance| no  | yes    |
|theoretical justification | minimal | PAC-Bayes bound     |
|implementation| modified objective   | standard objective with randomized labels    |

Reviewer comment (Evaluation):
Another limitation is that this method only improves calibration in the small data regime.

Author response:
As we discuss in the main paper the small data regime occurs frequently in practice and is also a well studied setting in machine learning conferences and journals.

Reviewer comment (Experiments):
Another limitation is that there are only two datasets used in the paper - CIFAR-10 and CIFAR-100. It would be nice to have additional datasets.

Author response:
We include additional datasets stl10 and svhn in the revised appendix.

Reviewer comment (Theory):
The paper says that the labels for unlabeled data points are chosen without replacement. What happens if we sample with replacement? One should expect the same empirical results to hold but maybe the theoretical argument will not hold?

Author response:
We include in the revised text a detailed analysis of the setting where we sample with replacement (Proposition 2). Our theoretical analysis implies that for the same number of ensemble members we would obtain worse diversity and thus worse calibration. We include experiments in the Appendix that validate our theoretical predictions, that is, \nu-ensembles without replacement result in improved diversity and test calibration.

Reviewer comment (Presentation):
I understand the text written at bottom of the Figure 1 but I don’t understand the figure. What are the 3 columns in the figure?

Author response:
The three columns of the Figure represent 3 different ensembles of different sizes. Column 1 is an ensemble with of size 1, column 2 is an ensemble of size 2 and column 3 is an ensemble of size 4. The true label here (the cat) is y=3. Ensemble member 1 fits label y=1,  ensemble member 2 fits label y=4, ensemble member 3 fits label y=2, ensemble member 4 fits label y=3. Thus only ensemble member 4 has p(y=3|x,f) = 1 (It's the only one that fits the correct label).

Reviewer comment (Experiments):
There are other works which also use this idea of diversifying using unlabeled datapoint for other problems. For example, DIVERSIFY AND DISAMBIGUATE: OUT-OF-DISTRIBUTION ROBUSTNESS VIA DISAGREEMENT. Can the authors please compare to this work also?

Author response:
Diversity and disambiguate is a method that has been developed for out-of-distribution generalization and is not based on an ensemble. In general we note that the weight averaging line of papers has made it clear that deep ensembles do not necessarily generalize better to out-of-distribution data. Thus, the two methods are not directly comparable.