Paper: Input-gradient space particle inference for neural network ensembles (ICLR-2024)

Reviewer comment (Experiments):
The scale of experiments are quite small to show the effectiveness of FoRDE.

Author response:
In our paper, we performed experiments using ResNet18 and PreActResNet18 architectures with CIFAR-10/100 and TinyImageNet datasets. These experiments have been designed to show effectiveness of FoRDE on scales comparable to earlier works on particle inference for neural networks [1, 2, 3] and neural network ensembles [4, 5].

Reviewer comment (Evaluation):
The overall performance gain compared to other baselines looks quite marginal for the out-of-distribution datasets, especially for the TinyImageNet which is the largest dataset. And shows lower performance compared to the other baselines for the in-distribution datasets.

Author response:
While our method achieve slightly lower performance on clean data compared to the baselines, we argue that the performance gain on corrupted data is much more substantial and is not trivial to achieve since we need not assume any prior knowledge of these corruptions, and FoRDE consistently outperforms the baselines on corruptions across different datasets. For instance, Table 1 and 2 in the manuscript shows that  FoRDE-PCA achieves a +1.3% gain on CIFAR-100-C and +2.4% gain on CIFAR-10-C in accuracy compared to the second-best results.

Reviewer comment (Evaluation):
Having empirical or theoretical evidence to demonstrate the effectiveness of FoRDE in enhancing input gradient diversity would be beneficial.

Author response:
To show that FoRDE indeed produces ensembles with higher input gradient diversity among member models, which in turn leads to higher functional diversity than DE, we visualize the input gradient distance and epistemic uncertainty of FoRDE and DE in Fig. 7 of Section D.1 in the Appendix of the revised manuscript. To measure the differences between input gradients, we use *cosine distance*, defined as $1-\cos(\mathbf{u}, \mathbf{v})$ where $\cos(\mathbf{u}, \mathbf{v})$ is the cosine similarity between two vectors $\mathbf{u}$ and $\mathbf{v}$. To quantify functional diversity, we calculate the epistemic uncertainty using the formula in [6], similarly to the transfer learning experiments. Fig. 7 shows that FoRDE has higher gradient distances among members compared to DE, while also having higher epistemic uncertainty across all levels of corruption severity. Intuitively, as the test inputs become more corrupted, epistemic uncertainty of both FoRDE and DE increases, and the input gradients between member models become more dissimilar for both methods. These results suggest that there could be a connection between input gradient diversity and functional diversity in neural network ensembles.

Reviewer comment (Experiments):
Providing empirical results that illustrate how the improved input gradient diversity effectively changes into enhanced functional diversity in deep neural network scenarios would be valuable.

Author response:
To show that FoRDE indeed produces ensembles with higher input gradient diversity among member models, which in turn leads to higher functional diversity than DE, we visualize the input gradient distance and epistemic uncertainty of FoRDE and DE in Fig. 7 of Section D.1 in the Appendix of the revised manuscript. To measure the differences between input gradients, we use *cosine distance*, defined as $1-\cos(\mathbf{u}, \mathbf{v})$ where $\cos(\mathbf{u}, \mathbf{v})$ is the cosine similarity between two vectors $\mathbf{u}$ and $\mathbf{v}$. To quantify functional diversity, we calculate the epistemic uncertainty using the formula in [6], similarly to the transfer learning experiments. Fig. 7 shows that FoRDE has higher gradient distances among members compared to DE, while also having higher epistemic uncertainty across all levels of corruption severity. Intuitively, as the test inputs become more corrupted, epistemic uncertainty of both FoRDE and DE increases, and the input gradients between member models become more dissimilar for both methods. These results suggest that there could be a connection between input gradient diversity and functional diversity in neural network ensembles.

Reviewer comment (Experiments):
Additional hyperparameters for the kernel would be another burden for this method.

Author response:
We did acknowledge this difficulty in our manuscript, and proposed setting the lengthscales based on the PCA of the training data which makes the model more robust against natural corruptions. For the revised version of the manuscript, we also included Section D.4 in the Appendix which shows that by choosing a lengthscale setting which is a linear combination of the identity lengthscales and the PCA lengthscales, FoRDE can achieve good performance on both clean and corrupted data. This can serve as a guideline for choosing the lengthscales when applying FoRDE on new datasets.

Reviewer comment (Reproducibility):
It is recommended to include an ethics statement and a reproducibility statement right after the main paper.

Author response:
Thank you for your recommendation. We have included the ethics statement and reproducibility statement at the end of the revised manuscript, noting that these sections do not count towards the page limit according to the ICLR author guidelines.

Reviewer comment (Experiments):
One thing I often worry about is that the experiments are performed only in the vision domain, on over-hygenic datasets. While I don't want to discount the amount of work needed to extend to other domains, projects like [WILDS](https://wilds.stanford.edu/) make this easier, and build confidence that demonstrated success isn't due to some quirk of CIFAR datasets.

Author response:
Thank you for your suggestion. We were not aware of the WILDS project. We will certainly start using this project in our future works for more robust evaluations. Here, we followed previous works on Repulsive deep ensembles [1, 2] which focused mainly on benchmarking on vision datasets. We also provided experiment results on TinyImageNet, which is a bigger dataset with a higher number of classes (200 classes) compared to CIFAR.

Reviewer comment (Theory):
Again in section 3.3, is the median heuristic required? I’m a little unsure at the outset why this is the solution chosen over any others that would reduce the effect of the dominant eigenvalues.

Author response:
Since we calculate the kernelized repulsion term using the RBF kernel, we need to use the median heuristic so that the RBF kernel does not vanish during optimization. To understand this, we can look at Eq. (14) in the manuscript. If the eigenvalues are large, then the square distance inside the exponential function of the RBF kernel can reach a large value during optimization, and thus the RBF kernel will quickly converge to 0 and the repulsion term will also reach 0. Therefore, we divide the square distance inside the exponential with a scalar estimated via the median heuristic so that the repulsion term does not vanish during optimization. Prior works [5, 6] also employ the median heuristic to avoid this vanishing problem when using the RBF kernel in particle variational inference.

Reviewer comment (Experiments):
The authors seem to have recognized this aspect; "Hence, we believe that the optimal lengthscales for good performance on both clean and corrupted data lie somewhere between unit lengthscales (the identity) and using the inverse square root eigenvalues as lengthscales." For this paper to be considered complete, it should not just acknowledge such ideal lengthscales but also offer experimental evidence of their practical identification.

Author response:
We have included in Section D.4 in the Appendix of the revised manuscript anecdotal evidence that the ideal lengthscale setting lies somewhere between the identity lengthscales and the PCA lengthscales for the ResNet18 / CIFAR-100 experiments. Particularly, we train FoRDE under the lengthscale settings $\alpha * \mathrm{pca\\_lengthscales} + (1-\alpha) * \mathrm{identity\\_lengthscale}$, where we increase $\alpha$ from $0$ to $1$, and visualize the accuracy on both clean and corrupted data as a function of $\alpha$ in Fig. 11 in the Appendix.
Fig. 11 shows that as $\alpha$ increases, FoRDE becomes more robust against corruptions, while exhibiting small degradation in performance on clean data. With $\alpha \in [0.1, 0.4]$, FoRDE achieves higher accuracy than DE on both clean and corrupted data, with $\alpha=0.4$ produces the best result.

Reviewer comment (Theory):
The paper mentions the reasons for the ineffectiveness of weight-space repulsion: (1) "Typically repulsion is done in the weight space to capture different regions in the weight posterior. However, due to the over-parameterization of neural networks, weight-space repulsion suffers from redundancy." (2) "Weight-space repulsion is ineffective due to difficulties in comparing extremely high-dimensional weight vectors and the existence of weight symmetries (Fort et al., 2019; Entezari et al., 2022)." Could you provide a more detailed explanation of this?

Author response:
Performance of an ensemble partly relies on the functional diversity of its members. The purpose of weight-space repulsion in ensemble learning is to find weight particles that are different from each other in the hope that they represent different functions in the function space. However, the weight posterior of a neural network contains weight symmetries, meaning that two different weight vectors can represent the same function [1]. If we average the predictions from these two weight vectors, it would be equivalent to using the prediction from one of those weight vectors, and we refer to this problem as **redundancy**. Additionally, modern neural networks contains large amounts of parameters (more than tens of millions of parameters) arranged into hierarchical structures with various components (such as non-linear activations, convolution, normalization and pooling layers), making the geometry of the weight posterior highly complex [2]. This complexity makes it difficult to define a meaningful kernel to measure the similarity between two weight vectors, which is needed to calculate the kernelized repulsion term. Previous works [3, 4] used the RBF kernel for weight-space repulsion and observed no improvement in performance compared to plain Deep ensembles.


[1] C. M. Bishop, Pattern recognition and machine learning. Chapter 5.1. 2006

[2] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur, “The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks,” in ICLR, 2022.

[3] F. D’Angelo and V. Fortuin, “Repulsive deep ensembles are Bayesian,” In NeuRIPS, 2021.

[4] S. Yashima, T. Suzuki, K. Ishikawa, I. Sato, and R. Kawakami, “Feature Space Particle Inference for Neural Network Ensembles,” in ICML 2022.

Reviewer comment (Experiments):
The paper outlines the advantages of ensemble methods in four specific areas: (1) predictive performance, (2) uncertainty estimation, (3) robustness to adversarial attacks, and (4) corruptions. In the experimental results, it delves into (1) using ACC, (2) using NLL and ECE, and (4) using cA, cNLL, and cECE. Did you carry out any experiments regarding (3) by any chance? Considering that the current experimental results are somewhat lacking in (1) and (2), it might be worthwhile to focus more on (3) and (4).

Author response:
We would like to start by pointing out that the purpose of our work is to study input gradient repulsion as a viable way to improve functional diversity of ensembles, and thus we performed experiments to confirm that it does improve functional diversity (through visualization on small experiments in Figs. 2 and 3 and through the transfer learning experiments in Section 5.4).  Following prior works on repulsive ensembles [1, 2], we evaluated FoRDE on natural image corruptions, and studied how to choose the hyperparameters that allow FoRDE to be more robust against corruptions than the baselines. While these hyperparameters do slightly decrease performance of FoRDE on clean data, we believe that the improvement in performance on natural image corruptions is much more substantial and is not trivial to achieve since FoRDE has no prior knowledge of these corruptions. For instance, Table 1 and 2 in the manuscript shows that FoRDE-PCA achieves a +1.3% gain on CIFAR-100-C and +2.4% gain on CIFAR-10-C in accuracy compared to the second-best results. Thus, we did not perform any experiments or analysis on robustness of FoRDE to adversarial attacks.

While we agree that further evaluation of FoRDE on adversarial robustness would enhance our paper, we think it would require a separate in-depth study to analyze how FoRDE can be modified to achieve robustness against adversarial attacks, and how it fares against other methods specifically designed for this task. Including this study in our current manuscript would make it much longer than a typical conference paper.

Reviewer comment (Experiments):
FoRDE-PCA exhibits robust performance in addressing common corruptions (although it shows a minor decrease in its in-distribution performance). Hence, I would like to suggest providing more detailed experimental results concerning common corruptions, e.g., if it operates similarly to EmpCov (Izmailov et al., 2021), it is worth exploring whether the most beneficial corruption type aligns as well.

Author response:
Thank you for the suggestion and we have provided more detailed experimental results comparing FoRDE-PCA to the EmpCov prior in Section D.3 in the Appendix of the revised manuscript. Specifically, we visualize the accuracy of DE, DE-EmpCov, FoRDE-PCA and FoRDE-PCA-EmpCov on each corruption type in CIFAR-10-C for the ResNet18/CIFAR-10 experiments in Fig. 10, where FoRDE-PCA-EmpCov means that we trained FoRDE-PCA with the EmpCov prior. Fig. 10 shows that both FoRDE-PCA and EmpCov operate similarly on the majority of the corruption types, meaning that if DE-EmpCov is more or less robust than DE on a corruption type then so is FoRDE-PCA. The exceptions are the *blur* corruption types (*{motion, glass, zoom, defocus, gaussian}*-blur), where DE-EmpCov is less robust than vanilla DE while FoRDE-PCA exhibits better robustness than DE. Finally, by combining FoRDE-PCA and EmpCov prior together, we achieve the best robustness on average.

[1] F. D’Angelo and V. Fortuin, “Repulsive deep ensembles are Bayesian,” In NeuRIPS, 2021.

[2] S. Yashima, T. Suzuki, K. Ishikawa, I. Sato, and R. Kawakami, “Feature Space Particle Inference for Neural Network Ensembles,” in ICML 2022.

Reviewer comment (Theory):
Some transformations between theory in Section 3 and steps in Algorithm (in Appendix) are not obvious

Author response:
We have added clarifications in Section C.1 in the Appendix of the revised manuscript.

Reviewer comment (Theory):
Section 3.1 doesn't address that the target distribution \pi is not available, or am I missing something?

Author response:
You are correct. We have clarified that the target distribution $\pi$ is intractable (and hence approximated with the particle distribution in Eq. 6)  in Section 3.1 of the revised manuscript.

Reviewer comment (Presentation):
It would help to clear some confusion of how Algorithm comes in place if steps in Algorithm would be linked to equations in Section 3.

Author response:
We have added clarifications in Section C.1 in the Appendix of the revised manuscript.

Reviewer comment (Experiments):
Section 3.4. "However, in practice we found no performance degradation nor convergence issues in our experiments" - though the convergence issues can easily be observed, in order to see no performance degradation one would need to compare the performance with and without mini-batches. This experiment is not presented in the paper (including Appendix).

Author response:
We wholeheartedly agree with this assessment. We however cannot perform the full-batch experiments due to resource limitations, since performing forward-backward passes on the entire 50000 training samples of CIFAR requires a large amount of GPU memory. One could compromise by varying the minibatch size, however it has been observed that large-batch training produces models with lower generalization performance than small-batch training [1], meaning that we would likely observe lower performance on large batch sizes compared to small batch sizes due to this phenomenon. Therefore, we have changed this statement to “However, in practice we found no convergence issues in our experiments'' in Section 3.4 of the revised manuscript.

Reviewer comment (Experiments):
No reference for CINIC10 dataset

Author response:
We have added the reference in our updated manuscript.

Reviewer comment (Reproducibility):
What exact corruption has been used in reported corruption experiments?

Author response:
For evaluations on input perturbations, we use CIFAR-10/100-C and TINYIMAGENET-C provided by Hendrycks & Gimpel (2017), which are datasets of corrupted test images containing 19 image corruption types across 5 levels of severity, and we report the accuracy, NLL and ECE averaged over all corruption types and severity levels (denoted cA, cNLL and cECE in Tables 1–4).