Paper: Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR-2024)

Reviewer comment (Experiments):
the experiments are quite bare-bones for a BNN paper, there is no evaluation of predictive uncertainty besides calibration -- we don't need a Bayesian approach do well on this metric. I would either suggest adding e.g. a temperature scaling baseline applied to a sparse deterministic net or (preferably) the usual out-of-distribution and distribution shift benchmarks.

Author response:
## Experiments about OOD detection and distribution shift benchmark

We extended our experiments to two distribution shift benchmarks including both Out-of-Distribution (OOD) detection tasks and robustness benchmarks. The results demonstrate that our SSVI consistently improves over the strongest baseline, mBNN, significantly across all sparsity levels, metrics, and tasks. The results are presented both in the subsequent sections and in the updated Appendix D.

For OOD detection tasks, we computed the area under the receiver operating characteristic (AUROC) and the area under the precision-recall curve (AUPR) using our models, which were trained using SSVI on CIFAR-10/100 and tested on CIFAR-100/10 and SVHN, following the methodology of previous studies [1]. In this context, we employed the mBNN algorithm, the strongest previous work, as our baseline, ensuring equal sparsity for a fair and meaningful comparison.

|  **Train set**  |  |           |   CIFAR-10   |       |      | |          |   CIFAR-100   |       |      |
|:-----------:|:--------:|:---------:|:----:|:-----:|:----:|:---------:|:--------:|:----:|:-----:|:----:|
|   Test set  | | CIFAR-100 |      |  SVHN |      |  | CIFAR-10 |      |  SVHN |      |
|       Metric      |    Sparsity      |   AUROC   | AUPR | AUROC | AUPR |    Sparsity       |   AUROC  | AUPR | AUROC | AUPR |
|     mBNN    |   96.4%  |    0.84   | 0.81 |  0.89 | 0.94 |   87.8%   |   0.76   | 0.72 |  0.78 | 0.88 |
| SSVI (ours) |    95%   |    **0.86**   | **0.84** |  **0.93** | **0.96** |    **90%**    |   **0.80**   | **0.77** |  **0.85** | **0.91** |


We proceed to assess our model, trained using SSVI on CIFAR-10/100, on the respective corrupted versions, CIFAR-10/100-C. Both datasets encompass 15 distinct corruptions, each classified into 5 corruption levels. The outcomes are presented in the following table, illustrating curves for the average accuracy, loss, and Expected Calibration Error (ECE) across all 15 different corruptions for each level. 

| Dataset       | Model              | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-10-C    | mBNN (ACC)         | 0.811   | 0.731   | 0.651   | 0.565   | 0.448   |
|               | **SSVI (ACC)**     | **0.829**   | **0.754**   | **0.670**    | **0.585**   | **0.471**   |
|               | mBNN (Loss)        | 0.721   | 1.075   | 1.469   | 1.931   | 2.659   |
|               | **SSVI (Loss)**    | **0.668**   | **1.001**   | **1.384**   | **1.810**    | **2.408**   |
|               | mBNN (ECE Loss)    | 0.066   | 0.092   | 0.110    | 0.129   | 0.151   |
|               | **SSVI (ECE Loss)**| **0.052**   | **0.078**   | **0.099**   | **0.121**   | **0.145**   |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-100-C   | mBNN (ACC)         | 0.583   | 0.499   | 0.437   | 0.369   | 0.279   |
|               | **SSVI (ACC)**     | **0.611**   | **0.522**   | **0.455**   | **0.388**   | **0.293**   |
|               | mBNN (Loss)        | 2.062   | 2.69    | 3.320    | 4.045   | 5.156   |
|               | **SSVI (Loss)**    | **1.725**   | **2.247**   | **2.821**   | **3.351**   | **4.090**    |
|               | mBNN (ECE Loss)    | 0.015   | 0.023   | 0.026   | 0.033   | 0.039   |
|               | **SSVI (ECE Loss)**| **0.012**   | **0.019**   | **0.024**   | **0.029**   | **0.036**   |
|---------------|--------------------|---------|---------|---------|---------|---------|


## Calibration metric

While we understand your perspective, we respectfully disagree with the notion that calibration is not valuable. Calibration is widely recognized as a crucial metric in Bayesian deep learning literature, particularly as an indicator of a model's uncertainty capabilities. This perspective is not only supported in the paper you referred to [1] but also in our most closely related baseline [8]. Additionally, calibration is considered a significant metric in various studies investigating the posterior of Bayesian models, as seen in references [9, 10].

Reviewer comment (Experiments):
primarily testing at a single sparsity level as in Table 2 also seems a bit limited to me. In my view, there are broadly two possible goals when using sparsity: opitimizing sparsity at a given performance level, e.g. close to optimal, or optimizing performance at a given sparsity level. I would have liked to see more figures in the style of Figure 2 left and Figure 3 to cover both of these settings also for the baselines.

Author response:
## Results of more sparsity levels for baseline algorithms

For our SSVI, we present outcomes at various sparsity levels in Figure 2, as you noticed. However, this does not extend to the baseline algorithms. Actually, all the baseline algorithms utilize implicit regularization to steer the model towards a level of sparsity that is not predefined, leading to a process that is frequently unstable and heavily dependent on precise hyperparameter tuning to achieve a satisfactory level of sparsity. Therefore, we first train the baselines following the settings in their original papers, then adopt the obtained sparsity level in our SSVI training to ensure a fair comparison. 

We want to highlight that this is one of the advantages of our SSVI —- the sparsity can be flexibly assigned before training. We demonstrate the efficacy of SSVI across a range of sparsity levels and also explore its performance with various optimization schedules within the same sparsity level, as shown in Figures 2 and 3.

Reviewer comment (Experiments):
I would have liked to see a bit more in-depth investigation of the pruning criteria, e.g. a plot of Spearman correlations between the preferred score and the others throughout training or a correlation matrix at various stages (say beginning, halfway through and end of training). I must say that I am not overly convinced that they matter too much, the variation of accuracy in Fig 2 seems to be only about 0.5% (although see questions). So I think it might be worth saving the page discussing the criteria in fewer of more thorough experiments.

Author response:
## Influence of pruning criteria

Indeed, the choice of pruning criteria is crucial. It is important to note that global metrics like Spearman correlations may not be the most suitable metric for assessing the differences between various scores in this context. Since we utilize the score solely for selecting the top-K weights without incorporating it into the training process, and considering that K is significantly smaller than the total number of weights, a high Spearman score may not accurately reflect the divergence in the chosen weights.

Therefore, we have opted for the ratio of intersection over union (IoU). Each score selects the top-K weights for a fixed number K, and we calculate the intersection over union for all pairs among our five different scores. 

In the following, we present the Intersection over Union (IoU) matrices for the five pruning criteria at the initial, middle, and final stages of training, respectively. Additionally, we include in the supplementary material the average of all 10 IoU values for different pairs across the training steps. The findings reveal that while the criteria show high similarity (reflected in high IoU scores) in the initial phase, they diverge as training progresses, potentially due to more precise uncertainty information and consequently more appropriate criteria. It is also notable that the traditional criteria, specifically $|\mu|$ and $\mathrm{SNR}{q_{\phi}}(\theta)$, exhibit a persistent similarity during the entire training period. On the other hand, the three novel criteria we introduced: $\mathbb{E}{q_\phi}|\theta|$, $\mathrm{SNR}{q_{\phi}}(|\theta|)$, and $\mathbb{E}{q_\phi}e^{\lambda|\theta|}$, show a marked deviation from these conventional metrics, emphasizing the innovative aspect of our methodology.


—-----------------IoU matrix at the beginning of training—-----------------

|                                          | $\|\mu\|$ | $SNR_{q_{\phi}}(\theta)$ | $\mathbb{E}_{q\phi}\|\theta\|$ | $SNR_{q_{\phi}}(\|\theta\|)$ | $\mathbb{E}_{q\phi}e^{\lambda\|\theta\|}$ |
|------------------------------------------|-----------|--------------------------|--------------------------------|------------------------------|------------------------------------------|
| $\|\mu\|$                                | 1.0       | 0.9995                   | 0.9122                         | 0.9981                       | 0.9122                                   |
| $SNR_{q_{\phi}}(\theta)$                 | 0.9995    | 1.0                      | 0.9118                         | 0.9985                       | 0.9118                                   |
| $\mathbb{E}_{q\phi}\|\theta\|$            | 0.9122    | 0.9118                   | 1.0                            | 0.9118                       | 0.9996                                   |
| $SNR_{q_{\phi}}(\|\theta\|)$             | 0.9981    | 0.9985                   | 0.9118                         | 1.0                          | 0.9118                                   |
| $\mathbb{E}_{q\phi}e^{\lambda\|\theta\|}$ | 0.9122    | 0.9118                   | 0.9996                         | 0.9118                       | 1.0                                      |

Reviewer comment (Novelty):
the paper makes some rather inaccurate claims vs the existing literature. In particular, it is not the first paper introducing a "fully sparse BNN framework that maintains a consistently sparse Bayesian model through- out the training and inference", this statement also applies to the (Ritter et al., 2021) paper, which is incorrectly cited as a post-hoc pruning paper (the paper does use post-hoc pruning as an optional step to further increase sparsity, but the core low-rank parameterization is maintained throughout training). This doesn't affect the contribution of course, but prior work needs to be contextualized correctly.

Author response:
## Comparison with Ritter et al. 2021

Although (Ritter et al., 2021) achieve parameter reduction by storing the inducing weights $U$ that have much lower dimensions, they still require sampling full-dimension original weights $W$ in each forward pass, as stated in Section 3.2 of their paper. As a result, their computation costs in both training and inference remain similar or even larger than the dense BNNs. In contrast, our method directly enforces sparsity in $W$, which has a substantial reduction of computation costs theoretically. Besides, the sampling rule in (Ritter et al., 2021) requires a Gaussian parameterization, whereas our method is more flexible and can generalize beyond Gaussian parametrization. Yet, we still thank you for the comment and will change the description of (Ritter et al., 2021) in the revision.

Reviewer comment (Experiments):
I don't really see the need to make such claims in the first place, it is not obvious that sparsity in training is desirable. Of course it may be the case that a larger network that would not fit into memory without sparsity performs better, but then this needs to be demonstrated (or like-wise any hypothetical training speed increases resulting from a reduced number of FLOPs - in the age of parallelized computation, that is a mostly meaningless metric if it cannot be shown that a practical implementation can lead to actual cost savings).

Author response:
## Why do we need sparsity in training

We note that the concern raised is not unique to our work on sparse Bayesian Neural Networks (BNNs) but extends to the broader field of sparse neural networks. The advantages of having sparsity in training have been addressed in previous discussions, such as [5]. Here we formulate the answer about why we need sparsity in training from two directions: speedup and performance.

**Real-time speedup:**

First, we want to clarify that sparsity in training can lead to actual, real-time performance improvements on modern hardware, beyond just theoretical advantages suggested by FLOPs reduction. As shown by [5], *“Another notable success was recently demonstrated by DeepSparse [11, 12], which successfully deploys large-scale BERT-level sparse models on modern Intel CPUs, obtaining 10× model size compression with < 1% accuracy drop, 10× CPU-inference speedup with < 2% drop, and 29× CPU-inference speedup with < 7.5% drop.”* Additionally, there are considerable efforts, such as those by NVIDIA, to develop advanced GPU kernels like Sputnik [13] and the NVIDIA Ampere Architecture [14], which *“built the momentum to better support finer-grained sparsity”* and thereby increase real-time speedup. We will explore how to achieve real wall-time speedup with the specific hardware in our future work.

Reviewer comment (Writing):
the abstract is simultaneously wordy and vague. I did not know what the paper was doing specifically after reading it, even though it really isn't hard to describe the method in 1 or 2 sentences. I would say that the low-rank/basis terminology led me in the wrong direction of thinking and a pruning-based description would have been clearer, but this may of course differ for readers with a different background.

Author response:
## Abstract presentation

We would like to clarify that our paper is not only about pruning. Our paper is about how to find an optimal sparse subspace to do variational inference. To solve this problem, we optimize the sparse subspace and the corresponding variational model together. In this context, a removal(pruning)-based algorithm emerges as a natural solution within an alternative optimization schedule. We appreciate your input and will refine the abstract to provide a clearer description in the revised draft.

Reviewer comment (Reproducibility):
how are the mean values initialized for weights that are added to the active set? I assume 0? Do you reuse the old value if a weight had been temporarily removed?

Author response:
## Initialization of weight mean

For initializing the mean values of weights added to the active set, we use a zero initialization.

Reviewer comment (Presentation):
I'm not sure I follow the discussion of Figure 2 right. For the initialization of the variance of a mean-field Gaussian, identical considerations apply as for the initialization of weights in deterministic networks, so loosely speaking we want to scale the sum of variance of the means and the initial value of the variance parameter inversely with the number of units to avoid the activations diverging with increasing depth. So to me it seems natural, that dense VI would reach this critical threshold before a pruned variance, as the latter is simply removing terms from a positive sum and thus decreasing variance. Am I missing something/misinterpreting the figure?

Author response:
## About Figure 2 right

Your interpretation is essentially accurate. The phenomenon you described underscores a key advantage of our SSVI over conventional VI in Bayesian Neural Networks (BNNs).
In Figure 2, we empirically verified this interpretation, as you correctly noted. 
It is important to recognize that training BNNs is more sensitive to initialization than training conventional neural networks. This makes our comparison between SSVI and VI particularly revealing. Consequently, the findings presented in Figure 2 right are neither trivial nor obvious and should be regarded as substantial.

Reviewer comment (Novelty):
A bit of context in terms of BNNs is missing. The review of the existing methods solely (though understandably) focuses on sparse models, leaving behind other approaches of making efficient Bayesian inference for NNs (e.g., Cobb, A.D. and Jalaian, B., 2021. Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting. In Uncertainty in Artificial Intelligence (pp. 675-685). PMLR).

Author response:
In the introduction section of our revised paper, we have incorporated the work you referenced, along with other pertinent studies. Specifically, we have expanded our discussion to include additional works on stochastic variational inference [2,3,4] and Monte Carlo methods [5,6,7], beyond those already cited in the current version.

Reviewer comment (Experiments):
The empirical evaluation of the model, though done rather thoroughly, is only on 2 medium-size/complexity datasets from the same domain on 1 architecture. It would be interesting to see more experiments.

Author response:
We extended our experiments to two distribution shift benchmarks including both Out-of-Distribution (OOD) detection tasks and robustness benchmarks. The results demonstrate that our SSVI consistently improves over the strongest baseline, mBNN, significantly across all sparsity levels, metrics, and tasks. The results are presented both in the subsequent sections and in the updated Appendix D.

For OOD detection tasks, we computed the area under the receiver operating characteristic (AUROC) and the area under the precision-recall curve (AUPR) using our models, which were trained using SSVI on CIFAR-10/100 and tested on CIFAR-100/10 and SVHN, following the methodology of previous studies [1]. In this context, we employed the mBNN algorithm, the strongest previous work, as our baseline, ensuring equal sparsity for a fair and meaningful comparison.

|  **Train set**  |  |           |   CIFAR-10   |       |      | |          |   CIFAR-100   |       |      |
|:-----------:|:--------:|:---------:|:----:|:-----:|:----:|:---------:|:--------:|:----:|:-----:|:----:|
|   Test set  | | CIFAR-100 |      |  SVHN |      |  | CIFAR-10 |      |  SVHN |      |
|       Metric      |    Sparsity      |   AUROC   | AUPR | AUROC | AUPR |    Sparsity       |   AUROC  | AUPR | AUROC | AUPR |
|     mBNN    |   96.4%  |    0.84   | 0.81 |  0.89 | 0.94 |   87.8%   |   0.76   | 0.72 |  0.78 | 0.88 |
| SSVI (ours) |    95%   |    **0.86**   | **0.84** |  **0.93** | **0.96** |    **90%**    |   **0.80**   | **0.77** |  **0.85** | **0.91** |


We proceed to assess our model, trained using SSVI on CIFAR-10/100, on the respective corrupted versions, CIFAR-10/100-C. Both datasets encompass 15 distinct corruptions, each classified into 5 corruption levels. The outcomes are presented in the following table, illustrating curves for the average accuracy, loss, and Expected Calibration Error (ECE) across all 15 different corruptions for each level. 

| Dataset       | Model              | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-10-C    | mBNN (ACC)         | 0.811   | 0.731   | 0.651   | 0.565   | 0.448   |
|               | **SSVI (ACC)**     | **0.829**   | **0.754**   | **0.670**    | **0.585**   | **0.471**   |
|               | mBNN (Loss)        | 0.721   | 1.075   | 1.469   | 1.931   | 2.659   |
|               | **SSVI (Loss)**    | **0.668**   | **1.001**   | **1.384**   | **1.810**    | **2.408**   |
|               | mBNN (ECE Loss)    | 0.066   | 0.092   | 0.110    | 0.129   | 0.151   |
|               | **SSVI (ECE Loss)**| **0.052**   | **0.078**   | **0.099**   | **0.121**   | **0.145**   |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-100-C   | mBNN (ACC)         | 0.583   | 0.499   | 0.437   | 0.369   | 0.279   |
|               | **SSVI (ACC)**     | **0.611**   | **0.522**   | **0.455**   | **0.388**   | **0.293**   |
|               | mBNN (Loss)        | 2.062   | 2.69    | 3.320    | 4.045   | 5.156   |
|               | **SSVI (Loss)**    | **1.725**   | **2.247**   | **2.821**   | **3.351**   | **4.090**    |
|               | mBNN (ECE Loss)    | 0.015   | 0.023   | 0.026   | 0.033   | 0.039   |
|               | **SSVI (ECE Loss)**| **0.012**   | **0.019**   | **0.024**   | **0.029**   | **0.036**   |
|---------------|--------------------|---------|---------|---------|---------|---------|

We agree that applying SSVI to more ML tasks, beyond standard models and applications, is an interesting question and we are happy to explore it further in our future work.

Reviewer comment (Evaluation):
Abstract: The claim of 20x compression and 20x FLOPs reduction is unclear — in comparison to what?

Author response:
The "20x compression, 20x FLOPs reduction" is benchmarked against the densely trained VI.

Reviewer comment (Writing):
Abstract: The phrase “surpassing VI-trained dense BNNs” is unclear — in terms of what?

Author response:
Our approach excels over VI-trained dense BNNs in terms of both accuracy and uncertainty metrics, such as the Expected Calibration Error (ECE).

Reviewer comment (Presentation):
Figure 1: Names of baselines are not defined.

Author response:
The experimental section contains references and descriptions of the baselines.

Reviewer comment (Writing):
Equation (3): It is better to include explicitly what is the operator in the last 2 equations.

Author response:
The operator is element-wise matrix multiplication.

Reviewer comment (Writing):
Around Equation (4): It is unclear what p, B, q, and W are.

Author response:
$B$ is batch size, $p,q$ are the input and output dimensions, $W$ is the weight matrix.

Reviewer comment (Writing):
Equation (5): It is unclear what \( l \) is.

Author response:
$l$ is the loss function received from the backward process.

Reviewer comment (Writing):
It is unclear regarding Equation (11) and Equation (12) (from the appendix) which one is used at the end.

Author response:
We use equation (12) in our SSVI.

Reviewer comment (Experiments):
The empirical results seem insufficient. The authors haven't mentioned the model architecture used in their experiments. Also, to really emphasize the usefulness of the proposed approach, it would be important to have empirical results on an expanded set of datasets + models + tasks (including UQ tasks such as OOD detection, misclassification detection, etc.)

Author response:
## The authors haven't mentioned the model architecture used in their experiments.

We have mentioned the model architecture used in experiments, ResNet-18, in the first line of Section 4.1 on page 7. This model architecture is standard and has been widely used in the sparse Bayesian Neural Network (BNN) literature such as mBNN (Kong et al., 2023) and AEB (Deng et al., 2019). In fact, many papers on sparse BNNs (including Kong et al., 2023 and Deng et al., 2019) used ResNet as the **only** model architecture for large-scale Bayesian deep learning experiments. Therefore, we also chose ResNet to ensure a fair comparison.

## It would be important to have empirical results on an expanded set of datasets + models + tasks

We extended our experiments to two distribution shift benchmarks including both Out-of-Distribution (OOD) detection tasks and robustness benchmarks. The results demonstrate that our SSVI consistently improves over the strongest baseline, mBNN, significantly across all sparsity levels, metrics, and tasks. The results are presented both in the subsequent sections and in the updated Appendix D.

For OOD detection tasks, we computed the area under the receiver operating characteristic (AUROC) and the area under the precision-recall curve (AUPR) using our models, which were trained using SSVI on CIFAR-10/100 and tested on CIFAR-100/10 and SVHN, following the methodology of previous studies [1]. In this context, we employed the mBNN algorithm, the strongest previous work, as our baseline, ensuring equal sparsity for a fair and meaningful comparison.

|  **Train set**  |  |           |   CIFAR-10   |       |      | |          |   CIFAR-100   |       |      |
|:-----------:|:--------:|:---------:|:----:|:-----:|:----:|:---------:|:--------:|:----:|:-----:|:----:|
|   Test set  | | CIFAR-100 |      |  SVHN |      |  | CIFAR-10 |      |  SVHN |      |
|       Metric      |    Sparsity      |   AUROC   | AUPR | AUROC | AUPR |    Sparsity       |   AUROC  | AUPR | AUROC | AUPR |
|     mBNN    |   96.4%  |    0.84   | 0.81 |  0.89 | 0.94 |   87.8%   |   0.76   | 0.72 |  0.78 | 0.88 |
| SSVI (ours) |    95%   |    **0.86**   | **0.84** |  **0.93** | **0.96** |    **90%**    |   **0.80**   | **0.77** |  **0.85** | **0.91** |


We proceed to assess our model, trained using SSVI on CIFAR-10/100, on the respective corrupted versions, CIFAR-10/100-C. Both datasets encompass 15 distinct corruptions, each classified into 5 corruption levels. The outcomes are presented in the following table, illustrating curves for the average accuracy, loss, and Expected Calibration Error (ECE) across all 15 different corruptions for each level. 

| Dataset       | Model              | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-10-C    | mBNN (ACC)         | 0.811   | 0.731   | 0.651   | 0.565   | 0.448   |
|               | **SSVI (ACC)**     | **0.829**   | **0.754**   | **0.670**    | **0.585**   | **0.471**   |
|               | mBNN (Loss)        | 0.721   | 1.075   | 1.469   | 1.931   | 2.659   |
|               | **SSVI (Loss)**    | **0.668**   | **1.001**   | **1.384**   | **1.810**    | **2.408**   |
|               | mBNN (ECE Loss)    | 0.066   | 0.092   | 0.110    | 0.129   | 0.151   |
|               | **SSVI (ECE Loss)**| **0.052**   | **0.078**   | **0.099**   | **0.121**   | **0.145**   |
|---------------|--------------------|---------|---------|---------|---------|---------|
| CIFAR-100-C   | mBNN (ACC)         | 0.583   | 0.499   | 0.437   | 0.369   | 0.279   |
|               | **SSVI (ACC)**     | **0.611**   | **0.522**   | **0.455**   | **0.388**   | **0.293**   |
|               | mBNN (Loss)        | 2.062   | 2.69    | 3.320    | 4.045   | 5.156   |
|               | **SSVI (Loss)**    | **1.725**   | **2.247**   | **2.821**   | **3.351**   | **4.090**    |
|               | mBNN (ECE Loss)    | 0.015   | 0.023   | 0.026   | 0.033   | 0.039   |
|               | **SSVI (ECE Loss)**| **0.012**   | **0.019**   | **0.024**   | **0.029**   | **0.036**   |
|---------------|--------------------|---------|---------|---------|---------|---------|

Reviewer comment (Presentation):
I think the presentation can be improved. The authors introduce complex notations, but after reading through section 3 twice, I believe that it can be greatly simplified. For e.g., where exactly is eq 2 reconciled in the algorithm? Based on my reading, the paper would have been fine to totally exclude eq 2.

Author response:
## Notations and Equation 2

Our intention behind introducing all notations is to provide a clearer illustration of both our intuition and algorithm. Regarding Equation 2, we want to emphasize that it serves as the objective function (loss function) for our Sparse Subspace Variational Inference (SSVI). This is similar to Equation 1, which is the objective function for standard Variational Inference (VI). Equation 2 is a slight modification of Equation 1, representing our goal to constrain the variational space to a sparse subspace. The proposed algorithm SSVI is then developed to solve the optimization problem stated in Equation 2.

Moreover, Equation 2 serves as the general form for sparse subspace VI. If we choose the common Gaussian as the variational distribution, it will result in a special case as shown in Equation 3. It is worth noting that other distribution families are possible but they may introduce unnecessary complexity and lead to alternative forms based on Equation 2. 

In summary, Equation 2 is important to our framework, providing the foundation for the problem this paper tackles and guiding subsequent developments in our approach. We will revise this point in the final version to improve the clarity of the paper.

Reviewer comment (Reproducibility):
What's the model architecture used in experiments?

Author response:
## The authors haven't mentioned the model architecture used in their experiments.

We have mentioned the model architecture used in experiments, ResNet-18, in the first line of Section 4.1 on page 7. This model architecture is standard and has been widely used in the sparse Bayesian Neural Network (BNN) literature such as mBNN (Kong et al., 2023) and AEB (Deng et al., 2019). In fact, many papers on sparse BNNs (including Kong et al., 2023 and Deng et al., 2019) used ResNet as the **only** model architecture for large-scale Bayesian deep learning experiments. Therefore, we also chose ResNet to ensure a fair comparison.

Reviewer comment (Novelty):
Apart from implementation, can the authors highlight the diff between their work and that of Vadera et al.?

Author response:
## About Vadera et al.

We have already added this paper Vadera et al. to our introduction section. We also agree that it’s an interesting direction to extend the method from MCMC to VI as outlined in Vadera et al., and evaluate the sparse structure of our SSVI in comparison to prior works using approaches from Vadera et al. However, this is beyond the scope of our current work. We will investigate this systematically in our future work.

Reviewer comment (Theory):
How would you extend this to other approximate Bayesian inference techniques, apart from mean-field VI?

Author response:
## Extention to other approximate Bayesian inference techniques

Bayesian neural networks based on other approximate Bayesian inference techniques, such as non-mean-field VI and MCMC, often involve distinct assumptions and approximations, such as those outlined in [4,5], which are beyond the scope of our current work as a conference submission. However, we appreciate the significance of this direction and plan to explore it in our future work.

Reviewer comment (Evaluation):
It'll be useful if the authors can demonstrate the quality of uncertainty metrics coming out of the BNNs in their approach on downstream tasks. See [3] for example.

Author response:
## Benefits of uncertainty metric in downstream tasks

We believe that the additional experiments on out-of-distribution (OOD) detection and distribution shift tasks, as outlined above and detailed in Appendix D.1, effectively address this question. The experiments we have are similar to those in [3]. Our SSVI surpasses all baseline models in terms of ECE for standard tasks, AUROC and AUPR for OOD tasks. This improvement in uncertainty metrics contributes to the enhanced accuracy on corrupted datasets, as evidenced by our results with CIFAR-10/100-C.