From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: All four reviewers unanimously recommend accepting the paper (6-8-8-8). They agree that the paper successfully connects several interesting ideas to produce a scalable approximation to the Laplace method. The experiments in the paper show results for inference settings that existing methods cannot handle, demonstrating the proposed method's usefulness. The reviewers also agree that the paper is well-written and somewhat easy to understand. The authors have made clarifications and added further experiments during the revision process.
justification_for_why_not_higher_score: The paper is well-written and the experiments are well done. It uses a combination of clever tricks to scale the Laplace method, which may not be groundbreaking individually but are effective when combined. For this reason, I do not recommend a higher score.
justification_for_why_not_lower_score: Experiments show convincing results on previously computationally intractable inference problems. 
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper4180/Authors
title: Summary of updates to the draft
comment: Thank you, AC, for starting the discussion.

Following the reviewers' suggestions, here is a list of the updates we made to our draft:

1. We have added an explanation of how to apply our closed-form precision update to diagonal but non-isotropic Gaussian priors in Appendix B.4, and referenced it from where it is used in the main text.

2. We have clarified the definition of $\bar \theta$ in equation (15).

3. We have removed all uses of the acronym "DIP".

4. We have computed the "credible interval size vs coverage" predictive distribution calibration metric for our two major experiments and plotted them in Appendix H of our revised draft. We reference the plots from the main text in Section 5. We will try to have these plots feature in the main text in the camera-ready version of the paper since we think they are very illustrative.


We are eager to hear the reviewers' thoughts on our rebuttal.

Best,

The authors.

From: ICLR.cc/2023/Conference/Paper4180/Area_Chair_yEGU
title: Please engage before the author-reviewer discussion closes
comment: Dear authors and reviewers,

The first phase of the discussion period is about to close on November 18.

For authors, please make sure to submit your rebuttal by the deadline. Leave some time for the reviewers to read it and respond while you are still allowed to further engage with them. Interactions between authors and reviewers are very important for the quality of the review process, so please make sure to engage.

For reviewers, please try to acknowledge and respond to the authors' rebuttal while the discussion period is still open for them to further interact with you. 

Thank you for your participation in the review process!

Best,
The AC

From: ICLR.cc/2023/Conference/Paper4180/Authors
title: Thank you for the insightful feedback!
comment: We thank the reviewer for taking the time to read our paper, and for their feedback! 

**Presentation**

> For example, the rightmost plot in Figure 3 shows different layer-wise precisions, but the exposition only discusses learning a single precision (controlled by $\alpha$). I can imagine how it is straightforward to extend to different layer-wise αs, but that should be clarified in the main text.

Thanks for the catch! We have added an explanation of how to apply our closed form precision update to diagonal but non isotropic Gaussian priors in Appendix B.4, and referenced it from where it is used in the main text.

> In Section 4, $\bar{w}$ is used without definition (presumably the optimized weights)

You are correct about $\bar w$ referring to the optimised weights. However, this variable is defined in the second line of section 4, right above equation 13. 

> Then, $\bar{\theta}$ is used (e.g., in equation (15)) without definition.

In this section, $\bar \theta$ follows the same definition from section 2. We can see how this could cause confusion and have clarified the definition in equation (15) of the updated draft.

> Similarly "DIP" is used in that Figure and Table, but never defined.

The use of the acronym DIP, which refers to Deep Image Prior (the technique used to train the U-net in our tomographic reconstruction tasks), was unintentional and it has been removed from the updated draft. 

**Evaluation Metrics**

> The reported metrics are all good, but one of the most interpretable measures of uncertainty quantification is simply coverage, and it would be nice to report empirical coverage at different posterior credible levels on some held out data.

We agree about the relevance of width-coverage as an easily interpretable metric of uncertainty calibration. *Please find these metrics in Appendix H of our revised draft and referenced from the main text in Section 5.*

However, coverage/calibration metrics are difficult to generalise to the setting of multiple output dimensions—whereas test-LL generalises naturally—and the multiple output setting is core to our work. For this reason, we chose test-LL as the main metric to report in the body of the paper.

> I found the results in Figure 5 difficult to interpret. Namely, it is not obvious to me how any Bayesian uncertainty quantification should perform when some datapoints are corrupted. Would the true posterior (perhaps with a learned prior) outperform approximate posteriors in terms of log likelihood on test data? I'm not sure that that is true, and so I'm not sure what the relative ordering of the different uncertainty quantifications methods in Figure 5 says about their "quality".

We agree that robustness against datapoint corruption is a misspecified problem—and that we ought not to expect the true posterior (under the misspecified assumptions) to outperform all approximate posteriors. In particular, if we were to write out a correct (well-specified) model for the corruption, we expect the posterior under that model to outperform the 'true posterior' under misspecification. However, robustness under corruption, a task motivated by real-world applications, is also perhaps the most common benchmark within the NN uncertainty quantification literature (see for example references [1 - 6]). We believe our readers (and likely other reviewers) will wish to see this experiment included.

Please let us know if you have any additional suggestions on our draft!

[1] https://arxiv.org/abs/1906.02530

[2] https://arxiv.org/abs/2006.08437

[3] https://arxiv.org/abs/2006.10108

[4] https://arxiv.org/abs/2005.07186

[5] https://arxiv.org/abs/2010.14689

[6] https://arxiv.org/abs/2207.07411

From: ICLR.cc/2023/Conference/Paper4180/Authors
title: Thank you for your review, and insightful comments!
comment: We thank the reviewer for their time reading our paper, and for their feedback!

**Novelty**

Our paper starts (in section 2) by describing the multi-output linear Gaussian regression model, and with an EM algorithm for inference in this setting. However, as the ultimate paragraph of this section outlines, this "simple EM" algorithm does not scale to large-data and large-model settings due to both the E and M step incurring a cubic cost.

The core contribution of our paper (as outlined in section 1) is the introduction of a stochastic EM algorithm that does scale to millions of parameters and millions of observations, where both the E-step and the M-step are novel. In particular:
1. We use a stochastic version of the E-step, where sampling is performed using a novel objective. We show empirically that the novel objective yields an improvement of magnitude roughly equivalent to increasing batch size by a factor of 16 (Figure 2) compared to existing objectives. The E-step is thus novel. 
2. We propose a sample-based M step. Specifically, we derive a closed-form fixed-point update for the regularisation parameter being optimised. This 'trick' is vital for the method’s performance, and is, in our understanding, novel. 

The reviewer doubts the novelty of the paper and states that the paper would be more novel if the EM algorithm was improved. As evidenced by our experimental results and our theory, the novel E and M steps from points 1 and 2 above do indeed result in a *much* improved EM algorithm for the multi-output Gaussian regression model. 

Furthermore, in the context of linearised Laplace, the specific forms of the E and M steps we choose are vital. This is because the sampling-based estimators we propose require only individual vector-Jacobian products—cost constant in the output dimension—as opposed to existing methods which require instantiating the full Jacobian matrix explicitly, at a cost of output-dimension-many vector-Jacobian products—cost linear in the output dimension. 

**Scale**

The reviewer asks for experiments with multiple datasets and larger models. We would like to again highlight that our work:
1. Applies linearised Laplace to the largest dataset tackled by any previous work on linearised Laplace (CIFAR100). Previous approaches have only attempted a dataset with lower output dim (m=10 for CIFAR10), as without our sampling-based inference which depends only on vector-Jacobian products, it is computationally intractable to even assemble the Jacobian.
2. Applies full-covariance linearised Laplace on a model (ResNet18) much larger than any in previous literature. 
In the context of linearised Laplace literature, this pushes the boundaries on both fronts.

To illustrate the complexity of linearised Laplace in the above setting: with ResNet18 on CIFAR100, we have $d \approx 11\mathrm{e}{+6}$ parameters, $n=5\mathrm{e}{+4}$ data points, and $m=100$ classes. This gives a Hessian that has either $1.21\mathrm{e}{+14}$  ($d^2$) or $2.5\mathrm{e}{+13}$ ($(nm)^2$) entries; stored as float32, these are 0.5 petabytes and 0.1 petabytes respectively. Inverting these matrices and computing their determinants present further challenges.

**Overall**

We hope that in light of this reply, the reviewer might reconsider their concern.

From: ICLR.cc/2023/Conference/Paper4180/Authors
title: Thank you for the review and comments!
comment: We thank the reviewer for their time reading our paper and for their kind words. 

**Downstream evaluation**

A downstream application in the style of a bandit or reinforcement algorithm using our method would indeed be interesting, particularly the E-step. We considered doing this, but we were worried that introducing the relevant concepts would make the paper more crowded, less focused and overall harder to read.

From: ICLR.cc/2023/Conference/Paper4180/Authors
title: Thank you for the insightful review!
comment: We thank the reviewer for their time reading our paper and for their kind words.

From: ICLR.cc/2023/Conference/Paper4180/Reviewer_d9Fu
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The computational cost associated with Bayesian linear models constrains this method’s application to small networks, small output spaces and small datasets. Moreover, the method is sensitive to the choice of regularisation strength for the linear model. This paper overcomes these limitations by introducing a scalable sample-based Bayesian inference method for conjugate Gaussian multi-output linear models, together with a matching method for hyperparameter selection. The experimental results demonstrate the contributions. 
strength_and_weaknesses: Strengths:
The paper is well-written.
The approach is easy to understand and presented well in the paper. 
The authors applied their method to some tasks to demonstrate the validity of the proposed method. 
The theoretical analysis is strong.

Weaknesses:
1. Novelty is somehow limited. The main contribution is that the authors simply apply EM algorithm to scale inference and hyperparameter selection in Gaussian linear regression. It would be more novel if the EM algorithm was improved. 
2. Experiments across multiple datasets and with larger models would make the results more convincing. 
3. The ResNet-18 is also a very small model. Why not try the method on larger models? Is it because of the difficulty to implement?

clarity,_quality,_novelty_and_reproducibility: 1. Novelty is somehow limited. The main contribution is that the authors simply apply EM algorithm to scale inference and hyperparameter selection in Gaussian linear regression. It would be more novel if the EM algorithm was improved. 

2. The paper is well-written.
summary_of_the_review: I am concern about the novelty of this paper. 

The experiments are also suggested over larger models.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4180/Reviewer_MCAU
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a number of tricks to scale linearization + Laplace approximations to large neural networks with the goal of performing uncertainty quantification.  The standard approach is to train a neural network and then post-hoc linearize the network, and use a Laplace approximation to obtain that the posterior should be Gaussian.  In this posterior, a regularization term (corresponding to the prior) appears, and this term should be fit using maximum likelihood.  To do so requires some computationally expensive numerical matrix algebra, and so the present paper instead proposes some sampling approaches (which in turn can be turned back into optimization problems) to more efficiently fit the prior and obtain samples from the posterior.  The authors apply this approach to a number of networks and datasets, finding that it can obtain better uncertainty estimates faster than existing methods (particularly in the case of simultaneously estimating uncertainty at several points).
strength_and_weaknesses: Strengths:
* The approach is interesting and combines a lot of disparate ideas to make this the linearization + Laplace approximation scalable.  Some of these ideas may be of (minor) independent interest.
* The empirical runtime and uncertainty quantification results are impressive, and show a benefit to this approach for real networks.
* The writing was usually clear (but see below), and the results were presented clearly.

Weaknesses:
* In a few places the presentation was hard to follow.  For example, the rightmost plot in Figure 3 shows different layer-wise precisions, but the exposition only discusses learning a single precision (controlled by $\alpha$).  I can imagine how it is straightforward to extend to different layer-wise $\alpha$s, but that should be clarified in the main text.  In Section 4 , $\bar{w}$ is used without definition (presumably the optimized weights).  Then, $\bar{\theta}$ is used (e.g., in equation (15)) without definition.  Finally, the acronym MCDO is introduced _after_ references to Figure 6 and Table 2, where that acronym is used.  Similarly "DIP" is used in that Figure and Table, but never defined.
* The reported metrics are all good, but one of the most interpretable measures of uncertainty quantification is simply coverage, and it would be nice to report empirical coverage at different posterior credible levels on some held out data.  
* I found the results in Figure 5 difficult to interpret.  Namely, it is not obvious to me how any Bayesian uncertainty quantification should perform when some datapoints are corrupted.  Would the true posterior (perhaps with a learned prior) outperform approximate posteriors in terms of log likelihood on test data?  I'm not sure that that is true, and so I'm not sure what the relative ordering of the different uncertainty quantifications methods in Figure 5 says about their "quality".
clarity,_quality,_novelty_and_reproducibility: I am not an expert in neural network uncertainty quantification, so I cannot comment on the novelty of the proposed approach in that space.  Overall, the proposed approach feels like a collection of clever tricks, each of which may not be groundbreaking, collectively allow the linearization + Laplace approximation to scale to very large datasets and neural networks.  For the most part the writing is clear, aside from some notational issues described above.  The code is available on an anonymous github repo.  I did not run the code, but it looks well-commented and well-documented.
summary_of_the_review: Overall, I found the paper to be a collection of interesting tricks that scale an existing approach to larger neural networks and datasets.  The presentation was mostly clear, and the presented approach seems like it will be useful.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4180/Reviewer_ejzP
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a scalable linearized Laplace procedure which can be applied to realistic BNN models.  The algorithm can be viewed as an extension of the sample-then-optimize procedure that allows for normalization layers, and hyperparameter selection is also implemented a an E-step.  The authors evaluated the method on ResNet-18 and U-Net models, demonstrating improvements on predictive performance and quality of uncertainty estimates.
strength_and_weaknesses: The authors proposed a scalable approximation to the Laplace method, which is always of interest.  The proposed method is easy to implement, and demonstrates competitive performance.  The authors provided new connections to the sample-then-optimize formulation.

As for limitations, it is easy to nitpick (e.g., the limitation of Laplace method, the conjugate Gaussian model, or the linearized approximation), but I don't think these are important concerns that must be addressed in a single paper, or a single thread of literature, and the applicable scope of the proposed method is quite clear.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written; the algorithm appears easy to implement; the modifications are novel to my knowledge.
summary_of_the_review: Nice work with methodological and empirical contributions.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper4180/Reviewer_ZeYJ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors consider the task of improving the scalability of the popular linearised Laplace method for which they develop a sampling-based EM approach. The value of the proposed approach is then demonstrated in several experimental settings.

strength_and_weaknesses: The authors tackle an important task as the linearised Laplace is a popular approach, especially in the Bayesian Deep Learning literature. The experimental evaluation is extensive and considers both small/large scale classification tasks as well as an image reconstruction task. A minor weakness is the lack of downstream evaluation of the quality provided by the method's predictive uncertainty. However, that can be justified by the necessary limit of the breadth of the work given the page constraints.

clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear and comprehensible manner. The proposed approach combines several known approaches into a joint new one and provides a novel solution to the task. The discussion in the appendix together with the provided code seems to provide full reproducibility.

summary_of_the_review: The paper sets out to solve a well-defined task and delivers both concerning the theoretical as well as empirical side.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper