Original: Cold Posteriors through PAC-Bayes
Revised: Cold Posteriors through PAC-Bayes

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds.
Revised: We investigate the cold posterior effect through the lens of PAC-Bayes generalization bounds.

Original: We argue that in the non-asymptotic setting, when the number oftraining samples is (relatively) small, discussions of the cold posterior effect shouldtake into account that approximate Bayesian inference does not readily provideguarantees of performance on out-of-sample data.
Revised: We argue that in the non-asymptotic setting, when the number of training samples is (relatively) small, discussions of the cold posterior effect should take into account that approximate Bayesian inference does not readily provide guarantees of performance on out-of-sample data.

Original: Instead, out-of-sample error isbetter described through a generalization bound.
Revised: Instead, out-of-sample error is better described through a generalization bound.

Original: In this context, we explore theconnections of the ELBO objective from variational inference and the PAC-Bayesobjectives.
Revised: In this context, we explore the connections of the ELBO objective from variational inference and the PAC-Bayes objectives.

Original: We note that, while the ELBO and PAC-Bayes objectives are similar, thelatter objectives naturally contain a temperature parameter λ which is not restrictedto be λ = 1 .
Revised: We note that, while the ELBO and PAC-Bayes objectives are similar, the latter objectives naturally contain a temperature parameter λ which is not restricted to be λ = 1 .

Original: For realistic classification tasks, in the case of Laplace approximationsto the posterior, we show how this PAC-Bayesian interpretation of the temperatureparameter captures important aspects of the cold posterior effect.
Revised: For realistic classification tasks, in the case of Laplace approximations to the posterior, we show how this PAC-Bayesian interpretation of the temperature parameter captures important aspects of the cold posterior effect.

Original:  Introduction
Revised:  Introduction

Original: [Figure]
Revised: [Figure]

Original: In their influential paper
Revised: In their influential paper

Original: Wenzel et al.
Revised: Wenzel et al.

Original: [38] highlighted the observation that Bayesian neural networkstypically exhibit better test time predictive performance if the posterior distribution is “sharpened”through tempering.
Revised: [38] highlighted the observation that Bayesian neural networks typically exhibit better test time predictive performance if the posterior distribution is “sharpened” through tempering.

Original: Their work has been influential primary because it serves as a well documentedexample of the potential drawbacks of the Bayesian approach to deep learning.
Revised: Their work has been influential primary because it serves as a well documented example of the potential drawbacks of the Bayesian approach to deep learning.

Original: While other subfieldsof deep learning have seen rapid adoption, and have had impact on real world problems, Bayesian deep learning has, to date, seen relatively limited practical use [22, 27, 12, 38].
Revised: While other subfields of deep learning have seen rapid adoption, and have had impact on real world problems, Bayesian deep learning has, to date, seen relatively limited practical use [22, 27, 12, 38].

Original: The “cold posterioreffect”, as the authors of Wenzel et al.
Revised: The “cold posterior effect”, as the authors of Wenzel et al.

Original: [38] named their observation, highlights an essential mismatchbetween Bayesian theory and practice.
Revised: [38] named their observation, highlights an essential mismatch between Bayesian theory and practice.

Original: As the number of training samples increases, Bayesian theorytells states that the posterior distribution should be concentrating more and more on the true modelparameters, in a frequentist sense.
Revised: As the number of training samples increases, Bayesian theory tells states that the posterior distribution should be concentrating more and more on the true model parameters, in a frequentist sense.

Original: At any time, the posterior is our best guess at the true modelparameters, without having to resort to heuristics.
Revised: At any time, the posterior is our best guess at the true model parameters, without having to resort to heuristics.

Original: Since the original paper, a number of works[35, 42, 1, 32, 15, 2, 6, 2] have attempted to explain the cold posterior effect, identify its origins,propose remedies and defend Bayesian deep learning in the process.
Revised: Since the original paper, a number of works [35, 42, 1, 32, 15, 2, 6, 2] have attempted to explain the cold posterior effect, identify its origins, propose remedies and defend Bayesian deep learning in the process.

Original: Here, we investigate PAC-Bayes generalization bounds [30, 9, 3, 13] as the model that governs performance on out-of-sample data.
Revised: Here, we investigate PAC-Bayes generalization bounds [30, 9, 3, 13] as the model that governs performance on out-of-sample data.

Original: PAC-Bayes bounds describe the performance on out-of-sample data,through an application of the convex duality relation between measurable functions and probabilitymeasures.
Revised: PAC-Bayes bounds describe the performance on out-of-sample data, through an application of the convex duality relation between measurable functions and probability measures.

Original: The convex duality relationship naturally gives rise to the log-Laplace transform of aspecial random variable [9].
Revised: The convex duality relationship naturally gives rise to the log-Laplace transform of a special random variable [9].

Original: Importantly the log-Laplace transform has a temperature parameter λwhich is not constrained to be λ = 1 .
Revised: Importantly the log-Laplace transform has a temperature parameter λ which is not constrained to be λ = 1 .

Original: We investigate the relationship of this temperature parameter tocold posteriors.
Revised: We investigate the relationship of this temperature parameter to cold posteriors.

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We show that PAC-Bayes bounds correlate with out-of-sample performance for different values of the temperature parameter λ .•
Revised: • We show that PAC-Bayes bounds correlate with out-of-sample performance for different values of the temperature parameter λ .

Original:  We find that the coldest temperature (such that the posterior is a Dirac delta centered on a MAP estimate of the weights) is empirically always optimal in terms of test accuracy.•
Revised: • We find that the coldest temperature (such that the posterior is a Dirac delta centered on a MAP estimate of the weights) is empirically always optimal in terms of test accuracy.

Original:  We derive a PAC-Bayes bound for the case of the widely used generalized Gauss–NewtonLaplace approximations to the posterior.
Revised: • We derive a PAC-Bayes bound for the case of the widely used generalized Gauss–Newton Laplace approximations to the posterior.

Original: This bound might explain why it is difficult to pinpoint an exact cause for the cold-posterior effect.
Revised: This bound might explain why it is difficult to pinpoint an exact cause for the cold-posterior effect.

Original: We also include a detailed FAQ section in the Appendix.
Revised: We also include a detailed FAQ section in the Appendix.

Original:  Cold posterior effect: misspecified and non-asymptotic setting
Revised:  Cold posterior effect: misspecified and non-asymptotic setting

Original: We denote the learning sample ( X, Y ) =
Revised: We denote the learning sample ( X, Y ) =

Original: { ( x i , y i ) } ni =1 ∈ ( X × Y ) n , that contains n input-outputpairs.
Revised: { ( x i , y i ) } ni =1 ∈ ( X × Y ) n , that contains n input-output pairs.

Original: Observations ( X, Y ) are assumed to be sampled randomly from a distribution D .
Revised: Observations ( X, Y ) are assumed to be sampled randomly from a distribution D .

Original: Thus, wedenote ( X, Y ) ∼ D n the i.i.d observation of n elements.
Revised: Thus, we denote ( X, Y ) ∼ D n the i.i.d observation of n elements.

Original: We consider loss functions ℓ : F ×X ×Y →46 R , where F is a set of predictors f : X → Y .
Revised: We consider loss functions ℓ : F ×X ×Y → R , where F is a set of predictors f : X → Y .

Original: We also denote the risk L ℓ D ( f ) =
Revised: We also denote the risk L ℓ D ( f ) =

Original: E ( x ,y ) ∼D ℓ ( f, x , y )and the empirical risk ˆ L ℓX,Y ( f ) =
Revised: E ( x ,y ) ∼D ℓ ( f, x , y ) and the empirical risk ˆ L ℓX,Y ( f ) =

Original: (1 /n ) (cid:80) i ℓ ( f, x i , y i ) .
Revised: (1 /n ) (cid:80) i ℓ ( f, x i , y i ) .

Original: We consider two probability measures,the prior π ∈ M ( F ) and the posterior ˆ ρ ∈ M ( F ) .
Revised: We consider two probability measures, the prior π ∈ M ( F ) and the posterior ˆ ρ ∈ M ( F ) .

Original: Here, M ( F ) denotes the set of all probabilitymeasures on F .
Revised: Here, M ( F ) denotes the set of all probability measures on F .

Original: We encounter cases where we make predictions using the posterior predictivedistribution E f ∼ ˆ ρ
Revised: We encounter cases where we make predictions using the posterior predictive distribution E f ∼ ˆ ρ

Original: [ p ( y | x , f )] .
Revised: [ p ( y | x , f )] .

Original: We will use two loss functions, the non-differentiable zero-one lossℓ 01 ( f, x , y )
Revised: We will use two loss functions, the non-differentiable zero-one loss ℓ 01 ( f, x , y )

Original: =
Revised: =

Original: I (arg max j f ( x ) j ̸ = y ) , and the negative log-likelihood, which is a commonly useddifferentiable surrogate ℓ nll ( f, x , y ) =
Revised: I (arg max j f ( x ) j ̸ = y ) , and the negative log-likelihood, which is a commonly used differentiable surrogate ℓ nll ( f, x , y ) =

Original: − log( p ( y | x , f )) , where we assume that the outputs of f arenormalized to form a probability distribution.
Revised: − log( p ( y | x , f )) , where we assume that the outputs of f are normalized to form a probability distribution.

Original: Given the above, the Evidence Lower Bound (ELBO)has the following form
Revised: Given the above, the Evidence Lower Bound (ELBO) has the following form

Original: [Equation]
Revised: [Equation]

Original: (1) where λ = 1 .
Revised: (1) where λ = 1 .

Original: Note that our temperature parameter λ is the inverse of the one typically used incold posterior papers.
Revised: Note that our temperature parameter λ is the inverse of the one typically used in cold posterior papers.

Original: In this form λ has a clearer interpretation as the temperature of a log-Laplacetransform.
Revised: In this form λ has a clearer interpretation as the temperature of a log-Laplace transform.

Original: Overall our setup is one of the cases discussed in Wenzel et al.
Revised: Overall our setup is one of the cases discussed in Wenzel et al.

Original: [38], p3 Section 2.3.
Revised: [38], p3 Section 2.3.

Original: Whilethey use MCMC to conduct their experiments, we opt for the ELBO for analytical tractability.
Revised: While they use MCMC to conduct their experiments, we opt for the ELBO for analytical tractability.

Original: Wenzelet al.
Revised: Wenzel et al.

Original: [38] also temper by λ both the likelihood and the prior in the MCMC inference setting.
Revised: [38] also temper by λ both the likelihood and the prior in the MCMC inference setting.

Original: Asdiscussed in Aitchison [2] and Wenzel et al.
Revised: As discussed in Aitchison [2] and Wenzel et al.

Original: [38] the relevant setting for the ELBO is the one weconsider (Eq. 1), where only the KL is tempered.
Revised: [38] the relevant setting for the ELBO is the one we consider (Eq. 1), where only the KL is tempered.

Original: One then typically models the posterior and priordistributions over weights using a parametric distribution (commonly a Gaussian) and optimizes the
Revised: One then typically models the posterior and prior distributions over weights using a parametric distribution (commonly a Gaussian) and optimizes the

Original: ELBO, using the reparametrization trick, to find the posterior distribution [8, 23, 31, 5, 38].
Revised: ELBO, using the reparametrization trick, to find the posterior distribution [8, 23, 31, 5, 38].

Original: The coldposterior is the following observation:
Revised: The cold posterior is the following observation:

Original: Even though the ELBO has the form (1) with λ = 1 , practitioners have found thatmuch larger values λ ≫ 1 typically result in better test time performance, forexample a lower test misclassification rate and lower test negative log-likelihood.
Revised: Even though the ELBO has the form (1) with λ = 1 , practitioners have found that much larger values λ ≫ 1 typically result in better test time performance, for example a lower test misclassification rate and lower test negative log-likelihood.

Original: 2.1 ELBO
Revised: 2.1 ELBO

Original: We assume a training sample ( X, Y ) ∼ D n as before, denote p ( w | X, Y ) the true posterior probabilityover predictors f parameterized by w (typically weights for neural networks), and π and ˆ ρ respectivelythe prior and variational posterior distributions as before.
Revised: We assume a training sample ( X, Y ) ∼ D n as before, denote p ( w | X, Y ) the true posterior probability over predictors f parameterized by w (typically weights for neural networks), and π and ˆ ρ respectively the prior and variational posterior distributions as before.

Original: The ELBO results from the followingcalculations
Revised: The ELBO results from the following calculations

Original: [Equation]
Revised: [Equation]

Original: Thus, maximizing the ELBO can be seen as minimizing the KL divergence between the true posteriorand the variational posterior over the weights KL(ˆ ρ ( w ) ∥
Revised: Thus, maximizing the ELBO can be seen as minimizing the KL divergence between the true posterior and the variational posterior over the weights KL(ˆ ρ ( w ) ∥

Original: p ( w | X, Y )) , and doesn’t directly boundthe test misclassification error.
Revised: p ( w | X, Y )) , and doesn’t directly bound the test misclassification error.

Original: We could ignore this problem if consistency theorems hold andthe posterior quickly contracts to a Dirac delta on the true parameters.
Revised: We could ignore this problem if consistency theorems hold and the posterior quickly contracts to a Dirac delta on the true parameters.

Original: However, operating in theregime of misspecification f ∗ / ∈ F and where n is (comparatively) small invalidates consistencytheorems such as the Blackwell–Dubins [7] theorem (for example, neural networks have multipleminima, implying misspecification).
Revised: However, operating in the regime of misspecification f ∗ / ∈ F and where n is (comparatively) small invalidates consistency theorems such as the Blackwell–Dubins [7] theorem (for example, neural networks have multiple minima, implying misspecification).

Original: This makes it important to derive a more precise certificate ofgeneralization through a generalization bound, which directly bounds the true risk.
Revised: This makes it important to derive a more precise certificate of generalization through a generalization bound, which directly bounds the true risk.

Original: In the followingwe focus on analyzing a PAC-Bayes bound generalization bound in order to obtain insights into whenthe cold posterior effect occurs.
Revised: In the following we focus on analyzing a PAC-Bayes bound generalization bound in order to obtain insights into when the cold posterior effect occurs.

Original: 2.2 PAC-Bayes
Revised: 2.2 PAC-Bayes

Original: For classification tasks, we are typically mainly interested in achieving low expected zero-one risk
Revised: For classification tasks, we are typically mainly interested in achieving low expected zero-one risk

Original: E f ∼ ˆ ρ L ℓ 01 D ( f ) .
Revised: E f ∼ ˆ ρ L ℓ 01 D ( f ) .

Original: The ELBO objective is not directly related to this risk, however in the PAC-Bayesianliterature there exist bounds specifically adapted to it.
Revised: The ELBO objective is not directly related to this risk, however in the PAC-Bayesian literature there exist bounds specifically adapted to it.

Original: In the following we will use one of the tightestand most commonly used bounds, the “Catoni" bound, denoted B Catoni .
Revised: In the following we will use one of the tightest and most commonly used bounds, the “Catoni" bound, denoted B Catoni .

Original: Theorem 1 ( B Catoni , 9) .
Revised: Theorem 1 ( B Catoni , 9) .

Original: Given a distribution D over X ×Y , a hypothesis set F , the 0-1 loss functionℓ 01 : F ×X ×Y → [0 , 1] , a prior distribution π over F , a real number δ ∈ (0 , 1] , and a real numberλ > 0 , with probability at least 1 − δ over the choice of ( X, Y ) ∼ D n , we have
Revised: Given a distribution D over X ×Y , a hypothesis set F , the 0-1 loss function ℓ 01 : F ×X ×Y → [0 , 1] , a prior distribution π over F , a real number δ ∈ (0 , 1] , and a real number λ > 0 , with probability at least 1 − δ over the choice of ( X, Y ) ∼ D n , we have

Original: [Equation]
Revised: [Equation]

Original: where Φ − 1 λ ( x ) = 1 − e − λx 1 − e − λ .
Revised: where Φ − 1 λ ( x ) = 1 − e − λx 1 − e − λ .

Original: The empirical risk term is the empirical mean of the loss of the classifier over all training samples.
Revised: The empirical risk term is the empirical mean of the loss of the classifier over all training samples.

Original: The KL term is the complexity of the model, which in this case is measured as the KL-divergencebetween the posterior and prior distributions.
Revised: The KL term is the complexity of the model, which in this case is measured as the KL-divergence between the posterior and prior distributions.

Original: The Moment term has been absorbed in this case in thefunction Φ −1 λ ( x ) =
Revised: The Moment term has been absorbed in this case in the function Φ −

Original: 1 − e − λx 1 − e − λ .
Revised: [Equation]

Original: 2.3 Safe-Bayes and other relevant work
Revised: 2.3 Safe-Bayes and other relevant work

Original: Germain et al.
Revised: Germain et al.

Original: [16] were the first to find connections between PAC-Bayes and Bayesian inference.
Revised: [16] were the first to find connections between PAC-Bayes and Bayesian inference.

Original: However they only investigate the case where λ = 1 .
Revised: However they only investigate the case where λ = 1 .

Original: After identifying two sources of misspecification,
Revised: After identifying two sources of misspecification,

Original: Grünwald and Langford [18] proposed a solution, through an approach which they named Safe-Bayes[17, 19].
Revised: Grünwald and Langford [18] proposed a solution, through an approach which they named Safe-Bayes [17, 19].

Original: Safe-Bayes corresponds to finding a temperature parameter λ for a generalized (tempered)posterior distribution with λ possibly different than 1.
Revised: Safe-Bayes corresponds to finding a temperature parameter λ for a generalized (tempered) posterior distribution with λ possibly different than 1.

Original: The optimal value of λ is found by taking asequential view of Bayesian inference.
Revised: The optimal value of λ is found by taking a sequential view of Bayesian inference.

Original: By contrast we provide an analytical expression of the boundon true risk, given λ , and also numerically investigate the case of λ > 1 .
Revised: By contrast we provide an analytical expression of the bound on true risk, given λ , and also numerically investigate the case of λ > 1 .

Original: Our analysis thus providesintuition regarding which parameters (for example the curvature) might result in cold posteriors.
Revised: Our analysis thus provides intuition regarding which parameters (for example the curvature) might result in cold posteriors.

Original:  Experiments on classification tasks
Revised:  Experiments on classification tasks

Original: The ELBO (1) is minimized at the probability density ρ ⋆ ( f ) given by: ρ ⋆ [Equation] f 
Revised: The ELBO (1) is minimized at the probability density ρ ⋆ ( f ) given by: ρ ⋆ ( f ) :

Original: [Equation]
Revised: [Equation]

Original: := π ( f ) e − λn ˆ L ℓ nll X,Y ( f ) / E f ∼ π (cid:104)
Revised: π ( f ) e − λn ˆ L ℓ nll X,Y ( f ) / E f ∼ π (cid:104)

Original: e − λn ˆ L ℓ nll X,Y ( f ) (cid:105) [9].
Revised: e − λn ˆ L ℓ nll X,Y ( f ) (cid:105) [9].

Original: We will use the Laplace approximation to the posterior in our experiments.
Revised: We will use the Laplace approximation to the posterior in our experiments.

Original: This is equivalent to approximating λn
Revised: This is equivalent to approximating λn

Original: ˆ L ℓ nll X,Y ( f ) using a second order
Revised: ˆ L ℓ nll X,Y ( f ) using a second order

Original: Taylor expansion around a minimum w ˆ ρ , such that λn ˆ L ℓ nll X,Y ( f w )
Revised: Taylor expansion around a minimum w ˆ ρ , such that λn ˆ L ℓ nll X,Y ( f w )

Original: ≈
Revised: ≈

Original: λn ˆ L ℓ nll X,Y ( f
Revised: λn ˆ L ℓ nll X,Y ( f

Original: w ˆ ρ )
Revised: w ˆ ρ )

Original: + λn ( w − w ˆ ρ ) ⊤ 12 ∇∇ ˆ L ℓ nll X,Y ( f w ) | w = w ˆ ρ ( w − w ˆ ρ ) .
Revised: + λn ( w − w ˆ ρ ) ⊤ 12 ∇∇ ˆ L ℓ nll X,Y ( f w ) | w = w ˆ ρ ( w − w ˆ ρ ) .

Original: Assuming a Gaussian prior π = N (0 , σ 2 π I ) , the Laplace approximation to the posterior ˆ ρ is again a Gaussian
Revised: Assuming a Gaussian prior π = N (0 , σ 2 π I ) , the Laplace approximation to the posterior ˆ ρ is again a Gaussian

Original: [Equation]
Revised: [Equation]

Original: where H is the network Hessian H = n ∇∇ ˆ L ℓ nll X,Y ( f w )
Revised: where H is the network Hessian H = n ∇∇ ˆ L ℓ nll X,Y ( f w )

Original: | w =
Revised: | w =

Original: w ˆ ρ .
Revised: w ˆ ρ .

Original: This Hessian is generally infeasibleto compute in practice for modern deep neural networks, such that many approaches employ thegeneralized Gauss–Newton (GGN) approximation H GGN : = (cid:80) ni
Revised: This Hessian is generally infeasible to compute in practice for modern deep neural networks, such that many approaches employ the generalized Gauss–Newton (GGN) approximation H GGN := (cid:80) ni

Original: =1 J w ( x i ) ⊤ Λ
Revised: =1 J w ( x i ) ⊤ Λ

Original: ( y i ; f i )
Revised: ( y i ; f i )

Original: J w ( x i ) ,where J w ( x ) is the network per-sample Jacobian [ J w ( x )]
Revised: J w ( x i ) , where J w ( x ) is the network per-sample Jacobian [ J w ( x )]

Original: c = ∇ w f c ( x ; w ˆ ρ ) , and Λ ( y ; f ) =−∇ 2 ff log p ( y ; f ) is the per-input noise matrix [25].
Revised: c = ∇ w f c ( x ; w ˆ ρ ) , and Λ ( y ; f ) = −∇ 2 ff log p ( y ; f ) is the per-input noise matrix [25].

Original: We will use two simplified versions of the GGN
Revised: We will use two simplified versions of the GGN

Original: • An isotropic approximation with variance σ 2ˆ ρ ( λ ) such that
Revised: • An isotropic approximation with variance σ 2ˆ ρ ( λ ) such that

Original: [Equation]
Revised: [Equation]

Original: + 1 σ 2 π , where h =(cid:80) i,j,k g ( i, k )( ∇ w f k ( x i ; w ˆ ρ ) j ) 2 is the trace of the Gauss–Newton approximation to theHessian, with g ( i, k ) =
Revised: where h = (cid:80) i,j,k g ( i, k )( ∇ w f k ( x i ; w ˆ ρ ) j ) 2 is the trace of the Gauss–Newton approximation to the Hessian, with g ( i, k ) =

Original: [ Λ ( y i ; f )] kk . 
Revised: [ Λ ( y i ; f )] kk . •

Original: • The Kronecker-Factorized Approximate Curvature (KFAC) [29] approximation, which retains only a block diagonal part of the GGN.
Revised: The Kronecker-Factorized Approximate Curvature (KFAC) [29] approximation, which retains only a block diagonal part of the GGN.

Original: When making predictions, we use the posterior predictive distribution E w ∼ ˆ ρ [ p ( y | x , f w )] of the fullneural network model , meaning that samples from ˆ ρ are inputted to the full neural network.
Revised: When making predictions, we use the posterior predictive distribution E w ∼ ˆ ρ [ p ( y | x , f w )] of the full neural network model , meaning that samples from ˆ ρ are inputted to the full neural network.

Original: Sincethe 0-1 loss is not differentiable, the posterior estimated with the cross entropy loss will be used forclassification problems.
Revised: Since the 0-1 loss is not differentiable, the posterior estimated with the cross entropy loss will be used for classification problems.

Original: We have tested extensively in classification tasks, scaling from simplified settings to realistic modelsand datasets.
Revised: We have tested extensively in classification tasks, scaling from simplified settings to realistic models and datasets.

Original: For the classification task we used the CIFAR-10, CIFAR-100 [24], SVHN [34]and FashionMnist [39] datasets.
Revised: For the classification task we used the CIFAR-10, CIFAR-100 [24], SVHN [34] and FashionMnist [39] datasets.

Original: In all experiments, we split the dataset into two sets.
Revised: In all experiments, we split the dataset into two sets.

Original: Thesethree are the typical prediction tasks sets: training set Z train , testing set Z test , and validation set
Revised: These three are the typical prediction tasks sets: training set Z train , testing set Z test , and validation set

Original: Z validation .
Revised: Z validation .

Original: We use Monte Carlo sampling to estimate the Empirical Risk term ( f ∼ ˆ ρ ).
Revised: We use Monte Carlo sampling to estimate the Empirical Risk term ( f ∼ ˆ ρ ).

Original: For theisotropic Laplace approximation, and a Gaussian isotropic prior, the KL divergence has a simpleanalytical expression KL(ˆ 
Revised: For the isotropic Laplace approximation, and a Gaussian isotropic prior, the KL divergence has a simple analytical expression KL(ˆ [Equation] || [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( λ ) + d
Revised: λ ) + d

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:17) .
Revised: σ 2 π (cid:17) .

Original: PACBayes bounds require correct control of the prior mean as the ℓ 2 distance between prior and posteriormeans in the KL term is often the dominant term in the bound.
Revised: PACBayes bounds require correct control of the prior mean as the ℓ 2 distance between prior and posterior means in the KL term is often the dominant term in the bound.

Original: To control this distance, we follow avariation of the approach in Dziugaite et al.
Revised: To control this distance, we follow a variation of the approach in Dziugaite et al.

Original: [14] to constructing our classifiers.
Revised: [14] to constructing our classifiers.

Original: We first use Z train tofind a prior mean w π .
Revised: We first use Z train to find a prior mean w π .

Original: We then set the posterior mean equal to the prior mean w ˆ ρ = w π but evaluatethe r.h.s of the bounds on Z validation .
Revised: We then set the posterior mean equal to the prior mean w ˆ ρ = w π but evaluate the r.h.s of the bounds on Z validation .

Original: Note that in this way ∥ w ˆ ρ − w π ∥ 22 = 0 , while the bound is stillvalid since the prior is independent from the evaluation set X, Y =
Revised: Note that in this way ∥ w ˆ ρ − w π ∥ 22 = 0 , while the bound is still valid since the prior is independent from the evaluation set X, Y =

Original: Z validation .
Revised: Z validation .

Original: For the CIFAR-10,
Revised: For the CIFAR-10,

Original: CIFAR-100, and SVHN datasets, we use a WideResNet22 [40], with Fixup initialization [43].
Revised: CIFAR-100, and SVHN datasets, we use a WideResNet22 [40], with Fixup initialization [43].

Original: For the
Revised: For the

Original: FashionMnist dataset, we use a convolutional architecture with three convolutional layers, followedby two fully connected non-linear layers.
Revised: FashionMnist dataset, we use a convolutional architecture with three convolutional layers, followed by two fully connected non-linear layers.

Original: More details on the experimental setup can be found in the
Revised: More details on the experimental setup can be found in the

Original: Appendix.
Revised: Appendix.

Original: 3.1 Classification experiments
Revised: 3.1 Classification experiments

Original: We find ten MAP estimates for the neural network weights of the CIFAR-10, CIFAR-100, SVHN and
Revised: We find ten MAP estimates for the neural network weights of the CIFAR-10, CIFAR-100, SVHN and

Original: FMNIST datasets by training on Z train using SGD.
Revised: FMNIST datasets by training on Z train using SGD.

Original: We then fit an Isotropic Laplace approximation toeach MAP estimate using X, Y = Z validation .
Revised: We then fit an Isotropic Laplace approximation to each MAP estimate using X, Y = Z validation .

Original: For different values of λ we then estimate the Catonibound (Theorem 1) using Z validation .
Revised: For different values of λ we then estimate the Catoni bound (Theorem 1) using Z validation .

Original: We also estimate the test 0-1 Loss, negative log-likelihood(NLL) and the Expected Calibration Error (ECE)
Revised: We also estimate the test 0-1 Loss, negative log-likelihood (NLL) and the Expected Calibration Error (ECE)

Original: [33] of the posterior predictive on Z test .
Revised: [33] of the posterior predictive on Z test .

Original: We usethe prior variance σ 2 π = 0 .
Revised: We use the prior variance σ 2 π = 0 .

Original: 1 , as optimizing the marginal likelihood leads to σ 2 π ≈ 0 which is notrelevant for BNNs.
Revised: 1 , as optimizing the marginal likelihood leads to σ 2 π ≈ 0 which is not relevant for BNNs.

Original: We also test two standard setups of increasing difficulty.
Revised: We also test two standard setups of increasing difficulty.

Original: First, the standard
Revised: First, the standard

Original: “Isotropic" case where we fit the Laplace on Z train .
Revised: “Isotropic" case where we fit the Laplace on Z train .

Original: Second, the KFAC case where we fit the Laplaceon Z train and also choose the prior through the marginal likelihood.
Revised: Second, the KFAC case where we fit the Laplace on Z train and also choose the prior through the marginal likelihood.

Original: In both of these last two cases,we estimate the evaluation metrics on the validation set Z validation as from the literature we knowthat any PAC-Bayes bound will be vacuous (larger than 1) as we do not control ∥ w ˆ ρ − w π ∥ 22 .
Revised: In both of these last two cases, we estimate the evaluation metrics on the validation set Z validation as from the literature we know that any PAC-Bayes bound will be vacuous (larger than 1) as we do not control ∥ w ˆ ρ − w π ∥ 22 .

Original: We plot the results for all datasets in Figure 2.
Revised: We plot the results for all datasets in Figure 2.

Original: The Catoni bound correlates tightly with test [-0-
 Loss-] for all datasets and we plot this correlation in Figure 1(b).
Revised: The Catoni bound correlates tightly with test [-0-
 Loss-]0Loss for all datasets and we plot this correlation in Figure 1(b).

Original: Again, in terms of test 0-1 Loss, the
Revised: Again, in terms of test 0-1 Loss, the

Original: MAP estimate (obtained where λ ≫ 1 and the posterior is “coldest") is optimal.
Revised: MAP estimate (obtained where λ ≫ 1 and the posterior is “coldest") is optimal.

Original: This bevaviour isreplicated both in the “Isotropic” and “KFAC” cases.
Revised: This bevaviour is replicated both in the “Isotropic” and “KFAC” cases.

Original: In the Laplace approximation literature fordeep neural networks, there are various similar results hidden in plain sight and to the best of ourknowledge never directly addressed [4, 10, 37].
Revised: In the Laplace approximation literature for deep neural networks, there are various similar results hidden in plain sight and to the best of our knowledge never directly addressed [4, 10, 37].

Original: The crucial point here is the choice of the evaluation metric .
Revised: The crucial point here is the choice of the evaluation metric .

Original: We plot in the Appendix the Isotropicand KFAC cases for the NLL.
Revised: We plot in the Appendix the Isotropic and KFAC cases for the NLL.

Original: We find that all three cases of temperatures (cold posterior, warmposterior, as well as posterior with λ = 1 ) can be optimal, for varying datasets.
Revised: We find that all three cases of temperatures (cold posterior, warm posterior, as well as posterior with λ = 1 ) can be optimal, for varying datasets.

Original: We discuss the ECEresults in the Appendix.
Revised: We discuss the ECE results in the Appendix.

Original:  Effect of temperature parameter λ on PAC-Bayes bound
Revised:  Effect of temperature parameter λ on PAC-Bayes bound

Original: In light of our empirical results, it would be interesting to derive an analytical form that elucidatesthe important variables that affect the bound.
Revised: In light of our empirical results, it would be interesting to derive an analytical form that elucidates the important variables that affect the bound.

Original: However, PAC-Bayes objectives are difficult to analyzetheoretically for the non-convex case.
Revised: However, PAC-Bayes objectives are difficult to analyze theoretically for the non-convex case.

Original: Thus in the following we make a number of simplifyingassumptions.
Revised: Thus in the following we make a number of simplifying assumptions.

Original: The Laplace approximation with the Generalized Gauss-Newton approximation to the
Revised: The Laplace approximation with the Generalized Gauss-Newton approximation to the

Original: Hessian corresponds to a linearization of the neural network around the MAP estimate w ˆ ρ ∈ R d [20]
Revised: Hessian corresponds to a linearization of the neural network around the MAP estimate w ˆ ρ ∈ R d [20]

Original: [Equation]
Revised: [Equation]

Original: When analyzing minima of the loss landscape linearization is reasonable even without assuminginfinite width Zancato et al.
Revised: When analyzing minima of the loss landscape linearization is reasonable even without assuming infinite width Zancato et al.

Original: [41], Maddox et al.
Revised: [41], Maddox et al.

Original: For appropriate modelling choices, we aim atderiving a bound for this linearized model.
Revised: For appropriate modelling choices, we aim at deriving a bound for this linearized model.

Original: We adopt the linear form (3) together with the Gaussian likelihood with σ = 1 , yieldingℓ nll ( w , x , y ) 
Revised: We adopt the linear form (3) together with the Gaussian likelihood with σ = 1 , yielding ℓ nll ( w , x , [Equation] ) = [Equation]

Original: =
Revised: + [Equation] y [Equation]

Revised: f

Revised: [Equation]

Revised: [Equation]

Revised: ˆ

Revised: [Equation]

Original: 12 ln(2 π ) + 12 ( y − f ( x ; w ˆ ρ ) − ∇ w f ( x ; w ˆ ρ ) ⊤ ( w − w ˆ ρ )) 2 .
Revised: [Equation] ⊤ [Equation] − w ˆ ρ )) 2 .

Original: We also make thefollowing modeling choices
Revised: We also make the following modeling choices

Original: • Prior over weights: w ∼ N ( w π , σ 2 π I ) .
Revised: • Prior over weights: w ∼ N ( w π , σ 2 π I ) .

Original: • Gradients as Gaussian mixture: ∇ w f ( x ; w ˆ ρ ) ∼ (cid:80) ki =
Revised: • Gradients as Gaussian mixture: ∇ w f ( x ; w ˆ ρ ) ∼ (cid:80) ki =

Original: 1 ϕ i N ( µ i , σ 2 x i I ) ; note that this as- sumption should be plausible for trained neural networks, in that previous works have shown that per sample gradients with respect to the weights, at w ˆ ρ , are clusterable [41].
Revised: 1 ϕ i N ( µ i , σ 2 x i I ) ; note that this assumption should be plausible for trained neural networks, in that previous works have shown that per sample gradients with respect to the weights, at w ˆ ρ , are clusterable [41].

Original: We consider that a Gaussian Mixture model for these clusters is reasonable.
Revised: We consider that a Gaussian Mixture model for these clusters is reasonable.

Original: • Labeling function: y = f
Revised: • Labeling function: y = f

Original: ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ )
Revised: ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ )

Original: + ϵ , where ϵ ∼ N (0 , σ 2 ϵ ) .
Revised: + ϵ , where ϵ ∼ N (0 , σ 2 ϵ ) .

Original: Thus y | x ∼ N ( f ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ ) , σ 2 ϵ ) .
Revised: Thus y | x ∼ N ( f ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ ) , σ 2 ϵ ) .

Original: The assumption that w ∗ is close to w ˆ ρis quite strong, and we furthermore argued in the previous sections that no single w is truly “correct”.
Revised: The assumption that w ∗ is close to w ˆ ρ is quite strong, and we furthermore argued in the previous sections that no single w is truly “correct”.

Original: However we note that for fine-tuning tasks linearized neural networks work remarkably well [28, 11].
Revised: However we note that for fine-tuning tasks linearized neural networks work remarkably well [28, 11].

Original: It is therefore at least somewhat reasonable to assume the above oracle labelling function, in thatfor deep learning architectures good w that fit many datasets can be found close to w ˆ ρ in practicalsettings.
Revised: It is therefore at least somewhat reasonable to assume the above oracle labelling function, in that for deep learning architectures good w that fit many datasets can be found close to w ˆ ρ in practical settings.

Original: We also assume that we have a deterministic estimate of the posterior weights w ˆ ρ which wekeep fixed , and we model the posterior as ˆ ρ = N ( w ˆ ρ , σ 2ˆ ρ ( λ ) I ) , similarly to our experimental section.
Revised: We also assume that we have a deterministic estimate of the posterior weights w ˆ ρ which we keep fixed , and we model the posterior as ˆ ρ = N ( w ˆ ρ , σ 2ˆ ρ ( λ ) I ) , similarly to our experimental section.

Original: Therefore estimating the posterior corresponds to estimating the variance σ 2ˆ ρ ( λ ) .
Revised: Therefore estimating the posterior corresponds to estimating the variance σ 2ˆ ρ ( λ ) .

Original: Proposition 1 ( B approximate ) .
Revised: Proposition 1 ( B approximate ) .

Original: With the above modeling choices, and given a distribution D over
Revised: With the above modeling choices, and given a distribution D over

Original: X × Y , real numbers δ ∈ (0 , 1] and λ ∈ (0 , 1 c ) with c = 2 nσ 2 x σ 2 π , with probability at least 1 − δover the choice ( X, Y ) ∼ D n , we have
Revised: X × Y , real numbers δ ∈ (0 , 1] and λ ∈ (0 , 1 c ) with c = 2 nσ 2 x σ 2 π , with probability at least 1 − δ over the choice ( X, Y ) ∼ D n , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: where h = (cid:80) i (cid:80) j ( ∇ w f ( x i ; w ˆ ρ ) j ) 2 is the curvature parameter

Original: [Equation]
Revised: [Equation]

Original: where h = (cid:80) i (cid:80) j ( ∇ w f ( x i ; w ˆ ρ ) j ) 2 is the curvature parameter, and σ 2 x = (cid:80) kj =1 ϕ j σ 2 x j is theposterior gradient variance.
Revised: and σ 2 x = (cid:80) kj = 1 ϕ j σ 2 x j is the posterior gradient variance.

Original: We now make a number of observations regarding Proposition 1.
Revised: We now make a number of observations regarding Proposition 1.

Original: Here, h is the trace of the Hessianunder the Gauss–Newton approximation (without a scaling factor n ).
Revised: Here, h is the trace of the Hessian under the Gauss–Newton approximation (without a scaling factor n ).

Original: Under the PAC-Bayesianmodeling of the risk, cold posteriors are the result of a complex interaction between various parametersresulting from 1) our model such as the prior variance σ 2 π and prior mean w π 2) our data σ 2 x andw ∗ (the curvature of the minimum h and the MAP estimate w ˆ ρ depend on the deep neural networkarchitecture, the optimization procedure and the data).
Revised: Under the PAC-Bayesian modeling of the risk, cold posteriors are the result of a complex interaction between various parameters resulting from 1) our model such as the prior variance σ 2 π and prior mean w π 2) our data σ 2 x and w ∗ (the curvature of the minimum h and the MAP estimate w ˆ ρ depend on the deep neural network architecture, the optimization procedure and the data).

Original: A number of works have tried to identify thecauses of the cold posterior effect [35, 15], with often contradictory results, typically identifyingsufficient but necessary conditions.
Revised: A number of works have tried to identify the causes of the cold posterior effect [35, 15], with often contradictory results, typically identifying sufficient but necessary conditions.

Original: Given the complex interactions in Proposition 1, our result mightshed light on why pinpointing the exact cause is difficult in practice.
Revised: Given the complex interactions in Proposition 1, our result might shed light on why pinpointing the exact cause is difficult in practice.

Original:  Discussion
Revised:  Discussion

Original: A number of interesting questions are raised by our results.
Revised: A number of interesting questions are raised by our results.

Original: How can we link our results to the
Revised: How can we link our results to the

Original: MCMC setting?
Revised: MCMC setting?

Original: Which metrics are relevant for the cold-posterior effect?
Revised: Which metrics are relevant for the cold-posterior effect?

Original: For which metrics andfor which approximations to the curvature is the Laplace approximation relevant for modern deeplearning?
Revised: For which metrics and for which approximations to the curvature is the Laplace approximation relevant for modern deep learning?

Original: We intend to answer these in future work.
Revised: We intend to answer these in future work.

Original: A Proofs main results 
Revised: A Proofs main results A.1 Proof of Proposition

Original: A.1 Proof of Proposition 1

Original: Recall that we model our predictor as f lin ( x ; w ) =
Revised: Recall that we model our predictor as f lin ( x ; w ) =

Original: f ( x ; w ˆ ρ ) − ∇ w f ( x ; w ˆ ρ ) ⊤ ( w − w ˆ ρ ) .
Revised: f ( x ; w ˆ ρ ) − ∇ w f ( x ; w ˆ ρ ) ⊤ ( w − w ˆ ρ ) .

Original: Thenfor the choice of a Gaussian likelihood, given a training signal x , a training label y and weightsw , the negative log-likelihood loss takes the form ℓ nll ( w , x , y )
Revised: Then for the choice of a Gaussian likelihood, given a training signal x , a training label y and weights w , the negative log-likelihood loss takes the form ℓ nll ( w , x , y )

Original: =
Revised: =

Original: 12 ln(2 π ) + 12 ( y − f ( x ; w ˆ ρ ) −∇ w f ( x ; w ˆ ρ ) ⊤ ( w − w ˆ ρ )) 2 .
Revised: 12 ln(2 π ) + 12 ( y − f ( x ; w ˆ ρ ) − ∇ w f ( x ; w ˆ ρ ) ⊤ ( w − w ˆ ρ )) 2 .

Original: We also define ˆ L ℓX,Y ( f ) =
Revised: We also define ˆ L ℓX,Y ( f ) =

Original: (1 /n ) (cid:80) i ℓ ( f, x i , y i ) .
Revised: (1 /n ) (cid:80) i ℓ ( f, x i , y i ) .

Original: Our derivationsclosely follow the approach of Germain et al.
Revised: Our derivations closely follow the approach of Germain et al.

Original: [16] p.11, section A.4.
Revised: [16] p.11, section A.4.

Original: Given the above definitions and modelling choices we develop the empirical risk term
Revised: Given the above definitions and modelling choices we develop the empirical risk term

Original: [Equation]
Revised: [Equation]

Original: In the penultimate line, we have used the fact that a real number is the trace of itself as well as thecyclic property of the trace.
Revised: In the penultimate line, we have used the fact that a real number is the trace of itself as well as the cyclic property of the trace.

Original: The second summation ( (cid:80) j over the parameters of the model) resultsfrom the fact that ˆ ρ = N ( w ˆ ρ , σ 2ˆ ρ I ) is isotropic with a common scaling factor σ 2ˆ ρ .
Revised: The second summation ( (cid:80) j over the parameters of the model) results from the fact that ˆ ρ = N ( w ˆ ρ , σ 2ˆ ρ I ) is isotropic with a common scaling factor σ 2ˆ ρ .

Original: The term in blue isexactly the Gauss–Newton approximation to the Hessian of the full neural network, for the squaredloss function [25, 20], and in the last line we set h
Revised: The term in blue is exactly the Gauss–Newton approximation to the Hessian of the full neural network, for the squared loss function [25, 20], and in the last line we set h

Original: = (cid:104)(cid:80) i (cid:80) j ( ∇ w f ( x i ; w ˆ ρ ) j ) 2 (cid:105) .
Revised: = (cid:104)(cid:80) i (cid:80) j ( ∇ w f ( x i ; w ˆ ρ ) j ) 2 (cid:105) .

Original: Since h is a sumof positive numbers, taking into account that the blue term is the Gauss–Newton approximation tothe Hessian and if we assume that the Gauss–Newton approximation is diagonal, then h is a measureof the curvature at minimum w ˆ ρ of the loss landscape.
Revised: Since h is a sum of positive numbers, taking into account that the blue term is the Gauss–Newton approximation to the Hessian and if we assume that the Gauss–Newton approximation is diagonal, then h is a measure of the curvature at minimum w ˆ ρ of the loss landscape.

Original: We finally get
Revised: We finally get

Revised: E

Original: [Equation]
Revised: [Equation]

Original: We continue with the KL term which is known to have the following analytical expression for
Revised: We continue with the KL term which is known to have the following analytical expression for

Original: Gaussian prior and posterior distributions
Revised: Gaussian prior and posterior distributions

Original: [Equation]
Revised: [Equation]

Original: We finally develop the moment term.
Revised: We finally develop the moment term.

Original: Using an intermediate variable λ n = λn 2 to simplify thecalculations, we get
Revised: Using an intermediate variable λ n = λn 2 to simplify the calculations, we get

Original: [Equation]
Revised: [Equation]

Original: λ n σ 2 ϵ ] .
Revised: σ 2 ϵ ] .

Original: Inequality in line 4 is because the exponential function is less than 1 on the negative half line.
Revised: Inequality in line 4 is because the exponential function is less than 1 on the negative half line.

Original: Inthe fifth line we use our modelling choice y = f
Revised: In the fifth line we use our modelling choice y = f

Original: ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ ) + ϵ , whereϵ ∼ N (0 , σ 2 ϵ ) .
Revised: ( x ; w ˆ ρ ) + ∇ w f ( x ; w ˆ ρ ) ⊤ ( w ∗ − w ˆ ρ ) + ϵ , where ϵ ∼ N (0 , σ 2 ϵ ) .

Original: To obtain the final line we note that the gradient of the neural network output withrespect to w , that is ∇ w f ( x ; w ˆ ρ ) , does not depend on the label y .
Revised: To obtain the final line we note that the gradient of the neural network output with respect to w , that is ∇ w f ( x ; w ˆ ρ ) , does not depend on the label y .

Original: We get the last line by applyingthe square and taking the expectation, given that the noise ϵ is centered.
Revised: We get the last line by applying the square and taking the expectation, given that the noise ϵ is centered.

Original: We now take into account the Gaussian mixture modelling for the gradients per data sample,∇ w f ( x ; w ˆ ρ ) ∼ (cid:80) kj =1 ϕ j N ( µ j , σ 2 x j I ) .
Revised: We now take into account the Gaussian mixture modelling for the gradients per data sample, ∇ w f ( x ; w ˆ ρ ) ∼ (cid:80) kj =1 ϕ j N ( µ j , σ 2 x j I ) .

Original: We get
Revised: We get

Original: E x
Revised: E x

Original: [Equation]
Revised: [Equation]

Original:  ( x ; w ˆ 
Revised: f ( x ; w ˆ ρ ) ⊤ (

Original: [Equation]
Revised: [Equation]

Original: ⊤ ( w ∗ − w )) 2 ] =
Revised: =

Original: E x
Revised: E x

Original: [Equation]
Revised: [Equation]

Original: ( (cid:80) i ∇ w f [Equation] x ; 
Revised: i ∇ w f ( x ; w ˆ ρ ) i

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: w

Original: [Equation]
Revised: [Equation]

Original: The red term cancels out because we assumed that each weight is independent from the others.
Revised: The red term cancels out because we assumed that each weight is independent from the others.

Original: Nextwe use the Gaussian mixture modelling to get E x [ ∇ w f ( x ; w ˆ ρ ) 2 i ] = (cid:80) kj =1 ( ϕ j σ 2 x j ) , and we finallyset σ 2 x = (cid:80) kj =1 ( ϕ j σ 2 x j ) , as each component of the mixture is isotropic, thus the second moment ofall weights is the same.
Revised: Next we use the Gaussian mixture modelling to get E x [ ∇ w f ( x ; w ˆ ρ ) 2 i ] = (cid:80) kj =1 ( ϕ j σ 2 x j ) , and we finally set σ 2 x = (cid:80) kj =1 ( ϕ j σ 2 x j ) , as each component of the mixture is isotropic, thus the second moment of all weights is the same.

Original: By completing the square above, one obtains the Gaussian expectation of thissquared norm and forms the moment term as follows
Revised: By completing the square above, one obtains the Gaussian expectation of this squared norm and forms the moment term as follows

Original: [Equation]
Revised: [Equation]

Original: which assumes 1 − 2 λ n σ 2 x σ 2 π > 0 .
Revised: which assumes 1 − 2 λ n σ 2 x σ 2 π > 0 .

Original: The second line above is obtained by using the moment generatingfunction of noncentral χ 2 variables, while the inequality comes from ln( u ) <
Revised: The second line above is obtained by using the moment generating function of noncentral χ 2 variables, while the inequality comes from ln( u ) <

Original: u − 1 for u > 1 .
Revised: u − 1 for u > 1 .

Original: Setting back λn 2 in place of λ n , we get
Revised: Setting back λn 2 in place of λ n , we get

Original: [Equation]
Revised: [Equation]

Original: We are now ready to minimize the following objective, where the moment term is absent since it does [Table] [Table]
Revised: We are now ready to minimize the following objective, where the moment term is absent since it does not depend on σ 2ˆ ρ

Original: [Equation]
Revised: [Equation]

Original: The derivative of the objective function w.r.t.
Revised: The derivative of the objective function w.r.t.

Original: σ 2ˆ ρ simply writes
Revised: σ 2ˆ ρ simply writes

Original: [Equation]
Revised: [Equation]

Original: Now setting the above to zero we get the typical prior-to-posterior update for a Gaussian precision term
Revised: Now setting the above to zero we get the typical prior-to-posterior update for a Gaussian precision term

Original: [Equation]
Revised: [Equation]

Original: The proposition is proven by replacing the terms in the bound from Theorem 1 with the resultsderived above.
Revised: The proposition is proven by replacing the terms in the bound from Theorem 1 with the results derived above.

Original: B ExperimentsB.1 Experimental setup
Revised: B Experiments B.1 Experimental setup

Original: We run our experiments on GPUs of the type NVIDIA GeForce RTX2080ti, on our local cluster.
Revised: We run our experiments on GPUs of the type NVIDIA GeForce RTX2080ti, on our local cluster.

Original: The total computation time was approximately 125 GPU hours.
Revised: The total computation time was approximately 125 GPU hours.

Original: In the following list we include thelibraries and datasets that we used together with their corresponding licences
Revised: In the following list we include the libraries and datasets that we used together with their corresponding licences

Original: • Laplace-Redux Package [10]: MIT License • Netcal package [26]: Apache Software License • Pytorch package [36]: Modified BSD Licence • CIFAR-10 dataset [24]: MIT Licence 
Revised: • Laplace-Redux Package [10]: MIT License • Netcal package [26]: Apache Software License • Pytorch package [36]: Modified BSD Licence • CIFAR-10 dataset [24]: MIT Licence • CIFAR-100 dataset [24]: MIT Licence • SVHN dataset [34]: - • FashionMnist dataset [39]: MIT Licence

Original: • CIFAR-100 dataset [24]: MIT Licence

Original: • SVHN dataset [34]: -

Original: • FashionMnist dataset [39]: MIT Licence

Original: B.2 Dataset splits
Revised: B.2 Dataset splits

Original: For the classification datasets CIFAR-10, CIFAR-100, SVHN, FMNIST we used the standard testand train splits.
Revised: For the classification datasets CIFAR-10, CIFAR-100, SVHN, FMNIST we used the standard test and train splits.

Original: We use 10% of the data for the validation set.
Revised: We use 10% of the data for the validation set.

Original: B.3 Models
Revised: B.3 Models

Original: For the classification datasets CIFAR-10, CIFAR-100 and SVHN we use the WideResNet22 [40]architecture.
Revised: For the classification datasets CIFAR-10, CIFAR-100 and SVHN we use the WideResNet22 [40] architecture.

Original: Because the Laplace approximation does not interact well [4] with BatchNorm [21] weinstead use Fixup Initialization Zhang et al.
Revised: Because the Laplace approximation does not interact well [4] with BatchNorm [21] we instead use Fixup Initialization Zhang et al.

Original: We train our networks using the softmax activationand the cross-entropy loss.
Revised: We train our networks using the softmax activation and the cross-entropy loss.

Original: We use the SGD optimizer with learning rate η = 0 . 1 , weight decay 5e-4,and momentum 0.9 and 300 epochs.
Revised: We use the SGD optimizer with learning rate η = 0 . 1 , weight decay 5e-4, and momentum 0.9 and 300 epochs.

Original: We furthermore divide the initial learning rate by 10, at the pointof 50%, 75% and 87% of the epochs.
Revised: We furthermore divide the initial learning rate by 10, at the point of 50%, 75% and 87% of the epochs.

Original: We also use dropout with 0.4 after all the Resnet blocks.
Revised: We also use dropout with 0.4 after all the Resnet blocks.

Original: Weevaluate the NLL using the cross-entropy loss.
Revised: We evaluate the NLL using the cross-entropy loss.

Original: For the classification dataset FMNIST we use a Convolutional Network with 3 nonlinear convolutionallayers followed by 2 non-linear fully connected layers.
Revised: For the classification dataset FMNIST we use a Convolutional Network with 3 nonlinear convolutional layers followed by 2 non-linear fully connected layers.

Original: We use the SGD optimizer with learning rateη = 0 . 001 , weight decay 5e-4, and momentum 0.9 and 10 epochs.
Revised: We use the SGD optimizer with learning rate η = 0 . 001 , weight decay 5e-4, and momentum 0.9 and 10 epochs.

Original: We evaluate the NLL using thecross-entropy loss.
Revised: We evaluate the NLL using the cross-entropy loss.

Original: We do not use data augmentation in any experiment.
Revised: We do not use data augmentation in any experiment.

Original: This partially explains the problems withthe CIFAR-100 dataset.
Revised: This partially explains the problems with the CIFAR-100 dataset.

Original: In particular, in preliminary experiments (which we include further in the
Revised: In particular, in preliminary experiments (which we include further in the

Original: Appendix) both the CIFAR-10 and the CIFAR-100 dataset improve significantly in accuracy with dataaugmentation (random flips and random crops) and the matrix inversion in the CIFAR-100 KFACcase is better posed and results in significantly improved accuracy 70% over the non augmentedcounterpart.
Revised: Appendix) both the CIFAR-10 and the CIFAR-100 dataset improve significantly in accuracy with data augmentation (random flips and random crops) and the matrix inversion in the CIFAR-100 KFAC case is better posed and results in significantly improved accuracy 70% over the non augmented counterpart.

Original: [Table]
Revised: [Table]

Original: B.3.1 Additional notes on bound evaluation
Revised: B.3.1 Additional notes on bound evaluation

Original: We try to make our bounds as tight as possible.
Revised: We try to make our bounds as tight as possible.

Original: To do this we try to control the term ∥ w ˆ ρ − w π ∥2 2which typically dominates the bound.
Revised: To do this we try to control the term ∥ w ˆ ρ − w π ∥ 2which typically dominates the bound.

Original: We follow for all tasks a variation of the approach of Dziugaiteet al.
Revised: We follow for all tasks a variation of the approach of Dziugaite et al.

Original: Specifically we use Z train to learn a prior mean w π then we set, w ˆ ρ = w π , such that∥ w ˆ ρ − w π ∥ 22{+
+} = 0 .
Revised: Specifically we use Z train to learn a prior mean w π then we set, w ˆ ρ = w π , such that ∥ w ˆ ρ − w π ∥ {+
+} = 0 .

Original: Note that we can still evaluate a valid bound so long as we set ( X, Y ) in Theorem1 to be independent of the prior mean.
Revised: Note that we can still evaluate a valid bound so long as we set ( X, Y ) in Theorem 1 to be independent of the prior mean.

Original: This is the reason why we separated a part of the training setin the form of Z validation .
Revised: This is the reason why we separated a part of the training set in the form of Z validation .

Original: We thus set ( X, Y ) =
Revised: We thus set ( X, Y ) =

Original: Z validation in Theorem 1.
Revised: Z validation in Theorem 1.

Original: In our experiments we test multiple values of λ and σ 2 π .
Revised: In our experiments we test multiple values of λ and σ 2 π .

Original: Typically one would need to take a unionbound over a grid on these parameters so as for the generalization bound to be valid [13].
Revised: Typically one would need to take a union bound over a grid on these parameters so as for the generalization bound to be valid [13].

Original: Howeverthis typically costs only logarithmically to the actual bound.
Revised: However this typically costs only logarithmically to the actual bound.

Original: We ignore these calculations as ourbounds are in general quite loose anyway, and these calculations would result in additional termswould make the final bound even more complex.
Revised: We ignore these calculations as our bounds are in general quite loose anyway, and these calculations would result in additional terms would make the final bound even more complex.

Original: For the bounds to be valid, one would typically want to show concentration inequalities such thatthe Monte Carlo estimates of the Empirical Risk and the Moment terms concentrate close to the trueexpected value with high probability.
Revised: For the bounds to be valid, one would typically want to show concentration inequalities such that the Monte Carlo estimates of the Empirical Risk and the Moment terms concentrate close to the true expected value with high probability.

Original: We do not provide such guarantees.
Revised: We do not provide such guarantees.

Original: Note however that, atleast for the Empirical Risk term, our sample size of m = 100 from the posterior distribution overweights is a sample size that is typically used in practice and provides good estimates.
Revised: Note however that, at least for the Empirical Risk term, our sample size of m = 100 from the posterior distribution over weights is a sample size that is typically used in practice and provides good estimates.

Original: We have triedto balance sampling sufficiently to approximate the expectation on the one hand, and also not toomuch such that the computations become prohibitive.
Revised: We have tried to balance sampling sufficiently to approximate the expectation on the one hand, and also not too much such that the computations become prohibitive.

Original: B.4 Additional classification resultsB.4.1 NLL results
Revised: B.4 Additional classification results B.4.1 NLL results

Original: We plot in Figure 3 the standard Isotropic and standard KFAC cases for the NLL.
Revised: We plot in Figure 3 the standard Isotropic and standard KFAC cases for the NLL.

Original: We find that all threecases of temperatures (cold posterior, warm posterior, as well as posterior with λ = 1 ) can be optimal,for varying datasets.
Revised: We find that all three cases of temperatures (cold posterior, warm posterior, as well as posterior with λ = 1 ) can be optimal, for varying datasets.

Original: Furthermore the test behaviour is dominated again by a sharp improvement aswe decrease the posterior variance ( λ ↑ ) followed by a plateau.
Revised: Furthermore the test behaviour is dominated again by a sharp improvement as we decrease the posterior variance ( λ ↑ ) followed by a plateau.

Original: An optimal λ strictly less than + ∞(when it exists) results in only a relatively modest variation of the overall trend.
Revised: An optimal λ strictly less than + ∞ (when it exists) results in only a relatively modest variation of the overall trend.

Original: Thus, we believe thatour bounds would be informative even in a hypothetical scenario where they would not be able tocapture these optimal λ < + ∞ .
Revised: Thus, we believe that our bounds would be informative even in a hypothetical scenario where they would not be able to capture these optimal λ < + ∞ .

Original: B.4.2 ECE results
Revised: B.4.2 ECE results

Original: We plot in Figure 4 the standard Isotropic and standard KFAC cases for the ECE.
Revised: We plot in Figure 4 the standard Isotropic and standard KFAC cases for the ECE.

Original: Even without dataaugmentation and even when we optimize the prior variance using the marginal likelihood, we findthat all three cases of temperatures (cold posterior, warm posterior, as well as posterior with λ = 1 )can be optimal, for varying datasets.
Revised: Even without data augmentation and even when we optimize the prior variance using the marginal likelihood, we find that all three cases of temperatures (cold posterior, warm posterior, as well as posterior with λ = 1 ) can be optimal, for varying datasets.

Original: Unfortunately we are not aware of approaches to directly boundthe ECE.
Revised: Unfortunately we are not aware of approaches to directly bound the ECE.

Original: In Figure 4 the ECE is notable for having a significantly different behaviour from the NLLand the 0-1 Loss.
Revised: In Figure 4 the ECE is notable for having a significantly different behaviour from the NLL and the 0-1 Loss.

Original: Better calibration in terms of ECE than a simple MAP estimate is one of the purported main benefitsof the Bayesian paradigm.
Revised: Better calibration in terms of ECE than a simple MAP estimate is one of the purported main benefits of the Bayesian paradigm.

Original: In Figure 5 we plot the Pareto front of the test 0-1 Loss with respect to thetest ECE.
Revised: In Figure 5 we plot the Pareto front of the test 0-1 Loss with respect to the test ECE.

Original: The top row is the standard Isotropic case and the bottom row is the standard KFAC case.
Revised: The top row is the standard Isotropic case and the bottom row is the standard KFAC case.

Original: We see that in most cases there is a clear tradeoff between the test 0-1 Loss and the test ECE.
Revised: We see that in most cases there is a clear tradeoff between the test 0-1 Loss and the test ECE.

Original: Theseresults might be relevant for the applicability of the Laplace approximation for improving the ECE,in that it seems that we cannot achieve a clear improvement in ECE without hurting test accuracy.
Revised: These results might be relevant for the applicability of the Laplace approximation for improving the ECE, in that it seems that we cannot achieve a clear improvement in ECE without hurting test accuracy.

Original: B.4.3 Augmentation results
Revised: B.4.3 Augmentation results

Original: In Figure 6 we see that data augmentation (random flips and crops) results in better test accuracyand makes the matrix inversion in the Laplace approximation better posed such that the accuracy on
Revised: In Figure 6 we see that data augmentation (random flips and crops) results in better test accuracy and makes the matrix inversion in the Laplace approximation better posed such that the accuracy on

Original: CIFAR-100 is within a normal range.
Revised: CIFAR-100 is within a normal range.

Original: mean, as well as 10 MAP trials , along with the validation 0-1 Loss (we denote λ = 1 by ) for the Standard Isotropic Laplace posterior (top) and standard KFAC (bottom) for CIFAR-10 and CIFAR-100 with data augmentation (random flips and crops).
Revised: mean, as well as 10 MAP trials , along with the validation 0-1 Loss (we denote λ = 1 by ) for the Standard Isotropic Laplace posterior (top) and standard KFAC (bottom) for CIFAR-10 and CIFAR-100 with data augmentation (random flips and crops).

Original: The performance on both improves significantly and the Laplace approximation becomes better posed.
Revised: The performance on both improves significantly and the Laplace approximation becomes better posed.