Original: Spike-and-Slab Probabilistic Backpropagation: When Smarter Approximations Make No Difference
Revised: Spike-and-Slab Probabilistic Backpropagation: When Smarter Approximations Make No Difference

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) isan approximate Bayesian inference method for deep neural networks, using amessage-passing framework.
Revised: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) is an approximate Bayesian inference method for deep neural networks, using a message-passing framework.

Original: These messages—which correspond to distributionsarising as we propagate our input through a probabilistic neural network—are approximated as Gaussian.
Revised: These messages—which correspond to distributions arising as we propagate our input through a probabilistic neural network—are approximated as Gaussian.

Original: However, in practice, the exact distributions may behighly non-Gaussian.
Revised: However, in practice, the exact distributions may be highly non-Gaussian.

Original: In this paper, we propose a more realistic approximationbased on a spike-and-slab distribution.
Revised: In this paper, we propose a more realistic approximation based on a spike-and-slab distribution.

Original: Unfortunately, in this case, better approximation of the messages does not translate to better downstream performance.
Revised: Unfortunately, in this case, better approximation of the messages does not translate to better downstream performance.

Original: Wepresent results comparing the two schemes and discuss why we do not see a benefitfrom this spike-and-slab approach
Revised: We present results comparing the two schemes and discuss why we do not see a benefit from this spike-and-slab approach.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks are flexible non-linear models common to domains with complicated datarelationships.
Revised: Deep neural networks are flexible non-linear models common to domains with complicated data relationships.

Original: However, despite having many unknown model parameters, many neural networksdo not attempt to represent model (epistemic) or data (aleatoric) uncertainty.
Revised: However, despite having many unknown model parameters, many neural networks do not attempt to represent model (epistemic) or data (aleatoric) uncertainty.

Original: Bayesian approachesto neural networks could capture this uncertainty but are typically rendered intractable in closedform.
Revised: Bayesian approaches to neural networks could capture this uncertainty but are typically rendered intractable in closedform.

Original: MCMC-based methods (Neal, 2012; Cobb and Jalaian, 2021) offer asymptotic guarantees butare typically slow to converge.
Revised: MCMC-based methods (Neal, 1995; Cobb and Jalaian, 2021) offer asymptotic guarantees but are typically slow to converge.

Original: Instead, it is typical to use approximate inference algorithms, suchas those based on Laplace approximations (MacKay, 1992; Daxberger et al., 2021) or variationalinference (Graves, 2011; Blundell et al., 2015).
Revised: Instead, it is typical to use approximate inference algorithms, such as those based on Laplace approximations (MacKay, 1992; Daxberger et al., 2021) or variational inference (Graves, 2011; Blundell et al., 2015).

Original: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) is an example of anapproximation-based approach for Bayesian deep learning.
Revised: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) is an example of an approximation-based approach for Bayesian deep learning.

Original: Like many such approaches, it uses amean-field approximation to the posterior, which is inferred via a message-passing algorithm basedon assumed density filtering (Opper, 1999).
Revised: Like many such approaches, it uses a mean-field approximation to the posterior, which is inferred via a message-passing algorithm based on assumed density filtering (Opper, 1999).

Original: The resulting algorithm is reminiscent of backpropagation; however, rather than propagating a single function estimate through a neural network, PBPpropagates distributions representing our estimated posterior uncertainty.
Revised: The resulting algorithm is reminiscent of backpropagation; however, rather than propagating a single function estimate through a neural network, PBP propagates distributions representing our estimated posterior uncertainty.

Original: PBP approximates thesedistributions, or “messages,” using a Gaussian, whose parameters are a known function of the meansand variances of the per-weight distributions, allowing us to update those weight parameters usinggradient information in a backwards pass.
Revised: PBP approximates these distributions, or “messages,” using a Gaussian, whose parameters are a known function of the means and variances of the per-weight distributions, allowing us to update those weight parameters using gradient information in a backwards pass.

Original: In practice, the true messages can be highly non-Gaussian, since they have been propagated througha ReLU or similar nonlinearity.
Revised: In practice, the true messages can be highly non-Gaussian, since they have been propagated through a ReLU or similar nonlinearity.

Original: Using a Gaussian approximation risks ignoring sparsity inherentin the true message structure.
Revised: Using a Gaussian approximation risks ignoring sparsity inherent in the true message structure.

Original: In this paper, we show that the PBP algorithm can be modified touse sparse messages, parameterized using a spike-and-slab distribution.
Revised: In this paper, we show that the PBP algorithm can be modified to use sparse messages, parameterized using a spike-and-slab distribution.

Original: Computational costs are notsignificantly increased over Gaussian messages.
Revised: Computational costs are not significantly increased over Gaussian messages.

Original: Unfortunately, we discover what many before us have discovered in different contexts: Gaussianapproximations usually work pretty well.
Revised: Unfortunately, we discover what many before us have discovered in different contexts: Gaussian approximations usually work pretty well.

Original: Closer investigation shows that, if we use bias terms in our neural network (as is typical), any theoretical advantage of our spike-and-slab approach evaporates.
Revised: Closer investigation shows that, if we use bias terms in our neural network (as is typical), any theoretical advantage of our spike-and-slab approach evaporates.

Original: In the absence of a bias term, our approximation does differ from the standard PBP approach, andat a per-message level does indeed provide better approximations to the true message.
Revised: In the absence of a bias term, our approximation does differ from the standard PBP approach, and at a per-message level does indeed provide better approximations to the true message.

Original: However,this advantage does not translate into a significant difference in overall performance.
Revised: However, this advantage does not translate into a significant difference in overall performance.

Original: While this is anegative result, and while it remains possible that alternative approximations could yield improvedperformance, our work indirectly highlights the fact that Gaussian approximations to non-Gaussiandistributions are often a good choice in practice.
Revised: While this is a negative result, and while it remains possible that alternative approximations could yield improved performance, our work indirectly highlights the fact that Gaussian approximations to non-Gaussian distributions are often a good choice in practice.

Original:  Probabilistic backpropagation
Revised:  Probabilistic backpropagation

Original: We consider the setting of feed-forward neural networks (FFNNs) with ReLU activation σ ( x ) =max( x, 0) , such that
Revised: We consider the setting of feed-forward neural networks (FFNNs) with ReLU activation σ ( x ) = max( x, 0) , such that

Original: [Equation]
Revised: [Equation]

Original: (1) with [ a ; b ] indicating concatenation, n ℓ indicating the number of nodes in layer ℓ , and z (0) ( x ) = x .
Revised: (1) with [ a ; b ] indicating concatenation, n ℓ indicating the number of nodes in layer ℓ , and z (0) ( x ) = x .

Original: Let W =
Revised: Let W =

Original: ( W ( ℓ ) )
Revised: ( W ( ℓ ) )

Original: Lℓ =1 where W ( ℓ ) =
Revised: Lℓ =1 where W ( ℓ ) =

Original: ( w ( ℓ ) ij ) n ℓ − 1 +1 i = nℓ j
Revised: ( w ( ℓ ) ij ) n ℓ − 1 +1 i = n

Original: =
Revised: [Equation]

Original:  be the set of all weights (which includes thebiases).
Revised: =1 be the set of all weights (which includes the biases).

Original: 1 We include a scaling factor of √ n ℓ − 1 + 1 to make the scale of the input to each neuroninvariant to the size of the previous layer, following Hernández-Lobato and Adams (2015).
Revised: 1 We include a scaling factor of √ n ℓ − 1 + 1 to make the scale of the input to each neuron invariant to the size of the previous layer, following Hernández-Lobato and Adams (2015).

Original: To make the model Bayesian, we place a spherical Gaussian prior on the weights, meaning thez ( ℓ ) ( x ) and ˆ y are now random, and we assume y
Revised: To make the model Bayesian, we place a spherical Gaussian prior on the weights, meaning the z ( ℓ ) ( x ) and ˆ y are now random, and we assume y

Original: i ∼ N (ˆ y i , γ − 1 ) .
Revised: i ∼ N (ˆ y i , γ − 1 ) .

Original: Since the number of weightscan be large, and the ReLU nonlinearity removes the possibility of analytic tractability, posteriorinference can be computationally challenging.
Revised: Since the number of weights can be large, and the ReLU nonlinearity removes the possibility of analytic tractability, posterior inference can be computationally challenging.

Original: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) is an approximate inference technique appropriate for this setting.
Revised: Probabilistic backpropagation (PBP, Hernández-Lobato and Adams, 2015) is an approximate inference technique appropriate for this setting.

Original: PBP uses assumed density filtering (ADF, Opper, 1999)to update a mean-field approximation to the true posterior,[-
-]
Revised: PBP uses assumed density filtering (ADF, Opper, 1999) to update a mean-field approximation to the true posterior,[-
-]

Original: [Equation]
Revised: [Equation]

Original: The mean and variance parameters of the weights ( m ( ℓ ) ij and v ( ℓ ) ij ) are updated by propagating uncertainty through the network, i.e., by evaluating the distributions over the representations z ( ℓ ) ( x ) .
Revised: The mean and variance parameters of the weights ( m ( ℓ ) ij and v ( ℓ ) ij ) are updated by propagating uncertainty through the network, that is, by evaluating the distributions over the representations z ( ℓ ) ( x ) .

Original: Weapproximate these distributions with Normal distributions, obtained via moment matching.
Revised: PBP approximates these distributions with Gaussian distributions, obtained through moment matching.

Revised: Leveraging some properties of Gaussian distributions (Minka, 2001), PBP is able to update the parameters of the approximate posterior in an approach analogous to backpropagation.

Revised: Finally, PBP also incorporates an expectation propagation step to further refine the approximate posterior after each full pass through the training data.

Original: Several extensions to the PBP framework have provided improvements, such as including approximations appropriate for classification (Ghosh et al., 2016), incorporating minibatching (Benatan and
Revised: Several extensions to the PBP framework have provided improvements, such as including approximations appropriate for classification (Ghosh et al., 2016), incorporating minibatching (Benatan and

Original: Pyzer-Knapp, 2018), and using non-diagonal Gaussians as the approximating distribution (Sun et al.,2017).
Revised: Pyzer-Knapp, 2018), and using non-diagonal Gaussians as the approximating distribution (Sun et al., 2017).

Original: Similar approaches have been used in a variational inference context (Roth and Pernkopf,2016; Wu et al., 2019; Dera et al., 2019; Haußmann et al., 2020) and in a hybrid Bayesian/maximumlikelihood context (Gast and Roth, 2018).
Revised: Similar approaches have been used in a variational inference context (Roth and Pernkopf, 2016; Wu et al., 2019; Dera et al., 2019; Haußmann et al., 2020) and in a hybrid Bayesian/maximum likelihood context (Gast and Roth, 2018).

Original: 2.1 Limitations of a Gaussian approximation
Revised: 2.1 Limitations of a Gaussian approximation

Original: PBP approximates the distribution of z ( ℓ ) ( x ) —i.e., the “messages” in the ADF algorithm—using
Revised: PBP approximates the distribution of z ( ℓ ) ( x ) —i.e., the “messages” in the ADF algorithm—using

Original: Gaussians.
Revised: Gaussians.

Original: However, the ReLU nonlinearity in Equation 1 means that the true distributions canbe highly non-Gaussian.
Revised: However, the ReLU nonlinearity in Equation 1 means that the true distributions can be highly non-Gaussian.

Original: As a simple example, consider the distribution over the j th element ofthe first-layer representation of an input x , z (1) j ( x ) = σ ( w (1) j ⊤ [ x ; 1]) .
Revised: As a simple example, consider the distribution over the j th element of the first-layer representation of an input x , z (1) j ( x ) = σ ( w (1) j ⊤ [ x ; 1]) .

Original: We are approximating theposterior distributions over w (1) ij using Gaussians, meaning that the distribution implied by w ⊤ j [ x ; 1]is also Gaussian.
Revised: We are approximating the posterior distributions over w (1) ij using Gaussians, meaning that the distribution implied by w ⊤ j [ x ; 1] is also Gaussian.

Original: However, once this has passed through a ReLU, the distribution over z (1) j ( x ) is amixture of a Dirac delta distribution (or “spike”) at x = 0 and a Gaussian truncated to the domain(0 , ∞ ) .
Revised: However, once this has passed through a ReLU, the distribution over z (1) j ( x ) is a mixture of a Dirac delta distribution (or “spike”) at x = 0 and a Gaussian truncated to the domain (0 , ∞ ) .

Original: See Figure 1 for a demonstration.
Revised: See Figure 1 for a demonstration.

Original:  Spike-and-slab probabilistic backpropagation
Revised:  Spike-and-slab probabilistic backpropagation

Original: An exact implementation of ADF in the FFNN would send messages based on the true distributionover the z ( ℓ ) ( x ) , given the current approximating distribution q ( W ) .
Revised: An exact implementation of ADF in the FFNN would send messages based on the true distribution over the z ( ℓ ) ( x ) , given the current approximating distribution q ( W ) .

Original: We will refer to this truedistribution as q ( z ( ℓ ) ) 3 .
Revised: We will refer to this true distribution as q ( z ( ℓ ) ) 3 .

Original: This is computationally infeasible: while we can calculate this distributionfor a single layer where the input is the observed x (as shown above), on later layers the inputs arethemselves the propagated distributions.
Revised: This is computationally infeasible: while we can calculate this distribution for a single layer where the input is the observed x (as shown above), on later layers the inputs are themselves the propagated distributions.

Original: PBP chooses to approximate these with diagonal Gaussiandistributions (cid:101)
Revised: PBP chooses to approximate these with diagonal Gaussian distributions (cid:101)

Original: q PBP ( z ( ℓ ) ) , obtained via moment matching.
Revised: q PBP ( z ( ℓ ) ) , obtained via moment matching.

Original: We propose using a more sophisticated approximation of the messages q ( z ( ℓ ) ) .
Revised: We propose using a more sophisticated approximation of the messages q ( z ( ℓ ) ) .

Original: At the first layer, weknow that q ( z (1) ) is a spike-and-slab distribution with truncated Gaussian slab.
Revised: At the first layer, we know that q ( z (1) ) is a spike-and-slab distribution with truncated Gaussian slab.

Original: We choose to modelthe resulting sparsity directly using a spike-and-slab distribution with a (non-truncated) Gaussianslab, such that for node i in layer ℓ , we have
Revised: We choose to model the resulting sparsity directly using a spike-and-slab distribution with a (non-truncated) Gaussian slab, such that for node i in layer ℓ , we have

Original: [Equation]
Revised: [Equation]

Original: (3) with slab probability (cid:101) ρ ( ℓ ) i , slab mean (cid:101) m ( ℓ ) i , and slab variance (cid:101) v ( ℓ ) i .
Revised: with slab probability (cid:101) ρ ( ℓ ) i , slab mean (cid:101) m ( ℓ ) i , and slab variance (cid:101) v ( ℓ ) i .

Original: We refer to the resulting algorithmas Spike-and-Slab PBP (SSPBP).
Revised: We refer to the resulting algorithm as Spike-and-Slab PBP (SSPBP).

Original: 3.1 Approximating messages using a spike-and-slab
Revised: 3.1 Approximating messages using a spike-and-slab

Original: We wish to approximate some distribution p , with a spike-and-slab distribution q of the form of
Revised: We wish to approximate some distribution p , with a spike-and-slab distribution q of the form of

Original: Equation 3.
Revised: Equation 3.

Original: We can do so by minimizing the Kullback-Leibler divergence KL ( p ∥ q ) , yielding thefollowing parameters (see Appendix A.1 for derivation):
Revised: We can do so by minimizing the Kullback-Leibler divergence KL ( p ∥ q ) , yielding the following parameters (see Appendix A.1 for derivation):

Original: [Equation]
Revised: [Equation]

Original: Note that this is similar to the moment-matching setting of PBP: we first match the spike probability( 1 − ρ ), and then match the first and second moments. 
Revised: Note that this is similar to the moment-matching setting of PBP: we first match the spike probability ( 1 − ρ ), and then match the first and second moments. [Table]

Original: Consider the j th node of the ℓ th layer of our FFNN.
Revised: Consider the j th node of the ℓ th layer of our FFNN.

Original: Following Hernández-Lobato and Adams(2015), we augment the incoming message with a bias term, so that (cid:101) ρ ( ℓ ) n ℓ +1 = 1 , (cid:101) m ( ℓ ) n ℓ +1 = 1 , and(cid:101) v ( ℓ ) n ℓ +1 = 0 .
Revised: Following Hernández-Lobato and Adams (2015), we augment the incoming message with a bias term, so that (cid:101) ρ ( ℓ ) n ℓ +1 = 1 , (cid:101) m ( ℓ ) n ℓ +1 = 1 , and (cid:101) v ( ℓ ) n ℓ +1 = 0 .

Original: Let M ( ℓ ) =
Revised: Let M ( ℓ ) =

Original: ( m ( ℓ ) i ) n ℓ
Revised: ( m ( ℓ ) i ) n ℓ

Original: i
Revised: i

Original: =1 and m ( ℓ ) i
Revised: =1 and m ( ℓ ) i

Original: =
Revised: =

Original: ( m ( ℓ ) ij ) n ℓ − 1 +1 j = be the means of the approximate posteriors q ( w ( ℓ ) ij ) of the weights of the FFNN, and let V ( ℓ )
Revised: ( m ( ℓ ) ij ) n ℓ − 1 +1 j = be the means of the approximate posteriors q ( w ( ℓ ) ij ) of the weights of the FFNN, and let V ( ℓ )

Original: =
Revised: =

Original: ( v ( ℓ ) i ) n ℓ i =1 and v ( ℓ ) i
Revised: ( v ( ℓ ) i ) n ℓ i =1 and v ( ℓ ) i

Original: = ( v ( ℓ ) ij ) n ℓ − 1 +1 j
Revised: = ( v ( ℓ ) ij ) n ℓ − 1 +1 j

Original: = be the corresponding variances (Equation 2).
Revised: = be the corresponding variances (Equation 2).

Original: In the absence of a nonlinearity (e.g., for a final regressionlayer), we approximate the message z ( ℓ ) j with a spike-and-slab distribution, with slab probability(cid:101)
Revised: In the absence of a nonlinearity (e.g., for a final regression layer), we approximate the message z ( ℓ ) j with a spike-and-slab distribution, with slab probability [Equation]

Original: ρ ( ℓ, linear ) j , slab mean (cid:101) m (ℓ, linear ) j , and slab variance (cid:101) v ( ℓ, linear ) j where
Revised: , linear [Equation] , slab mean (cid:101) m (

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] where ◦ indicates the element-wise Hadamard product and
Revised: , linear

Original: [Equation]
Revised: [Equation]

Original: ◦ (cid:101) 
Revised: , and slab variance (cid:101) v ( ℓ, linear ) j where

Original: [Equation]
Revised: [Equation]

Original: (cid:101)
Revised: where ◦ indicates the element-wise Hadamard product and

Original: [Equation]
Revised: [Equation]

Original: (cid:17) .

Original: If we pass this through a ReLU, we have
Revised: If we pass this through a ReLU, we have

Original: [Equation]
Revised: [Equation]

Original: with intermediate values α j = (cid:101) m ( ℓ, linear ) j / (cid:113) (cid:101) v ( ℓ, linear ) j and γ j = ϕ ( α j )/Φ
Revised: with intermediate values α j = (cid:101) m ( ℓ, linear ) j / (cid:113) (cid:101) v ( ℓ, linear ) j and γ j = ϕ ( α j )/Φ

Original: ( α j ) , where ϕ and Φ arethe PDF and CDF of a standard Gaussian, respectively.
Revised: ( α j ) , where ϕ and Φ are the PDF and CDF of a standard Gaussian, respectively.

Original: Derivations are provided in Appendix A.
Revised: Derivations are provided in Appendix A.

Original:  Empirical analysis
Revised:  Empirical analysis

Original: We begin by numerically validating our approximation, before exploring how well it performs whenused in a Bayesian FFNN.
Revised: We begin by numerically validating our approximation before exploring how well it performs when used in a Bayesian FFNN.

Original: Code can be found at the anonymous repository https://github.com/SSPBP/SSPBPcode .
Revised: Code can be found at the anonymous repository https://github. com/SSPBP/SSPBPcode .

Original: All hyperparameters follow those used by Hernández-Lobato and Adams(2015), unless otherwise stated.
Revised: All hyperparameters follow those used by Hernández-Lobato and Adams (2015), unless otherwise stated.

Original: 4.1 Quality of the spike-and-slab approximation
Revised: 4.1 Quality of the spike-and-slab approximation

Original: To explore why the spike-and-slab approximation should yield a better representation of the propagated probability distributions than a Gaussian, we performed a simulation study.
Revised: To explore why the spike-and-slab approximation should yield a better representation of the propagated probability distributions than a Gaussian, we performed a simulation study.

Original: Here, we explorea ReLU transformation T = ReLU ( XW ) applied to the product of two independent Gaussians
Revised: Here, we explore a ReLU transformation T = ReLU ( XW ) applied to the product of two independent Gaussians

Original: X and W .
Revised: X and W .

Original: We apply the PBP and SSPBP approximations to determine how the two approacheswould approximate the resultant distribution, then we compare 10,000 samples of the approximatedistributions to a ground truth sample by computing the maximum mean discrepancy (MMD) with asquared exponential kernel K ( x, y ) =
Revised: We apply the PBP and SSPBP approximations to determine how the two approaches would approximate the resultant distribution, then we compare 10,000 samples of the approximate distributions to a ground truth sample by computing the maximum mean discrepancy (MMD) with a squared exponential kernel K ( x, y ) =

Original: exp( − γ ( x − y ) 2 ) , setting γ = 1 .
Revised: exp( − γ ( x − y ) 2 ) , setting γ = 1 .

Original: The results are summarizedin Table 1.
Revised: The results are summarized in Table 1.

Original: The first two columns show the parameters of the underlying distributions, and the thirdcolumn shows the percent of the distribution that is saturated to zero by the ReLU nonlinearity.
Revised: The first two columns show the parameters of the underlying distributions, and the third column shows the percent of the distribution that is saturated to zero by the ReLU nonlinearity.

Original: Thefourth column shows the MMD between two size-10,000 samples from the true distribution, to givean idea of the scale of the differences.
Revised: The fourth column shows the MMD between two size-10,000 samples from the true distribution, to give an idea of the scale of the differences.

Original: Next, we see the MMD between the PBP approximationand the true distribution (column 5) and the MMD between our SSPBP approximation and the truedistribution (column 6).
Revised: Next, we see the MMD between the PBP approximation and the true distribution (column 5) and the MMD between our SSPBP approximation and the true distribution (column 6).

Original: We see from these results that, in cases where the true distribution hasnon-trivial sparsity, samples from SSPBP more closely match the true distribution compared with
Revised: We see from these results that, in cases where the true distribution has non-trivial sparsity, samples from SSPBP more closely match the true distribution compared with

Original: PBP, giving credence to the intuition that a spike-and-slab approximation should improve the PBPframework.
Revised: PBP, giving credence to the intuition that a spike-and-slab approximation should improve the PBP framework.

Original: 4.2 Evaluation as part of a Bayesian FFNN
Revised: 4.2 Evaluation as part of a Bayesian FFNN

Original:  Since our spike-and-slab approximation mimics the sparsity inherent in the FFNN, we expected itto achieve better performance.
Revised: [Table] Since our spike-and-slab approximation mimics the sparsity inherent in the FFNN, we expected it to achieve better performance.

Original: To evaluate this, we compared our approach to PBP, modifying the[Table] existing official Theano (Theano Development Team, 2016) implementation, which was released under a BSD 3-Clause “New" or “Revised" License.
Revised: To evaluate this, we compared our approach to PBP, modifying the existing official Theano (Theano Development Team, 2016) implementation, which was released under a BSD 3-Clause “New" or “Revised" License.

Original: We applied several different model architecturesto eight regression datasets used in (Hernández-Lobato and Adams, 2015).
Revised: We applied several different model architectures to eight regression datasets used in Hernández-Lobato and Adams (2015).

Original: We trained the modelsusing 40 training epochs (without early stopping), with an 80%-20% training-test split, repeatingthe process five times for each dataset-model comparison (following the original PBP settings).
Revised: We trained the models using 40 training epochs (without early stopping), with an 80%-20% training-test split, repeating the process five times for each dataset-model comparison (following the original PBP settings).

Original: Wereport the mean and standard error of the average test set RMSE across those five repetitions in Table 2.
Revised: We report the mean and standard error of the average test set RMSE across those five repetitions in Table 2.

Original: Running all experiments in Tables 2 and 3 took under 12 hours on an Intel i5 4 core 3.2 GHzdesktop, using CPU computation.
Revised: Running all experiments in Tables 2 and 3 took under 12 hours on an Intel i5 4 core 3.2 GHz desktop, using CPU computation.

Original: As Table 2 indicates, the performance is comparable across the two models.
Revised: As Table 2 indicates, the performance is comparable across the two models.

Original: Disappointingly, thereis no significant difference between the two approaches.
Revised: Disappointingly, there is no significant difference between the two approaches.

Original: A hint at why this is can be seen by considering the estimate of the spike probability in Equation 4.
Revised: A hint at why this is can be seen by considering the estimate of the spike probability in Equation 4.

Original: Our approximation only captures spikes at zero.
Revised: Our approximation only captures spikes at zero.

Original: However, if our FFNN includes bias terms, this biaswill translate spikes at zero in z ( ℓ − 1) to non-zero locations in z ( ℓ ) , meaning our approximation willnot capture them.
Revised: However, if our FFNN includes bias terms, this bias will translate spikes at zero in z ( ℓ − 1) to non-zero locations in z ( ℓ ) , meaning our approximation will not capture them.

Original: This means all mass is placed on the slab, rendering our approximation identicalto modeling solely with a Gaussian as in the original PBP framework.
Revised: This means all mass is placed on the slab, rendering our approximation identical to modeling solely with a Gaussian as in the original PBP framework.

Original: We show this formally in
Revised: We show this formally in

Original: Appendix B. The variation in Table 2 is likely due to numerical instability, using different randomseeds, or other similar issues, not as a result of improving the internal approximations of the PBPframework.
Revised: Appendix B. The variation in Table 2 is likely due to numerical instability, using different random seeds, or other similar issues, not as a result of improving the internal approximations of the PBP framework.

Original: The algorithms do however differ if we do not include a bias term.
Revised: The algorithms do however differ if we do not include a bias term.

Original: In Table 3 we repeat our experiments without a bias term.
Revised: In Table 3 we repeat our experiments without a bias term.

Original: Unfortunately, despite the fact that we are now indeed comparingdifferent update rules, there remains no significant difference between the algorithms.
Revised: Unfortunately, despite the fact that we are now indeed comparing different update rules, there remains no significant difference between the algorithms.

Original: We hypothesise that this is because, in practice, the slab probabilities  ρ ( ℓ, linear )  tend to be close to one  in all butthe most extreme cases. 
Revised: We hypothesise that this may be because the slab probabilities (cid:101) ρ ( ℓ, linear ) would tend toward one given that the product in Equation 4 will tend toward zero in all but the most extreme cases. To test this,

Original: [Table]
Revised: [Table]

Original: Energy Efficiency
Revised: we compute (cid:101) ρ ( ℓ, linear ) for the linear layers in several architectures trained on the Boston dataset. See Table 4 for results.

Original: [Table]
Revised: Note that (cid:101) ρ (1 , linear ) is always one by construction, since the signal has not passed through any nonlinearities. For later layers, even though the signal has passed through nonlinearities, we see that the slab probability does tend towards one in practice, even for narrow networks

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Discussion
Revised:  Discussion

Original: In this paper
Revised: In this paper

Revised: [Table]

Revised: we presented SSPBP

Original: [Table]
Revised: [Table]

Original: we presented SSPBP, a spike-and-slab variant of probabilistic backpropagation, ultimately determining that empirically and analytically, it made no difference in its current formulation,despite seeming to be a more intuitive approximation for the problem.
Revised: a spike-and-slab variant of probabilistic backpropagation. Ultimately [Table] we determine that, empirically and analytically, the use of a spike-and-slab approximation does not improve performance, despite seeming to be a more intuitive approximation for the problem.

Original: While using a spike-and-slabapproximate posterior distribution for model parameters could—in theory—provide better results ina bias-free FFNN, our investigation of this setting casts doubt on this intuition: the spike-and-spabapproximation is able to model sparsity inherent to the ReLU activation function but fails to produce better empirical results.
Revised: While using a spike-and-slab approximate posterior distribution for model parameters could— in theory—provide better results in a bias-free FFNN, our investigation of this setting casts doubt on this intuition: the spike-and-slab approximation is able to model sparsity inherent to the ReLU activation function but fails to produce better empirical results, likely due to the fact that, in practice, the slab probability tends to saturate towards one.

Original: Possible future directions could include assessing alternative approximations or incorporating spike-and-slab approximations in formulations of approximate Bayesianinference like variational inference; however it appears that a Gaussian approximation is hard tobeat.
Revised: Possible future directions could include assessing alternative approximations or incorporating spike-and-slab approximations in formulations of approximate Bayesian inference like variational inference; however, it appears that a Gaussian approximation is hard to beat Checklist

Original: A Approximating messages using a spike-and-slab distribution
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [No] Since the paper explores unsuccessful alternatives to an existing algorithm, we do not believe there to be any negative societal impacts.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] We include a link to an anonymous GitHub repository.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] Unless otherwise stated, we used the same parameter settings as Hernández-Lobato and Adams (2015).

Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [Yes] We included license information where available.

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: We take the following for our approximating distribution q , that is, the spike and (non-central) slab:(cid:101)
Revised: [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [No] All datasets are commonly used and publicly available and do not include human-level information. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] All datasets are commonly used and publicly available and do not include human-level information, or text/images.

Original: q ( z ; (cid:101) ρ, (cid:101) m, (cid:101) v ) =
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: (1 − (cid:101) ρ ) δ 0 ( z ) + (cid:101) ρ N ( z ; (cid:101) m, (cid:101) v ) .
Revised: [N/A] A Approximating messages using a spike-and-slab distribution We take the following for our approximating distribution q , that is, the spike and (non-central) slab: [Equation]

Original:  We seek to minimize KL ( p ∥ (cid:101) q ) with respect to (cid:101) ρ, (cid:101) m, (cid:101) v .
Revised: [Equation] We seek to minimize KL ( p ∥ (cid:101) q ) with respect to (cid:101) ρ, (cid:101) m, (cid:101) v .

Original: Were (cid:101)
Revised: Were (cid:101)

Original: q a Gaussian, this would be solvedby moment matching (cid:101) m =
Revised: q a Gaussian, this would be solved by moment matching (cid:101) m =

Original: E p
Revised: E p

Original: [ X ] and (cid:101) v = V p
Revised: [ X ] and (cid:101) v = V p

Original: [ X ] (Minka, 2001).
Revised: [ X ] (Minka, 2001).

Original: Here, we derive the appropriatevalues of (cid:101) ρ, (cid:101) m, (cid:101) v .
Revised: Here, we derive the appropriate values of (cid:101) ρ, (cid:101) m, (cid:101) v .

Original: For simplicity, and since we make use of a mean-field approximation, we’ll focuson a univariate version, similar to PBP’s approach.min
Revised: For simplicity, and since we make use of a mean-field approximation, we’ll focus on a univariate version, similar to PBP’s approach:

Original: [Equation]
Revised: [Equation]

Original: We begin by giving values for (cid:101) m , (cid:101) v and (cid:101) ρ in terms of the mean, variance, and probability at zero ofthe distribution being approximated.
Revised: We begin by giving values for (cid:101) m , (cid:101) v and (cid:101) ρ in terms of the mean, variance, and probability at zero of the distribution being approximated.

Original: A.1 Approximating a distribution with known mean, variance and probability of zeroSlab

Original:  meanparameter (cid:101) m
Revised: Slab mean (cid:101) m

Original: We begin by seeking to minimize KL ( p ∥ (cid:101) q ) with respect to the slab’smean parameter, (cid:101) m .
Revised: We begin by seeking to minimize KL ( p ∥ (cid:101) q ) with respect to the slab’s mean parameter, (cid:101) m :

Original: [Equation]
Revised: [Equation]

Original: The integral in the final line of Equation 5 is nontrivial.
Revised: The integral in the final line of Equation 5 is nontrivial.

Original: We approximate this term by considering thedelta function as a limit of the uniform distribution u a ( z ) =
Revised: We approximate this term by considering the delta function as a limit of the uniform distribution u a ( z ) =

Original: Unif ( z ; [ − 1 / 2 a, 1 / 2 a ]) , such that asa → ∞ , we recover the delta function at 0 δ 0 ( z ) .
Revised: Unif ( z ; [ − 1 / 2 a, 1 / 2 a ]) , such that as a → ∞ , we recover the delta function at 0 δ 0 ( z ) .

Original: Because u a is 0 outside the range [ − 1 / 2 a, 1 / 2 a ] ,we can restrict the domain of the integral:
Revised: Because u a is 0 outside the range [ − 1 / 2 a, 1 / 2 a ] [Equation] we can restrict the domain of the integral:

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that (1 − (cid:101) ρ ) a ≫ ρ N ( z ; (cid:101) m, (cid:101) v ) (as a → ∞ ), we have 
Revised: Under the assumption that (1 − (cid:101) ρ ) a ≫ ρ N ( z ; (cid:101) m, (cid:101) v ) (as a → ∞ ) [Equation] we have I

Original: [Equation]
Revised: [Equation]

Original: Setting the derivative in Equation 5 to zero, we have
Revised: Setting the derivative in Equation 5 to zero

Original: [Equation]
Revised: [Equation]

Original: This yields our solution
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: This yields our solution (cid:101) m =

Revised: E p [ Z ] 1 − P p [ Z = 0] .

Original: Slab variance parameter (cid:101) v
Revised: Slab variance parameter (cid:101) v

Original: For the variance parameter (cid:101) v , we similarly obtain:[Equation] [Equation] dz
Revised: For the variance parameter (cid:101) v , we similarly obtain:

Original: [Equation]
Revised: [Equation]

Original: Thus, we have E
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: p [Equation] [Equation] =
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: (1 − P p [ Z = 0]) + (cid:101)
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: 2 P p

Original: [ Z = 0] + 0 . This yields(cid:101) v = E p [ Z 2 ] − 2 (cid:101) m
Revised: Slab probability parameter (cid:101) ρ Finally, for the mass of the slab (cid:101) ρ , we have

Original: E p

Original: [ Z ] + (cid:101) m 2 (1 − P p

Original: [ Z = 0]) 1 − P p

Original: [ Z = 0] = 1 − P p [

Original: [Equation]
Revised: [Equation]

Original: Slab probability parameter (cid:101) ρ Finally, for the mass of the slab (cid:101) ρ , we have
Revised: z ;

Original: [Equation]
Revised: [Equation]

Original: yielding the simple and intuitive result:
Revised: yielding the simple and intuitive result:

Original: [Equation]
Revised: [Equation]

Original: A.1.1 Connection to moment matching
Revised: A.1.1 Connection to moment matching

Original: As it turns out, minimizing KL ( p ∥ q ) for the spike-and-slab turns out to be equivalent to matchingthe slab probability and then matching the first and second moments:
Revised: As it turns out, minimizing KL ( p ∥ q ) for the spike-and-slab turns out to be equivalent to matching the slab probability and then matching the first and second moments:

Original: [Equation]
Revised: [Equation]

Original: Substituting in the values of ( (cid:101) ρ, (cid:101) m, (cid:101) v ) obtained above in the KL-minimization, we find that we yieldthe highly intuitive result:
Revised: Substituting in the values of ( (cid:101) ρ, (cid:101) m, (cid:101) v ) obtained above in the KL-minimization, we find that we yield the highly intuitive result:

Original: [Equation]
Revised: [Equation]

Original: A.2 Message parameters following a linear layer
Revised: A.2 Message parameters following a linear layer

Original: Above, we established the values of parameters ( (cid:101) ρ, (cid:101) m, (cid:101) v ) in terms of properties of the distribution weintend to approximate.
Revised: Above, we established the values of parameters ( (cid:101) ρ, (cid:101) m, (cid:101) v ) in terms of properties of the distribution we intend to approximate.

Original: We now shift our focus to deriving the specific values for the transformationsthat occur within spike-and-slab PBP, namely a linear combination and a ReLU.
Revised: We now shift our focus to deriving the specific values for the transformations that occur within spike-and-slab PBP, namely a linear combination and a ReLU.

Original: We begin with thelinear combination z ( ℓ, linear ) j = (cid:80) i
Revised: We begin with the linear combination z ( ℓ, linear ) j = (cid:80) i

Original: z ( ℓ − 1)
Revised: z ( ℓ − 1)

Original: i w ij , seeking the values of E p [ z ( ℓ, linear j ] , V p
Revised: i w ij , seeking the values of E p [ z ( ℓ, linear j ] , V p

Original: [ z ( ℓ, linear j ] , and
Revised: [ z ( ℓ, linear [Equation] ] , and

Original: P p [ z ( ℓ, linear j = 0] under the assumption that
Revised: P p [ z ( ℓ, linear j = 0] under the assumption that

Original: [Equation]
Revised: [Equation]

Original: From this, we can calculate the mean and variance,
Revised: From this [Equation] we can calculate the mean and variance

Original: [Equation]
Revised: [Equation]

Original: Now, incorporating the rescaling transformation  z ( ℓ, linear ) =
Revised: Now, incorporating the rescaling transformation used by Hernández-Lobato and Adams (2015), z ( ℓ, linear ) =

Original: W ( ℓ ) z ( ℓ − 1) / √ n ℓ − 1 + 1 used by{+
 we have+}
Revised: W ( ℓ ) z ( ℓ − 1) / √ n ℓ − 1 + {+
 we have+}

Original: Hernández-Lobato and Adams (2015), we have
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Combining these moments with the general forms derived earlier
Revised: ( ℓ − 1)

Original: [Equation]
Revised: [Equation]

Original:  we have the following equations(replacing Equations 13-14 of the original PBP method) for the distribution of linear layers in SSPBPwith parameters (cid:101) ρ ( ℓ, linear ) , (cid:101) m ( ℓ, linear ) , 
Revised: Combining these moments with the general forms derived earlier, we have the following equations (replacing Equations 13-14 of the original PBP method) for the distribution of linear layers in SSPBP with parameters (cid:101) ρ ( ℓ, linear ) , (cid:101) m ( ℓ, linear ) , (cid:101) v ( ℓ, linear ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: A.3 Message parameters following a linear layer plus ReLU activation
Revised: A.3 Message parameters following a linear layer plus ReLU activation

Original: We now consider passing the messages from Equation 6 through a ReLU, to get the parameters( (cid:101) ρ ( ℓ, ReLU ) , (cid:101) m ( ℓ, ReLU ) , (cid:101) v ( ℓ, ReLU ) ) of the resulting message.
Revised: We now consider passing the messages from Equation 6 through a ReLU, to get the parameters ( (cid:101) ρ ( ℓ, ReLU ) , (cid:101) m ( ℓ, ReLU ) , (cid:101) v ( ℓ, ReLU ) ) of the resulting message.

Original: To compute these parameters
Revised: To compute these parameters

Original: [Equation]
Revised: [Equation]

Original: it’s useful to employ a hierarchical model:
Revised: it’s useful to employ a hierarchical model:

Original: [Equation]
Revised: [Equation]

Original: implying
Revised: implying

Original: [Equation]
Revised: [Equation]

Original: where TN indicates a truncated normal.
Revised: where TN indicates a truncated normal.

Original: Similarly, let’s now introduce a hierarchy for Bi , along witha “dummy” variable
Revised: Similarly [Equation] let’s now introduce a hierarchy for B

Original: X
Revised: i

Revised: [Equation]

Original:  i to make the notation below a little more straightforward.
Revised: along with a “dummy” variable X i to make the notation below a little more straightforward:

Original: [Equation]
Revised: [Equation]

Original: Thus, E [ X i ] , V [ X i ] , etc .
Revised: Thus, E [ X i ] , V [ X i ] , etc .

Original: correspond to the conditional expectation and variance E
Revised: correspond to the conditional expectation and variance E

Original: [ B i | L i =1] , V [ B i | L i = 1] .
Revised: [ B i | L i = 1] , V [ B i | L i = 1] .

Original: We next compute the moments of ( (cid:101) ρ( ℓ, ReLU ) , (cid:101) m ( ℓ, ReLU ) , (cid:101) v ( ℓ, ReLU ) ) by the moment-matching we derived above.
Revised: We next compute the moments of ( (cid:101) ρ

Original: [Equation]
Revised: [Equation]

Original: Using the spike-and-slab moment-matching results above

Original: [Equation]
Revised: [Equation]

Original:  we obtain:
Revised: ) , (cid:101) m ( ℓ, ReLU ) , (cid:101) v ( ℓ, ReLU ) ) by the moment-matching we derived above:

Original: [Equation]
Revised: [Equation]

Original: In other words, we match the probability mass of the spike, and match the mean and variance inour normal approximation to the mean and variance of the truncated normal!

Original: This yields our finalmessage parameters, (cid:101) m ( ℓ, ReLU ) i = (cid:101) m ( ℓ, linear ) i + γ i (cid:113) (cid:101) v ( ℓ, linear )
Revised: Using the spike-and-slab moment-matching results above, we obtain:

Original: i (cid:101) v ( ℓ, ReLU )

Original: i = (cid:101) v ( ℓ, linear )

Original: i   1 − γ i   γ i + (cid:101) m ( ℓ, linear ) i (cid:113) (cid:101) v ( ℓ, linear )

Original: i   

Original:  (cid:101)

Original: [Equation]

Original: ( ℓ

Original: [Equation]

Original: ReLU ) i =

Original: [Equation]

Original: ℓ

Original: [Equation]

Original: linear )

Original: i Φ 

Original: [Equation]
Revised: [Equation]

Original: m ( ℓ, linear ) i (cid:113) [Equation] ( ℓ, linear ) i  
Revised: In other words

Original: [Equation]
Revised: [Equation]

Original: = [Equation] (cid:32) [Equation]
Revised: we match the probability mass of the spike and match the mean and variance in our normal approximation to the mean and variance of the truncated normal!

Original: m ( ℓ, linear ) i (cid:113)(cid:101) [Equation] linear ) i (cid:33) Φ
Revised: This yields our final message parameters,

Original: [Equation]
Revised: [Equation]

Original: linear )

Original: i (cid:113)(cid:101)

Original: [Equation] linear ) i (cid:33) . The intermediate parameter γ i can be replaced with a “robust” version as appropriate, following(Hernández-Lobato and Adams, 2015).
Revised: The intermediate parameter γ i can be replaced with a “robust” version as appropriate, following (Hernández-Lobato and Adams, 2015).

Original: A.4 Normalization Constant
Revised: A.4 Normalization Constant

Original: The normalization constant Z  in Equation 12of (Hernández-Lobato and Adams, 2015) (and itsuses elsewhere in the automatic differentiation for the model parameters) is modified slightly inour approach.
Revised: The normalization constant Z of Hernández-Lobato and Adams (2015) in Equation 12 (and its uses elsewhere in the automatic differentiation for the model parameters) is modified slightly in our approach.

Original: We produce a final output  ( (cid:101) ρ ( L ) , (cid:101) m ( L ) , (cid:101) v ( L ) ) , along with the homoscedastic noiseestimate β γ / ( α γ − 1) (unchanged from PBP).
Revised: As final output parameters, we produce ( (cid:101) ρ ( L ) , (cid:101) m ( L ) , (cid:101) v ( L ) ) , along with the homoscedastic noise estimate β γ / ( α γ − 1) (unchanged from PBP).

Original: As such, the relevant replacement for Equation 12 of(Hernández-Lobato and Adams, 2015) is
Revised: As such, the relevant replacement for Equation 12 of (Hernández-Lobato and Adams, 2015) is

Original: [Equation]
Revised: [Equation]

Original: ( L ) N ( y n | (cid:101) m ( L ) , β γ / ( α γ − 1) + (cid:101) v ( L ) ) .

Original: B In the presence of a bias term, PBP is numerically equivalent to SSPBP
Revised: B In the presence of a bias term, PBP is numerically equivalent to SSPBP

Original: Here, we show that after the linear combination step, the models produce the same resultant distribution.
Revised: Here, we show that after the linear combination step, the models produce the same resultant distribution.

Original: Consider the following model for the output Y of a single node (for simplicity in notation)with random K -dimensional input x , noting that this corresponds exactly to the first hidden layer’sactivation function and the second layer’s linear combination step 5 :
Revised: Consider the following model for the output Y of a single node (for simplicity in notation) with random K -dimensional input x , noting that this corresponds exactly to the first hidden layer’s activation function and the second layer’s linear combination step 5 :

Original: [Equation]
Revised: [Equation]

Original:  the bias term
Revised: ← the bias term

Original: [Equation]
Revised: [Equation]

Original: Following Hernández-Lobato and Adams (2015), we also assume
Revised: Following Hernández-Lobato and Adams (2015), we also assume

Original: [Equation]
Revised: [Equation]

Original: B.1 Parameters of the PBP message
Revised: B.1 Parameters of the PBP message

Original: Under standard PBP, we model the distribution of t i with a Gaussian with mean m ( t ) i and variancev ( t ) i , where
Revised: Under standard PBP, we model the distribution of t i with a Gaussian with mean m ( t ) i and variance v ( t ) i , where

Original: [Equation]
Revised: [Equation]

Original: We then approximate the distribution of y using a Gaussian with mean m ( y ) and variance v ( y ) , where
Revised: We then approximate the distribution of y using a Gaussian with mean m ( y ) and variance v ( y ) [Equation] where

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: B.2 Parameters of the SSPBP message
Revised: B.2 Parameters of the SSPBP message

Original: We repeat our analysis using spike-and-slab approximations, so the distribution of t i is parameterized by ( ρ ( t ) i , m ( t ) i , v ( t ) i ) , and the distribution of y is parameterized by ( ρ ( y ) , m ( y ) , v ( y ) ) , such thatρ ( t ) i =
Revised: We repeat our analysis using spike-and-slab approximations, so the distribution of t i is parameterized by ( ρ ( t ) i , m ( t ) i , v ( t ) i ) [Table] and the distribution of y is parameterized by ( ρ ( y ) , m ( y ) , v ( y ) ) , such that ρ ( t ) i =

Original: Φ
Revised: Φ( α i ) , i = 1 : K, ρ ( t ) K +1 = 1 , m ( t ) i

Original: [Equation]
Revised: = v ′

Original: i) , i = 1 : K, ρ
Revised: i , i = 1 : K, m ( t )

Original: [Equation]
Revised: [Equation]

Original: +{++
+}
Revised: {++
+}

Original: [Equation] = [Equation] t ) 
Revised: = 1 , v ( t ) i

Original: [Equation] = [Equation]
Revised: = (cid:101)

Original: [Equation]
Revised: [Equation]

Original: m
Revised: i

Original: [Equation]
Revised: [Equation]

Original: v ( t ) i = (cid:101)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: = 1 : K v ( t ) K +1 = 0 , and ρ (

Original: [Equation]
Revised: [Equation]

Original: γ i ( γ i + α i )) , i = 1 : K v ( t ) K +1 =andρ ( y ) =1 − K +
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: i
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Comparing Equations 7 and 8, we see the two methods are exactly equivalent after a linear combination step, with slab probability equal to 1 in the spike-and-slab variant.
Revised: Comparing Equations 7 and 8, we see the two methods are exactly equivalent after a linear combination step, with slab probability equal to 1 in the spike-and-slab variant.

Original: As such, any future hiddenlayers will behave identically.
Revised: As such, any future hidden layers will behave identically.

Original: In the regression setting of PBP, the final transformation is solely alinear combination, so the final output is the same.[Table] [Table] -2.824 ± 0.
Revised: In the regression setting of PBP, the final transformation is solely a linear combination, so the final output is the same.

Original: -2.844 ± 0.

Original: -2.845 ± 0.

Original: Concrete Compression Strength -3.253 ± 0.

Original: -3.284 ± 0. -3.136 ± 0. -3.138 ± 0. Energy Efficiency -2.179 ± 0.
Revised: C Additional Empirical Results

Original: -2.169 ± 0. -2.056 ± 0. -2.066 ± 0. Kin8nm 0.549 ± 0.
Revised: We include here additional results for RMSE and log-likelihood of different model configurations.

Original: 0.567 ± 0. 0.968 ± 0. 0.975 ± 0. Naval Propulsion 3.275 ± 0.
Revised: Table 5 and Table 6 report the RMSE and log-likelihood for the standard version of PBP and SSPBP, respectively.

Original: 3.276 ± 0. 3.949 ± 0. 3.945 ± 0. Wine Quality Red -0.941 ± 0.
Revised: Similarly, Table 7 and Table 8 report the RMSE and log-likelihood of the “bias-free” versions of PBP and SSPBP, respectively.

Original: -0.937 ± 0.

Original: -0.959 ± 0.

Original: -0.955 ± 0.

Original: Yacht Hydrodynamics -2.022 ± 0.

Original: -2.012 ± 0.

Original: -1.751 ± 0.

Original: -1.739 ± 0.

Original: [Table]
Revised: [Table]

Original: Layer

Original: [Table]Dataset PBP SSPBP
Revised: [Table]

Original: PBP

Original: SSPBP

Original: Boston Housing -2.771 ± 0.

Original: -2.787 ± 0.

Original: -2.576 ± 0.

Original: -2.535 ± 0.

Original: Combined Cycle Power Plant -2.834 ± 0.

Original: -2.834 ± 0.

Original: -2.828 ± 0.

Original: -2.830 ± 0.

Original: Concrete Compression Strength -3.149 ± 0.

Original: -3.157 ± 0.

Original: -3.218 ± 0.

Original: -3.201 ± 0.

Original: Energy Efficiency -2.049 ± 0.

Original: -2.047 ± 0. [Table]C Additional Empirical Results We include here additional results for RMSE and log-likelihood of different model configurations.
Revised: [Table]

Original: Table 4 and Table 5 report the RMSE and log-likelihood for the standard version of PBP and SSPBP,respectively. Similarly, Table 6 and Table 7 report the RMSE and log-likelihood of the “bias-free”versions of PBP and SSPBP, respectively.
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]