Original: B LURS M AKE R ESULTS C LEARER : S PATIAL S MOOTHINGS TO I MPROVE A CCURACY , U NCERTAINTY , AND R OBUSTNESS
Revised: B LUR I S AN E NSEMBLE : S PATIAL S MOOTHINGS TO I MPROVE A CCURACY , U NCERTAINTY , AND R OBUSTNESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Bayesian neural networks (BNNs) have shown success in the areas of uncertaintyestimation and robustness.
Revised: Bayesian neural networks (BNNs) have shown success in the areas of uncertaintyestimation and robustness.

Original: However, a crucial challenge prohibits their use inpractice.
Revised: However, a crucial challenge prohibits their use inpractice.

Original: Bayesian NNs require a large number of predictions to produce reliableresults, leading to a signiﬁcant increase in computational cost.
Revised: Bayesian NNs require a large number of predictions to produce reliableresults, leading to a signiﬁcant increase in computational cost.

Original: To alleviate thisissue, we propose spatial smoothing , a method that ensembles neighboring featuremap points of CNNs.
Revised: To alleviate thisissue, we propose spatial smoothing , a method that ensembles neighboring featuremap points of CNNs.

Original: By simply adding a few blur layers to the models, weempirically show that spatial smoothing improves accuracy, uncertainty estimation,and robustness of BNNs across a whole range of ensemble sizes.
Revised: By simply adding a few blur layers to the models, weempirically show that spatial smoothing improves accuracy, uncertainty estimation,and robustness of BNNs across a whole range of ensemble sizes.

Original: In particular,
Revised: In particular,

Original: BNNs incorporating spatial smoothing achieve high predictive performance merelywith a handful of ensembles.
Revised: BNNs incorporating spatial smoothing achieve high predictive performance merelywith a handful of ensembles.

Original: Moreover, this method also can be applied to canonicaldeterministic neural networks to improve the performances.
Revised: Moreover, this method also can be applied to canonicaldeterministic neural networks to improve the performances.

Original: A number of evidencessuggest that the improvements can be attributed to the stabilized feature mapsand the ﬂattening of the loss landscape.
Revised: A number of evidencessuggest that the improvements can be attributed to the stabilized feature mapsand the ﬂattening of the loss landscape.

Original: In addition, we provide a fundamentalexplanation for prior works—namely, global average pooling, pre-activation, and
Revised: In addition, we provide a fundamentalexplanation for prior works—namely, global average pooling, pre-activation, and

Original: ReLU6 —by addressing them as special cases of spatial smoothing.
Revised: ReLU6 —by addressing them as special cases of spatial smoothing.

Original: These notonly enhance accuracy, but also improve uncertainty estimation and robustness bymaking the loss landscape smoother in the same manner as spatial smoothing.
Revised: These notonly enhance accuracy, but also improve uncertainty estimation and robustness bymaking the loss landscape smoother in the same manner as spatial smoothing.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: both accuracy and uncertainty (NLL) .
Revised: both accuracy and uncertainty (NLL) .

Original: Smooth means spatial smoothing.
Revised: Smooth means spatial smoothing.

Original: Downward from left to the right ( (cid:38) ) means better accuracy and uncertainty.
Revised: Downward from left to the right ( (cid:38) ) means better accuracy and uncertainty.

Original: In a real-world environment where many unexpectedevents occur, machine learning systems cannot be guaranteed to always produce accurate predictions.
Revised: In a real-world environment where many unexpectedevents occur, machine learning systems cannot be guaranteed to always produce accurate predictions.

Original: In order to handle this issue, we make system decisionsmore reliable by considering estimated uncertainties,in addition to predictions.
Revised: In order to handle this issue, we make system decisionsmore reliable by considering estimated uncertainties,in addition to predictions.

Original: Uncertainty quantiﬁcation isparticularly crucial in building a trustworthy system inthe ﬁeld of safety-critical applications, including medical analysis and autonomous vehicle control.
Revised: Uncertainty quantiﬁcation isparticularly crucial in building a trustworthy system inthe ﬁeld of safety-critical applications, including medical analysis and autonomous vehicle control.

Original: However,canonical deep neural networks (NNs)—or deterministic NNs—cannot produce reliable estimations of uncertainties (Guo et al., 2017), and their accuracy is oftenseverely compromised by natural data corruptions fromnoise, blur, and weather changes (Engstrom et al., 2019;
Revised: However,canonical deep neural networks (NNs)—or deterministic NNs—cannot produce reliable estimations of uncertainties (Guo et al., 2017), and their accuracy is oftenseverely compromised by natural data corruptions fromnoise, blur, and weather changes (Engstrom et al., 2019;

Original: Azulay & Weiss, 2019).
Revised: Azulay & Weiss, 2019).

Original: Bayesian neural networks (BNNs), such as MC dropout(Gal & Ghahramani, 2016), provide a probabilistic representation of NN weights.
Revised: Bayesian neural networks (BNNs), such as Monte Carlo(MC) dropout (Gal & Ghahramani, 2016), provide aprobabilistic representation of NN weights.

Original: They combine a number ofmodels selected based on weight probability to makepredictions of desired results.
Revised: They combine a number of models selected based on weight probability to make predictions of desired results.

Original: Thanks to this feature, BNNs have been widely used in the areas of uncertainty estimation (Kendall & Gal, 2017) androbustness (Ovadia et al., 2019).
Revised: Thanks tothis feature, BNNs have been widely used in the areas of uncertainty estimation (Kendall & Gal, 2017)and robustness (Ovadia et al., 2019).

Original: They are also promising in other ﬁelds like out-of-distributiondetection (Malinin & Gales, 2018) and meta-learning (Yoon et al., 2018). 
Revised: They are also promising in other ﬁelds like out-of-distributiondetection (Malinin & Gales, 2018) and meta-learning (Yoon et al., 2018). [Figure] Importance

Original: [Figure]
Revised: [Figure]

Original: t= t=-1 t=-
Revised: t= t=-1 t=-

Original: Temporal smoothing ✕ ✕ ✕ ✕ πxxx - ππ - ppp -
Revised: Temporal smoothing ✕ ✕ ✕ ✕ πxxx - ππ - ppp -

Original: Temporally proximate data stream
Revised: Temporally proximate data stream

Original: Spatial smoothing ✕ ✕ ppppxxxxππππ
Revised: Spatial smoothing ✕ ✕ ppppxxxxππππ

Original: Spatially proximate data points
Revised: Spatially proximate data points

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: xpp
Revised: xppp

Original: [Figure]
Revised: [Figure]

Original: Observed 
Revised: Observed data

Original: [Figure]

Original: Nevertheless, there remains a signiﬁcant challenge that prohibits their use in practice.
Revised: Nevertheless, there remains a signiﬁcant challenge that prohibits their use in practice.

Original: BNNs requirean ensemble size of up to ﬁfty to achieve high predictive performance, which results in a ﬁftyfoldincrease in computational cost (Kendall & Gal, 2017; Loquercio et al., 2020).
Revised: BNNs requirean ensemble size of up to ﬁfty to achieve high predictive performance, which results in a ﬁftyfoldincrease in computational cost (Kendall & Gal, 2017; Loquercio et al., 2020).

Original: Therefore, if BNNscan achieve high predictive performance merely with a handful of ensembles, they could be appliedto a much wider range of areas.
Revised: Therefore, if BNNscan achieve high predictive performance merely with a handful of ensembles, they could be appliedto a much wider range of areas.

Original: 1. P RELIMINARY
Revised: 1. P RELIMINARY

Original: We would ﬁrst like to discuss BNN inference in detail, then move on to VQ-BNN inference (Parket al., 2021), an efﬁcient approximated BNN inference.
Revised: We would ﬁrst like to discuss BNN inference in detail, then move on to Vector-Quantized BNN(VQ-BNN) inference (Park et al., 2021), an efﬁcient approximated BNN inference.

Original: BNN inference.
Revised: BNN inference.

Original: Suppose we have access to posterior probability of NN weight p ( w |D ) for training dataset D .
Revised: Suppose we have access to posterior probability of NN weight p ( w |D ) for training dataset D .

Original: The predictive result of BNN is given by the following predictive distribution:
Revised: The predictive result of BNN is given by the following predictive distribution:

Original: [Equation]
Revised: [Equation]

Original: where x 0 is observed input data vector, y is output vector, and p ( y | x , w ) is the probabilistic prediction parameterized by the result of NN for an input x and weight w .
Revised: where x 0 is observed input data vector, y is output vector, and p ( y | x , w ) is the probabilistic prediction parameterized by the result of NN for an input x and weight w .

Original: In most cases, the integral cannot be solved analytically.
Revised: In most cases, the integral cannot be solved analytically.

Original: Thus, we use the MC estimator to approximate it as follows:
Revised: Thus, we use the MC estimator to approximate it as follows:

Original: [Equation]
Revised: [Equation]

Original: where w i ∼ p ( w |D ) and N is the number of the samples.
Revised: where w i ∼ p ( w |D ) and N is the number of the samples.

Original: The equation indicates that BNN inferenceis ensemble average of NN predictions for one observed data point as shown on the left of Fig. 2.
Revised: The equation indicates that BNN inferenceis ensemble average of NN predictions for one observed data point as shown on the left of Fig. 2.

Original: Using N neural networks in the ensemble would requires N times more computational complexitythan one NN execution.
Revised: Using N neural networks in the ensemble would requires N times more computational complexitythan one NN execution.

Original: [Figure]
Revised: [Figure]

Original: CIFAR-

Original: Data-complemented BNN inference.
Revised: Data-complemented BNN inference.

Original: To reduce the computational cost of BNN inference, VQBNN (Park et al., 2021) executes NN for an observed data only once and complements the result with previously calculated predictions for other data .
Revised: To reduce the computational cost of BNN inference, VQBNN (Park et al., 2021) executes NN for an observed data only once and complements the result with previously calculated predictions for other data .

Original: If we have access to previous predictions, the computational performance of VQ-BNN becomes comparable to that of one NN execution.
Revised: If we have access to previous predictions, the computational performance of VQ-BNN becomes comparable to that of one NN execution.

Original: To be speciﬁc, VQ-BNN inference is:
Revised: To be speciﬁc, VQ-BNN inference is:

Original: [Equation]
Revised: [Equation]

Original: where π ( x i | x 0 ) is the importance of data x i with respect to the observed data x 0 , and it is deﬁned as asimilarity between x i and x 0 .
Revised: where π ( x i | x 0 ) is the importance of data x i with respect to the observed data x 0 , and it is deﬁned as asimilarity between x i and x 0 .

Original: p ( y | x 0 , w 0 ) is the newly calculated prediction, and { p ( y | x 1 , w 1 ) , · · · }are previously calculated predictions.
Revised: p ( y | x 0 , w 0 ) is the newly calculated prediction, and { p ( y | x 1 , w 1 ) , · · · }are previously calculated predictions.

Original: To accurately infer the results, the previous predictions shouldconsist of predictions for “data similar to the observed data” .
Revised: To accurately infer the results, the previous predictions shouldconsist of predictions for “data similar to the observed data” .

Original: Thanks to the temporal consistency of real-world data streams, aggregating predictions for similardata in data streams is straightforward.
Revised: Thanks to the temporal consistency of real-world data streams, aggregating predictions for similardata in data streams is straightforward.

Original: Since temporally proximate data sequences tend to be similar,we can memorize recent predictions and calculates their average using exponentially decreasingimportance.
Revised: Since temporally proximate data sequences tend to be similar,we can memorize recent predictions and calculates their average using exponentially decreasingimportance.

Original: In other words, VQ-BNN inference for data streams is simply temporal smoothing ofrecent predictions as shown in the middle of Fig. 2.
Revised: In other words, VQ-BNN inference for data streams is simply temporal smoothing ofrecent predictions as shown in the middle of Fig. 2.

Original: VQ-BNN has two limitations, although it may be a promising approach to obtain reliable resultsin an efﬁcient way.
Revised: VQ-BNN has two limitations, although it may be a promising approach to obtain reliable resultsin an efﬁcient way.

Original: First, it was only applicable to data streams such as video sequences.
Revised: First, it was only applicable to data streams such as video sequences.

Original: Applying
Revised: Applying

Original: VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.
Revised: VQ-BNN to images is challenging because it is impossible to memorize all similar images in advance.

Original: Second, Park et al.
Revised: Second, Park et al.

Original: (2021) used VQ-BNN only in the testing phase, not in the training phase.
Revised: (2021) used VQ-BNN only in the testing phase, not in the training phase.

Original: We ﬁndthat ensembling predictions for similar data helps in NN training by smoothing the loss landscape.
Revised: We ﬁndthat ensembling predictions for similar data helps in NN training by smoothing the loss landscape.

Original: 1. M AIN C ONTRIBUTION
Revised: 1. M AIN C ONTRIBUTION

Original: 1 Spatially neighboring points in visual imagery tend to be similar, as do feature maps of convolutional neural networks (CNNs).
Revised: 1 Spatially neighboring points in visual imagery tend to be similar, as do feature maps of convolutional neural networks (CNNs).

Original: By exploiting this spatial consistency, we propose spatial smoothingas a method of ensembling nearby feature map points to improve the efﬁciency of ensemble size in BNN inference.
Revised: By exploiting this spatial consistency, we propose spatial smoothingas a method of ensembling nearby feature maps to improve the efﬁciency of ensemble size in BNNinference.

Original: The right side of Fig.
Revised: The right side of Fig.

Original: 2 visualizes spatial smoothinglayer aggregating neighboringfeature
Revised: 2 visualizes spatial smoothing aggregating neighboring feature

Original: maps.2 We empirically demonstrate that spatial smoothing improves the efﬁciency in vision tasks, suchas image classiﬁcation on CIFAR  and ImageNet, without any additional training parameters.
Revised: maps.2 We empirically demonstrate that spatial smoothing improves the efﬁciency in vision tasks, suchas image classiﬁcation on CIFAR (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015),without any additional training parameters.

Original: Figure 3shows that negative log-likelihood (NLL) of “MC dropout + spatial smoothing” with an ensemblesize of two is comparable to that of vanilla MC dropout with an ensemble size of ﬁfty.
Revised: Figure 3 shows that negative log-likelihood (NLL) of“MC dropout + spatial smoothing” with an ensemble size of two is comparable to that of vanilla MCdropout with an ensemble size of ﬁfty.

Original: We alsodemonstrate that spatial smoothing improves accuracy, uncertainty, and robustness all at the sametime.
Revised: We also demonstrate that spatial smoothing improves accuracy,uncertainty, and robustness all at the same time.

Original: Figure 1 shows that spatial smoothing improves both the accuracy and uncertainty of variousdeterministic and Bayesian NNs with an ensemble size of ﬁfty on CIFAR-100.3 Global average pooling (GAP) (Lin et al., 2013;
Revised: Figure 1 shows that spatial smoothing improves boththe accuracy and uncertainty of various deterministic and Bayesian NNs with an ensemble size ofﬁfty on CIFAR-100.3 Global average pooling (GAP) (Lin et al., 2014;

Original: Zhou et al., 2016), pre-activation (He et al.
Revised: Zhou et al., 2016), pre-activation (He et al.

Original: ,2016b), and ReLU6 (Krizhevsky & Hinton, 

Revised: ,2016b), and ReLU6 (Krizhevsky & Hinton,

Original: ; Sandler et al., 2018) have been widely used in visiontasks.
Revised: ; Sandler et al., 2018) have been vision

Original: However, their motives are largely justiﬁed by experiments an explanation smoothing

Original: P

Original: [Figure]
Revised: [Figure]

Original: Stage i
Revised: Stage i

Original: Stage i+
Revised: Stage i+

Original: H ✕ W ✕ C
Revised: H ✕ W ✕ C

Original: H/2 ✕ W/2 ✕ 2 C
Revised: H/2 ✕ W/2 ✕ 2 C

Original: H ✕ W ✕ C
Revised: H ✕ W ✕ C

Original: Stage i
Revised: Stage i

Original: Stage i+
Revised: Stage i+

Original: Smooth
Revised: Smooth

Original: Prob (tanh-ReLU)
Revised: Prob (tanh-ReLU)

Original: Blur (AvgPool)
Revised: Blur (AvgPool)

Original: H/2 ✕ W/2 ✕ 2 C
Revised: H/2 ✕ W/2 ✕ 2 C

Original: H ✕ W ✕ C
Revised: H ✕ W ✕ C

Original: To improve the computational performance of BNN inference, VQ-BNN(Park et al., 2021) executes NN prediction only once and complementsthe result with previously calculatedpredictions.
Revised: To improve the computational performance of BNN inference, VQ-BNN(Park et al., 2021) executes NN prediction only once and complementsthe result with previously calculatedpredictions.

Original: The key to the successof this approach largely depends onthe collection of previous predictionsfor proximate data.
Revised: The key to the successof this approach largely depends onthe collection of previous predictionsfor proximate data.

Original: Gathering temporally proximate data and their predictions from data streams is easy because recent data and predictions canbe aggregated using temporal consistency.
Revised: Gathering temporally proximate data and their predictions from data streams is easy because recent data and predictions canbe aggregated using temporal consistency.

Original: On the other hand, gatheringtime-independent proximate data, e.g. images, is more difﬁcult because they lack such consistency.
Revised: On the other hand, gatheringtime-independent proximate data, e.g. images, is more difﬁcult because they lack such consistency.

Original: 2. M ODULE A RCHITECTURE FOR E NSEMBLING N EIGHBORING F EATURE M AP P OINTS
Revised: 2. M ODULE A RCHITECTURE FOR E NSEMBLING N EIGHBORING F EATURE M AP P OINTS

Original: So instead of temporal consistency, we use spatial consistency—where neighboring pixels of imagesare similar—for real-world images.
Revised: So instead of temporal consistency, we use spatial consistency—where neighboring pixels of imagesare similar—for real-world images.

Original: Under this hypothesis, we take the feature maps as predictionsand aggregate neighboring feature maps.
Revised: Under this hypothesis, we take the feature maps as predictionsand aggregate neighboring feature maps.

Original: Most CNN architectures, including ResNet, consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume.
Revised: Most CNN architectures, including ResNet, consist of multiple stages that begin with increasing the number of channels while reducing the spatial dimension of the input volume.

Original: We decompose an entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows:
Revised: We decompose an entire BNN inference into several steps by rewriting each stage in a recurrence relation as follows:

Original: [Equation]
Revised: [Equation]

Original: where z i is input volume of the i -th stage, and the ﬁrst and the last volume are input data andoutput.
Revised: where z i is input volume of the i -th stage, and the ﬁrst and the last volume are input data andoutput.

Original: w i and p ( w i |D ) are NN weight in the i -th stage and its probability.
Revised: w i and p ( w i |D ) are NN weight in the i -th stage and its probability.

Original: p ( z i +1 |
Revised: p ( z i +1 |

Original: z i , w i ) is outputprobability of z i +1 with respect to the input volume z i .
Revised: z i , w i ) is outputprobability of z i +1 with respect to the input volume z i .

Original: To derive the probability from the outputfeature map, we transform each point of the feature map into a Bernoulli distribution.
Revised: To derive the probability from the outputfeature map, we transform each point of the feature map into a Bernoulli distribution.

Original: To do so, acomposition of tanh and ReLU , a function from value of range [ −∞ , ∞ ] into probability, is addedafter each stage.
Revised: To do so, acomposition of tanh and ReLU , a function from value of range [ −∞ , ∞ ] into probability, is addedafter each stage.

Original: Put shortly, we use neural networks for point-wise binary feature classiﬁcation .
Revised: Put shortly, we use neural networks for point-wise binary feature classiﬁcation .

Original: Since Eq.
Revised: Since Eq.

Original: (4) is a kind of BNN inference, it can be approximated using Eq.
Revised: (4) is a kind of BNN inference, it can be approximated using Eq.

Original: In other words, eachstage predicts feature map points only once and complements predictions with similar feature maps.
Revised: In other words, eachstage predicts feature map points only once and complements predictions with similar feature maps.

Original: Under spatial consistency, it averages probabilities of spatially neighboring feature map points, whichis well known as blur operation in image processing.
Revised: Under spatial consistency, it averages probabilities of spatially neighboring feature map points, whichis well known as blur operation in image processing.

Original: For the sake of implementation simplicity,average pooling with a kernel size of 2 and a stride of 1 is used as a box blur.
Revised: For the sake of implementation simplicity,average pooling with a kernel size of 2 and a stride of 1 is used as a box blur.

Original: This operation ensemblesfour neighboring probabilities with the same importances.
Revised: This operation ensemblesfour neighboring probabilities with the same importances.

Original: In summary, as shown in Fig. 4, we propose the following probabilistic spatial smoothing layer:
Revised: In summary, as shown in Fig. 4, we propose the following probabilistic spatial smoothing layer:

Original: [Equation]
Revised: [Equation]

Original: where Prob ( · ) is a point-wise function from a feature map to probability, and Blur ( · ) is importanceweighted average for ensembling spatially neighboring probabilities from feature maps.
Revised: where Prob ( · ) is a point-wise function from a feature map to probability, and Blur ( · ) is importanceweighted average for ensembling spatially neighboring probabilities from feature maps.

Original: Smooth layeris added after each stage.
Revised: Smooth layeris added after each stage.

Original: Prob and Blur are further elaborated below.
Revised: Prob and Blur are further elaborated below.

Original: Prob : Feature map to probability.
Revised: Prob : Feature map to probability.

Original: Prob is a function that transforms a real-valued feature map into probability.
Revised: Prob is a function that transforms a real-valued feature map into probability.

Original: We use tanh – ReLU composition for this purpose.
Revised: We use tanh – ReLU composition for this purpose.

Original: However, tanh is commonly known to suffer from the vanishing gradient problem.
Revised: However, tanh is commonly known to suffer from the vanishing gradient problem.

Original: To alleviate this issue, we propose the following temperature-scaled tanh :
Revised: To alleviate this issue, we propose the following temperature-scaled tanh :

Original: [Equation]
Revised: [Equation]

Original: where τ is a hyperparameter called temperature.
Revised: where τ is a hyperparameter called temperature.

Original: τ is 1 in conventional tanh and ∞ in identityfunction as shown in Fig. B.2.
Revised: τ is 1 in conventional tanh and ∞ in identityfunction. tanh τ imposes an upper bound

Original: [Figure]
Revised: [Figure]

Original: An unnormalized probability, ranging from 0 to τ , is allowed as the output of Prob .
Revised: An unnormalized probability, ranging from 0 to τ , is allowed as the output of Prob .

Original: Then, thanks to the linearity of integration, we obtain an unnormalized predictive distribution accordingly.
Revised: Then, thanks to the linearity of integration, we obtain an unnormalized predictive distribution accordingly.

Original: Taking this into account, we propose the following Prob :
Revised: Taking this into account, we propose the following Prob :

Original: [Equation]
Revised: [Equation]

Original: ) where τ > 1 . CNN stages often end with ReLU .
Revised: where τ > 1 . We expect upper-bounded functions, e.g., ReLU6 ( z ) =

Original: Therefore, ReLU can be omitted from Prob in thesecases to improve computational performance because tanh τ  and  ReLU are commutative.
Revised: ReLU ◦ min ( z , 6) and featuremap scaling z /τ with τ > 1 which is BatchNorm , to be able to replace tanh τ in Prob ; and asexpected, these alternatives improve uncertainty estimation in addition to accuracy. See Appendix C.2and Appendix C.3 for detailed discussions on activation ( ReLU ◦ BatchNorm ) and ReLU6 as Prob .

Original: We empirically determine τ to minimize NLL, a metric that measures both accuracy and uncertainty.
Revised: We empirically determine τ to minimize NLL, a metric that measures both accuracy and uncertainty.

Original: When τ is small, the regularization of Prob dominates and better uncertainty estimation can beobtained; however, accuracy is degraded.
Revised: When τ is small, the regularization of Prob dominates and better uncertainty estimation can beobtained; however, accuracy is degraded.

Original: NLL is optimized when accuracy and uncertainty estimationare balanced.
Revised: NLL is optimized when accuracy and uncertainty estimationare balanced.

Original: SeeAppendix B.1 and Fig.
Revised: See Fig.

Original: B.3 for more details.
Revised: B.3 for more details.

Original: We expect upper-bounded functions, e.g., ReLU6 ( z )

Original: = min ( max ( z , 0) , 6) and constant scaling z /τwhich is non-trainable BatchNorm as represented in Fig.

Original: B.1, to be able to replace tanh τ in Prob .

Original: Appendix B.1 and Table B.1 provide experimental results indicating that these alternatives alsoimprove uncertainty estimation in addition to accuracy.

Original: The results suggest that the bounded functionsin front of Blur help ensemble spatial information.

Original: Moreover, similarities between Prob representedby Eq.

Original: (7) and activation ReLU ◦ BatchNorm imply that placing activations before convolutionallayers, an alternative to Blur , is effective.

Original: Experiments show that this pre-activation arrangementnot only improves accuracy but also uncertainty.

Original: See Appendix C.2 and Appendix C.3 for detaileddiscussions on pre-activation and ReLU6 .

Original: Blur : Averaging neighboring probabilities.
Revised: Blur : Averaging neighboring probabilities.

Original: Blur averages the probabilities from feature maps.
Revised: Blur averages the probabilities from feature maps.

Original: In this work, we primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation of Blur for the sake of simplicity.
Revised: We primarily use the average pool with a kernel size of 2 and a stride of 1 as the implementation of Blur for the sake of simplicity.

Original: Nevertheless, we could generalize Blur by using the following depth-wise convolution, which acts on each input channel separately, with non-trainable kernel
Revised: Nevertheless, we could generalize Blur by using the following depth-wise convolution, which acts on each input channel separately, with non-trainable kernel

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where k is a one-dimensional matrix, e.g. k ∈ { (1) , (1 , 1) , (1 , 2 , 1) , (1 , [Equation]
Revised: where k is a 1D matrix, e.g., k ∈ { (1) , (1 , 1) , (1 , 2 , 1) , (1 , 4 , 6 , 4 , 1) } .

Original: Different k sderive different importances for neighboring feature map points as shown in Fig.
Revised: Different k s derive differentimportances for neighboring feature maps.

Original: B.4. Appendix B.2 and Table B.2 showthe predictive performance of Blur s for different K s. Theexperiment shows that most Blur s improve the predictive performance and that optimal K varies bymodel.
Revised: We empirically show that most Blur s improve thepredictive performance and that optimal K varies by model. For more details, see Table B.2. 2. H OW D OES S PATIAL S MOOTHING H ELP O PTIMIZATION ?

Revised: We present theoretical and empirical aspects to show that spatial smoothing ensembles feature maps.

Revised: Feature map variance.

Revised: BNNs have two types of uncertainties: One is model uncertainty and theother is data uncertainty (Park et al., 2021).

Original: 1.0 π 0.0 π w (a) Frequency mask 0.
Revised: These randomness increase the variance of the featuremaps. To demonstrate that spatial smoothing is an ensemble, we use the following proposition: Proposition 1. Ensembles reduce variance of predictions.

Revised: We omit the proof since it is straightforward.

Revised: In our context, predictions are output feature maps of astage.

Revised: We investigate model and data uncertainties of the predictions along NN layers to show that

Revised: 1.0 π 0.0 π w (a) Frequency mask 0.

Original: Frequency 2.
Revised: Frequency 2.

Original: L o g a m p l i t u d e (b) Log amplitude at c 0.
Revised: L o g a m p l i t u d e (b) Log amplitude at c 0.

Original: Noise frequency
Revised: Noise frequency

Original: A cc u r a c y ( % ) (c) Robustness for noise frequency 0.
Revised: A cc u r a c y ( % ) (c) Robustness for noise frequency 0.

Original: Frequency 0.
Revised: Frequency 0.

Original: L o g a m p l i t u d e
Revised: L o g a m p l i t u d e

Original: Deterministic

Original: [Figure]
Revised: [Figure]

Original: 2. H OW D OES SPATIAL SMOOTHING H ELP O PTIMIZATION ?

Original: To demonstrate how spatial smoothing works, this section presents theoretical and empirical aspects.

Original: Theoretical analysis.

Original: BNNs have two types of uncertainties: One is model uncertainty and theother is data uncertainty (Park et al., 2021).

Original: Appendix D.1 proves that these randomnesses sharpenthe loss landscape, and a sharp loss function disturbs NN optimization (Keskar et al., 2016; Santurkaret al., 2018; Foret et al., 2020).

Original: Appendix D.2 shows that training phase NN ensemble averagesout the randomnesses, and it ﬂattens the loss function; the sharpness is inversely proportional tothe ensemble size as in Eq.

Original: In consequence, an ensemble of BNN outputs in training phasesigniﬁcantly improves the predictive performance as demonstrated in Appendix D.2.

Original: However, we donot use training phase ensemble because it signiﬁcantly increases the training time.

Original: Spatial smoothing is a method that ensembles feature maps without sacriﬁcing training time. Weempirically demonstrate below that spatial smoothingalso helps NNs learn strong representations bystabilizing feature maps and ﬂattening the loss landscape. Feature map behavior. We investigate model and data uncertainties along NN layers to show thatspatial smoothing reduces the randomnesses and ensembles feature maps.
Revised: spatial smoothing reduces the randomnesses and ensembles feature maps.

Original: Figure 5 shows the modeluncertainty and data uncertainty of Bayesian ResNet including MC dropout layers.
Revised: Figure 5 shows the modeluncertainty and data uncertainty of Bayesian ResNet including MC dropout layers.

Original: In this ﬁgure, theuncertainty of MC dropout’s feature map only accumulates, and almost monotonically increases inevery NN layer.
Revised: In this ﬁgure, theuncertainty of MC dropout’s feature map only accumulates, and almost monotonically increases inevery NN layer.

Original: In contrast, the uncertainty of “MC dropout + spatial smoothing”’s feature map issigniﬁcantly decreases at the end of stages, suggesting that the smoothing layers ensemble the featuremap.
Revised: In contrast, the uncertainty of “MC dropout + spatial smoothing”’s feature map issigniﬁcantly decreases at the end of stages, suggesting that the smoothing layers ensemble the featuremap.

Original: In other words, they make the feature map more accurate and stabilized input volumes for thenext stages.
Revised: In other words, they make the feature map more accurate and stabilized input volumes for thenext stages.

Original: Figure B.5 also shows consistent results, i.e., the spatial smoothing layer close to the lastlayer signiﬁcantly improves performance because it reduces the uncertainty of predictions largely. 
Revised: In addition, consistently, the spatial smoothing layer close to the last layer signiﬁcantlyimproves performance because it reduces the uncertainty of predictions largely. See Fig. B.5 for moredetails.

Original: Deterministic NNs do not have model uncertainty but data uncertainty.
Revised: Deterministic NNs do not have model uncertainty but data uncertainty.

Original: Therefore, spatial smoothingimproves the performance of deterministic NNs as well as Bayesian NNs.
Revised: Therefore, spatialsmoothing improves the performance of deterministic NNs as well as Bayesian NNs.

Original:  We also analyze spatial smoothing through the lens of Fourier transform.
Revised: Fourier analysis. We also analyze spatial smoothing through the lens of Fourier transform:

Original: Figure 6b shows the 2D
Revised: Proposition 2. Ensembles ﬁlter high-frequency signals. The proof is provided in Eqs. (16) to (17).

Original:  Fourier transformed output feature map at the end of the stage 1.
Revised: Figure 6b shows the 2D Fourier transformed outputfeature map at the end of the stage 1.

Original: This ﬁgure reveals that MC dropoutalmost does not affect low-frequency ( < 0 .
Revised: This ﬁgure reveals that MC dropout almost does not affectlow-frequency ( < 0 .

Original: 3 π ) ranges, and it adds high-frequency ( ≥ 0 .
Revised: 3 π ) ranges, and it adds high-frequency ( ≥ 0 .

Original: 3 π ) noises.
Revised: 3 π ) noises.

Original: Since spatial smoothing is a low-pass ﬁlter, it effectively ﬁlters high-frequency signals, including thenoises caused by MC dropout.
Revised: Since spatial smoothingis a low-pass ﬁlter, it effectively ﬁlters high-frequency signals, including the noises caused by MCdropout.

Original: See Fig.
Revised: See Fig.

Original: D.3 for more results.
Revised: D.2 for more results.

Original: We  ﬁnd that CNNs are particularly vulnerable to high-frequency noises.
Revised: We also ﬁnd that CNNs are particularly vulnerable to high-frequency noises.

Original: To demonstrate this claim,following Shao et al.
Revised: To demonstrate thisclaim, following Shao et al.

Original: (2021), we measure accuracy with respect to data with frequency-based randomnoise x noise = x 0 + F − 1 ( F ( δ ) (cid:12) M f ) , where x 0 is clean data, F ( · ) and F − 1 ( · ) are Fouriertransform and inverse Fourier transform, δ is random noise, and M f is frequency mask as shownin Fig.
Revised: (2021), we measure accuracy with respect to data with frequency-basedrandom noise x noise = x 0 + F − 1 ( F ( δ ) (cid:12) M f ) , where x 0 is clean data, F ( · ) and F − 1 ( · ) are Fouriertransform and inverse Fourier transform, δ is random noise, and M f is frequency mask as shownin Fig.

Original: 6a.
Revised: 6a.

Original: Figure 6c exhibits the results.
Revised: Figure 6c exhibits the results.

Original: In sum, high-frequency noises, including those caused by
Revised: In sum, high-frequency noises, including those caused by

Original: [Figure]

Original: MC dropout, signiﬁcantly impair accuracy.
Revised: MC dropout, signiﬁcantly impair accuracy.

Original: Spatial smoothing improves the robustness by effectivelyremoving high-frequency noises. 
Revised: Spatial smoothing improves the robustness by effectivelyremoving high-frequency noises. Loss landscape.

Original: For more details, see Appendix D.3.
Revised: Lastly, we show that the randomness hinders NN training as follows: Proposition 3. Randomness of predictions sharpens loss landscape, and ensembles ﬂatten it. The proof is provided in Eqs.

Original: [Figure]
Revised: (18) to (25).

Original: i e., they smoothen the loss function. 
Revised: Since a sharp loss function disturbs NN optimization(Keskar et al., 2017; Santurkar et al., 2018; Foret et al., 2020), reducing the uncertainty helps NNlearn strong representations. For example, training phase NN ensemble averages out the randomness,and it ﬂattens the loss function. In consequence, an ensemble of BNN outputs in training phasesigniﬁcantly improves the predictive performance .

Original: See Fig. 
Revised: See Fig. D.4 for more details.

Original: qualitative
Revised: However, we donot use training phase ensemble because it signiﬁcantly increases the training time. Instead, we usespatial smoothing as a method that ensembles feature maps without sacriﬁcing training time. [Figure] [Figure]

Original: Loss landscape.
Revised: Hessian eigenvalue outliers , i.e., they ﬂatten the loss landscapes. Compare with Fig.

Original: We visualizes the loss landscapes (Li et al.,2017), the contours of NLL on training dataset.
Revised: We visualizes the loss landscapes (Li et al., 2018), the contours of NLL on training dataset.

Original: Figure 7band Fig. C.2b shows that the loss landscapes of MC dropout ﬂuctuateand have irregular surfaces due to the randomness.
Revised: Figure 7b shows that the loss landscapesof MC dropout ﬂuctuate and have irregular surfaces due to therandomness.

Original: As Li et al.(2017)
Revised: As Li et al.

Original: ; Foret et al.
Revised: ; Foret et al.

Original: and Appendix D.1 pointed out, this maylead to poor generalization and predictive performance.
Revised: (2020) pointed out,this may lead to poor generalization and predictive performance.

Original: Spatialsmoothing reduces randomness as discussed above, and spatialsmoothing aids in optimization by stabilizing and ﬂattening the losslandscape of BNN as shown in Fig.
Revised: Spatial smoothing reduces randomness as discussed above, andspatial smoothing aids in optimization by stabilizing and ﬂatteningthe loss landscape of BNN as shown in Fig.

Original: 7c and Fig. C.2c.
Revised: 7c.

Original: Furthermore, we use Hessian to quantitatively represent the sharpness of the loss landscapes and training stability.
Revised: Furthermore, we use Hessian to quantitatively represent the sharpness of the loss landscapes.

Original: Figure 8 showsthe Hessian max eigenvalue spectra of the models in Fig.
Revised: Figure 8 shows the Hessian max eigenvalue spectra of the models in Fig.

Original: 7 with abatch size of 128, which reveals that spatial smoothing reduces themagnitude of Hessian eigenvalues and suppresses outliers.
Revised: 7 with a batch size of 128, whichreveals that spatial smoothing reduces the magnitude of Hessianeigenvalues and suppresses outliers.

Original: Sincelarge Hessian eigenvalue outliers disturb NN training (Ghorbaniet al., 2019), we come to the same conclusion that spatial smoothinghelps NN optimization.
Revised: Since large Hessian eigenvalues disturb NN training (Ghorbani et al., 2019), we come to thesame conclusion that spatial smoothing helps NN optimization.

Original: See Appendix C.1 for more details on the Hessian max eigenvalue spectra.
Revised: See Appendix C.1 for more details on the Hessian max eigenvalue spectra.

Original: In addition, from these observations and experiments, we propose the conjecture that the smoother the loss landscape, the better theuncertainty estimation, and vice versa.
Revised: In addition, from these observations, we propose the conjecturethat the ﬂatter the loss landscape, the better the uncertainty estimation, and vice versa.

Original: 2. R EVISITING G LOBAL A VERAGE P OOLING
Revised: 2. R EVISITING G LOBAL A VERAGE P OOLING

Original: The success of GAP classiﬁer in image classiﬁcation is indisputable.
Revised: The success of GAP classiﬁer in image classiﬁcation is indisputable.

Original: In terms of predictive performance, the initial motivation and the most widely accepted explanation for this success is that GAPprevents overﬁtting by using far fewer parameters than multi-layer perceptron (MLP) (Lin et al.,2013).
Revised: The initial motivation andthe most widely accepted explanation for this success is that GAP prevents overﬁtting by usingfar fewer parameters than multi-layer perceptron (MLP) (Lin et al., 2014).

Original: In addition, global max pooling (GMaxP), which uses the same number of parameters as GAP,shows an accuracy comparable to that of GAP (Zhou et al., 2016; Christlein et al., 2019).

Original: However, we discover that the explanation is poorly supported.
Revised: However, we discoverthat the explanation is poorly supported.

Original: Appendix C.1 and Table C.1 compares GAP with other classiﬁers including MLP. Contrary to popular belief, results suggest that MLP doesnot overﬁt the training dataset . 
Revised: We compares GAP with other classiﬁers including MLP,and contrary to popular belief, results suggest that MLP does not overﬁt the training dataset . MLPunderﬁts or gives comparable performance to GAP on the training dataset.

Original: On the test dataset, GAP provides better results compared with MLP.
Revised: On the test dataset, GAPprovides better results compared with MLP.

Original: The accuracy of GMaxP is comparable to that of GAP, which is consistent with prior works; however,in terms of NLL and ECE, the uncertainty of GMaxP is worse than that of GAP.
Revised: See Table C.1 for more details.

Original: Our argument is that GAP is an extreme case of spatial smoothing.
Revised: Our argument is that GAP is an extreme case of spatial smoothing.

Original: In other words, GAP is successfulbecause itspatially ensemblesthe feature map and smoothens the loss landscape to help optimization.
Revised: In other words, GAP is successfulbecause it ensembles feature maps and smoothens the loss landscape to help optimization.

Original: To support this claim, we visualizes the loss landscape of MLPclassiﬁer as shown in Fig.
Revised: To supportthis claim, we visualizes the loss landscape of MLP as shown in Fig.

Original: 7a.
Revised: 7a.

Original: It isirregular and chaotic compared to that of GAP as demonstrated by Fig.
Revised: It is chaotic compared tothat of GAP as shown in Fig.

Original: 7b and Fig. C.2a.
Revised: 7b.

Original: Likewise,as shown in Fig. C.1, GAP is robust against corrupted data, but MPL is not.
Revised: Likewise, GAP is robust against data corruption, but MPL is not.

Original: In conclusion, averagingfeature maps tends to help neural networks learn strong representations .

Original:  Hessian shows the consistentresults as shown in Fig. For more detailed discussion, see Appendix C.1.
Revised: See Fig. C.1 for more details. Hessian shows the consistent results as demonstrated by Fig. 8.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section presents two experiments.
Revised: This section presents two experiments.

Original: The ﬁrst experiment is image classiﬁcation through whichwe show that spatial smoothing not only improves the ensemble efﬁciency, but also the accuracy,uncertainty, and robustness of both deterministic NN and MC dropout.
Revised: The ﬁrst experiment is image classiﬁcation through whichwe show that spatial smoothing not only improves the ensemble efﬁciency, but also the accuracy,uncertainty, and robustness of both deterministic NN and MC dropout.

Original: The second experiment issemantic segmentation on data streams through which we show that spatial smoothing and temporalsmoothing (Park et al., 2021) are complementary.
Revised: The second experiment issemantic segmentation on data streams through which we show that spatial smoothing and temporalsmoothing (Park et al., 2021) are complementary.

Original: See Appendix A for more detailed conﬁgurations.
Revised: See Appendix A for more detailed conﬁgurations.

Revised: Ensemble Size 0.

Original: {+N LL 
+}
Revised: {+N LL 
+}

Revised: Ensemble Size 75.

Original: {+A cc u r a c y ( % ) 
+}
Revised: {+A cc u r a c y ( % ) 
+}

Revised: Ensemble Size 2.

Original:  Three metrics are measured in these experiments: NLL( ↓ 1 ), accuracy ( ↑ ), and expected calibrationerror (ECE, ↓ )
Revised: E C E ( % ) 0. Deterministic [Figure] Three metrics are measured in these experiments: NLL ( ↓ 1 ), accuracy ( ↑ ), and expected calibrationerror (ECE, ↓ )

Original: (Guo et al., 2017).
Revised: (Guo et al., 2017).

Original: NLL represents both accuracy and uncertainty, and is the mostwidely used as a proper scoring rule.
Revised: NLL represents both accuracy and uncertainty, and is the mostwidely used as a proper scoring rule.

Original: ECE measures discrepancy between accuracy and conﬁdence.
Revised: ECE measures discrepancy between accuracy and conﬁdence.

Original: 3. I MAGE C LASSIFICATION
Revised: 3. I MAGE C LASSIFICATION

Original: We mainly discuss ResNet-18 (He et al., 2016a) on CIFAR-100 (Krizhevsky et al., 2009).
Revised: We mainly discuss ResNet (He et al., 2016a).

Original: In Appendix E.1, we also discuss other settings that show the same trend: e.g., VGG (Simonyan & Zisserman, 2014), ResNeXt (Xie et al., 2017),pre-activation models (He et al., 2016a), and ResNet-50 on ImageNet (Russakovsky et al., 2015).
Revised: In Table E.1, we also discuss other settings that show thesame trend: e.g., VGG (Simonyan & Zisserman, 2015), ResNeXt (Xie et al., 2017), and pre-activationmodels (He et al., 2016a).

Original: Spatial smoothing also improves deep ensemble (Lakshminarayanan et al., 2017), another non-Bayesian probabilistic NN method.
Revised: Spatial smoothing also improves deep ensemble (Lakshminarayanan et al. ,2017), another non-Bayesian probabilistic NN method.

Original: See Appendix E.1.
Revised: See Fig. E.1.

Original: Performance.
Revised: Performance.

Original: Fig.
Revised: Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: E.1 show the predictive performances of ResNet-18 on CIFAR-100andvarious models including ResNet-50 on ImageNet, respectively.
Revised: 9 show the predictive performances of ResNet-18 on CIFAR-100and ResNet-50 on ImageNet, respectively.

Original: The results indicate that spatialsmoothing improves both accuracy and uncertainty in many respects.
Revised: The results indicate that spatial smoothing improves bothaccuracy and uncertainty in many respects.

Original: Let us be more speciﬁc.
Revised: Let us be more speciﬁc.

Original: First,spatial smoothing improves the efﬁciency of ensemble size.
Revised: First, spatial smoothing improvesthe efﬁciency of ensemble size.

Original: In these examples, the NLL of “MCdropout + spatial smoothing” with an ensemble size of 2 is comparable to  that of MC dropout withan ensemble size of 50.
Revised: In these examples, the NLL of “MC dropout + spatial smoothing”with an ensemble size of 2 is comparable to or even better than that of MC dropout with an ensemblesize of 50.

Original: In other words, “MC dropout + spatial smoothing” is 25 × faster than MCdropout with a similar predictive performance.
Revised: In other words, “MC dropout + spatial smoothing” is 25 × faster than MC dropout witha similar predictive performance.

Original: Second, the predictive performance of “MC dropout+ spatial smoothing” is better than that of MC dropout, at an ensemble size of 50.
Revised: Second, the predictive performance of “MC dropout + spatialsmoothing” is better than that of MC dropout, at an ensemble size of 50.

Original: Third, spatialsmoothing improves the predictive performance of deterministic NN, as well as MC dropout.
Revised: Third, spatial smoothingimproves the predictive performance of deterministic NN, as well as MC dropout.

Original: [Figure]
Revised: [Figure]

Original: improves the robustness.
Revised: improves the robustness.

Original: See
Revised: See

Original: Fig.
Revised: Fig.

Original: E 3 for 
Revised: E.3 for more details.

Original: Robustness.
Revised: Robustness.

Original: To evaluate robustness against data corruption, wemeasure predictive performance of ResNet-18 on CIFAR-100C (Hendrycks & Dietterich, 2019).
Revised: To evaluate robustness against data corruption, wemeasure predictive performance of ResNet-18 on CIFAR-100C (Hendrycks & Dietterich, 2018).

Original: This dataset consists of datacorrupted by 15 different types, each with 5 levels of intensityeach.
Revised: This dataset consists of datacorrupted by 15 different types, each with 5 levels of intensityeach.

Original: We use mean corruption NLL (mCNLL, ↓ ), the averagesof NLL over intensities and corruption types, to summarize theperformance of corrupted data in a single value. 
Revised: We use mean corruption NLL (mCNLL, ↓ ), the averagesof NLL over intensities and corruption types, to summarize theperformance of corrupted data in a single value. See Eq.

Revised: (32) fora more rigorous deﬁnition.

Original: Figure 9 showsthat spatial smoothing not only improves the efﬁciency but alsocorruption robustness across a whole range of ensemble size.
Revised: Figure 10 shows that spatial smoothingnot only improves the efﬁciency but also corruption robustnessacross a whole range of ensemble size.

Original: See Fig.
Revised: See Fig.

Original: E.3 for more details.
Revised: E.3 for moredetails.

Original: Spatial smoothing also improves adversarial robustness. See Appendix E.1. Consistency.
Revised: Spatial smoothing also improves adversarial robustnessand perturbation consistency ( ↑ ) (Hendrycks & Dietterich, 2018;

Original: Neural networks for image classiﬁcation should beinvariant against shift-transformation, but they are not (Engstromet al., 2019; Azulay & Weiss, 2019). We measure consistency ( ↑ )(Hendrycks & Dietterich, 2019; Zhang, 2019a), a metric to evaluate the shift-invariance of models,and the results show that spatial smoothing improves comsistency.
Revised: Zhang, 2019a), shift-transformation invariance.

Original: See Eq.
Revised: See Table E.2,

Original: (25) for a rigorousdeﬁnition of consistency. Qualitative and quantitative results are provided in Appendix E.
Revised: Table E.3, and Fig. E.4 for more details.

Original: 3. S EMANTIC S EGMENTATION
Revised: 3. S EMANTIC S EGMENTATION

Original: [Figure] Table 1 summarizes the result of semantic segmentation on CamVid dataset (Brostow et al.
Revised: Table 1 summarizes the result of semantic segmentation on CamVid dataset (Brostow et al.

Original: 2008)that consists of real-world 360 × 480 pixels videos.
Revised: 2008)that consists of real-world 360 × 480 pixels videos.

Original: The table shows that spatial smoothing improvespredictive performance, which is consistent with the image classiﬁcation experiment.
Revised: The table shows that spatial smoothing improvespredictive performance, which is consistent with the image classiﬁcation experiment.

Original: Moreover, theresult reveals that spatial smoothing and temporal smoothing (Park et al., 2021) are complementary .
Revised: Moreover, theresult reveals that spatial smoothing and temporal smoothing (Park et al., 2021) are complementary .

Original: See Appendix A for more detailed conﬁgurations.

Original: SeeAppendix E.2 and Table E.4 for more results.
Revised: See Table E.4 for more results.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Spatial smoothing can be compared with prior works in the following areas.
Revised: Spatial smoothing can be compared with prior works in the following areas.

Original: Anti-aliased CNNs.
Revised: Anti-aliased CNNs.

Original: Local means (Zhang, 2019a; Zou et al., 2020; Vasconcelos et al., 2020)were introduced for the shift-invariance of deterministic CNNs in image classiﬁcation.
Revised: Local means (Zhang, 2019a; Zou et al., 2020; Vasconcelos et al., 2020; Sinhaet al., 2020) were introduced for the shift-invariance of deterministic CNNs in image classiﬁcation.

Original: They weremotivated to prevent the aliasing effect of subsampling.
Revised: They were motivated to prevent the aliasing effect of subsampling.

Original: Although the local ﬁltering can result ina loss of information, Zhang (2019a) experimentally observed an increase in accuracy that wasbeyond expectation.
Revised: Although the local ﬁltering canresult in a loss of information, Zhang (2019a) experimentally observed an increase in accuracy thatwas beyond expectation.

Original: We provide a fundamental explanation for this phenomenon: Local means area spatial ensemble .
Revised: We provide a fundamental explanation for this phenomenon: Local meansare a spatial ensemble .

Original: An ensemble not only improves accuracy, but also uncertainty and robustnessof deterministic and Bayesian NNs.
Revised: An ensemble not only improves accuracy, but also uncertainty and robustnessof deterministic and Bayesian NNs.

Original: In Appendix F.1, we also show that the predictive performanceimprovement is not due to anti-aliasing of local mean.
Revised: In Fig. F.1, we also show that the predictive performanceimprovement is not due to anti-aliasing of local mean.

Original: See Appendix F for more discussion onlocal means.
Revised: See Appendix F for more discussion on localmeans.

Original: For a discussion on non-local means (Wang et al., 2018; Dosovitskiy et al., 2020), see Section 5.
Revised: For a discussion on non-local means (Wang et al., 2018) and self-attention (Dosovitskiy et al.,2021), see Section 5.

Original: Sampling-free BNNs.
Revised: Sampling-free BNNs.

Original: Sampling-free BNNs (Hernández-Lobato & Adams, 2015; Wang et al.
Revised: Sampling-free BNNs (Hernández-Lobato & Adams, 2015; Wang et al.

Original: ,2016; Wu et al., 2018) predict results based on a single or couple of NN executions.
Revised: ,2016; Wu et al., 2019) predict results based on a single or couple of NN executions.

Original: To this end, it isassumed that posterior and feature maps follow Gaussian distributions.
Revised: To this end, it isassumed that posterior and feature maps follow Gaussian distributions.

Original: However, the discrepancybetween reality and assumption accumulates in every NN layer.
Revised: However, the discrepancybetween reality and assumption accumulates in every NN layer.

Original: Consequently, to the best of ourknowledge, most of the sampling-free BNNs could only be applied to shallow models, such as LeNet,and were tested on small datasets.
Revised: Consequently, to the best of ourknowledge, most of the sampling-free BNNs could only be applied to shallow models, such as LeNet,and were tested on small datasets.

Original: Postels et al.
Revised: Postels et al.

Original: applied sampling-free BNNs to SegNet;nonetheless, Park et al.
Revised: applied sampling-free BNNs to SegNet;nonetheless, Park et al.

Original: (2021) argued that they do not predict well-calibrated results.
Revised: (2021) argued that they do not predict well-calibrated results.

Original: Efﬁcient deep ensembles.
Revised: Efﬁcient deep ensembles.

Original: Deep ensemble (Lakshminarayanan et al., 2017; Fort et al., 2019)
Revised: Deep ensemble (Lakshminarayanan et al., 2017; Fort et al., 2019)

Original: isanother probabilistic NN approach for predicting reliable results.
Revised: isanother probabilistic NN approach for predicting reliable results.

Original: BatchEnsemble (Wen et al., 2020;
Revised: BatchEnsemble (Wen et al., 2020;

Original: Dusenberry et al., 2020) ensembles over a low-rank subspace to make deep ensemble more efﬁcient.
Revised: Dusenberry et al., 2020) ensembles over a low-rank subspace to make deep ensemble more efﬁcient.

Original: Depth uncertainty network (Antorán et al., 2020) aggregates feature maps from different depths ofa single NN to predict results efﬁciently.
Revised: Depth uncertainty network (Antoran et al., 2020) aggregates feature maps from different depths ofa single NN to predict results efﬁciently.

Original: Despite being robust against data corruption, it providesweaker predictive performance compared to deterministic NN and MC dropout.
Revised: Despite being robust against data corruption, it providesweaker predictive performance compared to deterministic NN and MC dropout.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Revised: We propose spatial smoothing to improve the performance of BNN.

Revised: Three different perspectives,namely, feature map variance, Fourier analysis, and loss landscape, suggest that spatial smoothingensembles feature maps.

Original: The limitation of spatial smoothing is that designing its components requires inductive bias.
Revised: The limitation of spatial smoothing is that designing its components requiresinductive bias.

Original: Inother words, the optimal shape of the blur kernel is model-dependent.
Revised: In other words, the optimal shape of the blur kernel is model-dependent.

Original: We believe this problem canbe solved by introducing self-attention (Vaswani et al., 2017).
Revised: We believethis problem can be solved by introducing self-attention (Vaswani et al., 2017).

Original: Self-attentions for computer vision(Dosovitskiy et al., 2020; Touvron et al., 2020; Carion et al., 2020) can be deemed as trainableimportance-weighted ensembles of feature maps.
Revised: Self-attentions forcomputer vision (Dosovitskiy et al., 2021; Touvron et al., 2021; Carion et al., 2020) can be deemedas trainable importance-weighted ensembles of feature maps.

Original: The observation that Transformers are more robustthan expected (Bhojanapalli et al., 2021; Shao et al., 2021) supports this claim.
Revised: The observation that Transformers aremore robust than expected (Bhojanapalli et al., 2021; Shao et al., 2021) supports this claim.

Original: Therefore, usingself-attentions to generalize spatial smoothing would be a promising future work because it not onlyexpands our work, but also helps deepen our understanding of self-attention. 
Revised: Therefore,using self-attentions to generalize spatial smoothing would be a promising future work because it notonly expands our work, but also helps deepen our understanding of self-attention. R EPRODUCIBILITY S TATEMENT

Revised: To ensure reproducibility we provide comprehensive resources, such as code and experimental details.

Revised: The codebase will be released as open source under the Apache License 2.0.

Original:  A E XPERIMENTAL S ETUP AND D ATASETS
Revised: See the supplementalmaterial for the code. Appendix A provides the speciﬁcations of all models used in this work. Detailedexperimental setup including hyperparameters and ablation study are also available in Appendix Aand Appendix B. De-facto image datasets are used for all experiments as described in Appendix A A E XPERIMENTAL S ETUP AND D ATASETS

Original: We obtain the main experimental results with the Intel Xeon W-2123 Processor, 32GB memory, anda single GeForce RTX 2080 Ti for CIFAR (Krizhevsky et al., 2009) and CamVid (Brostow et al.
Revised: We obtain the main experimental results with the Intel Xeon W-2123 Processor, 32GB memory, anda single GeForce RTX 2080 Ti for CIFAR (Krizhevsky et al., 2009) and CamVid (Brostow et al.

Original: For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core
Revised: For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core

Original: Processor, 256GB memory, and four GeForce RTX 

Revised: Processor, 256GB memory, and four GeForce RTX

Original: Ti.
Revised: Ti.

Original: We conduct ablation studies with four
Revised: We conduct ablation studies with four

Original: Intel Intel Broadwell CPUs, 15GB memory, and a single NVIDIA T4.
Revised: Intel Intel Broadwell CPUs, 15GB memory, and a single NVIDIA T4.

Original: Models are implementedin PyTorch(Paszke et al., 2019).
Revised: Models are implementedin PyTorch(Paszke et al., 2019).

Original: The detailed conﬁgurations of image classiﬁcation and semanticsegmentation are as follows.
Revised: The detailed conﬁgurations of image classiﬁcation and semanticsegmentation are as follows.

Original: A. I MAGE C LASSIFICATION
Revised: A. I MAGE C LASSIFICATION

Original: We use VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016a), pre-activation ResNet (Heet al., 2016a), and ResNeXt (Xie et al., 2017) in image classiﬁcation.
Revised: We use VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016a), pre-activation ResNet (Heet al., 2016a), and ResNeXt (Xie et al., 2017) in image classiﬁcation.

Original: According to the structuresuggested by Zagoruyko & Komodakis (2016), each block of Bayesian NNs contains one MC dropoutlayer.
Revised: According to the structuresuggested by Zagoruyko & Komodakis (2016), each block of Bayesian NNs contains one MC dropoutlayer.

Original: NNs are trained using categorical cross-entropy loss and SGD optimizer with initial learning rate of0.1, momentum of 0.9, and weight decay of 5 × 10 − 4 .
Revised: NNs are trained using categorical cross-entropy loss and SGD optimizer with initial learning rate of0.1, momentum of 0.9, and weight decay of 5 × 10 − 4 .

Original: We also use multi-step learning rate schedulerwith milestones at 60, 130, and 160, and gamma of 0.2 on CIFAR, and with milestones at 30, 60,and 80, and gamma of 0.2 on ImageNet.
Revised: We also use multi-step learning rate schedulerwith milestones at 60, 130, and 160, and gamma of 0.2 on CIFAR, and with milestones at 30, 60,and 80, and gamma of 0.2 on ImageNet.

Original: We train NNs for 200 epochs with batch size of 128 on
Revised: We train NNs for 200 epochs with batch size of 128 on

Original: CIFAR, and for 90 epochs with batch size of 256 on ImageNet.
Revised: CIFAR, and for 90 epochs with batch size of 256 on ImageNet.

Original: We start training with gradual warmup(Goyal et al., 2017) for 1 epoch on CIFAR.
Revised: We start training with gradual warmup(Goyal et al., 2017) for 1 epoch on CIFAR.

Original: Basic data augmentations, namely random cropping andhorizontal ﬂipping, are used.
Revised: Basic data augmentations, namely random cropping andhorizontal ﬂipping, are used.

Original: One exception is the training of ResNeXt on ImageNet.
Revised: One exception is the training of ResNeXt on ImageNet.

Original: In this case, weuse the batch size of 128 and learning rate of 0.05 because of memory limitation.
Revised: In this case, weuse the batch size of 128 and learning rate of 0.05 because of memory limitation.

Original: We use hyperparameters that minimizes NLL of ResNet.
Revised: We use hyperparameters that minimizes NLL of ResNet.

Original: Table A.1 provides hyperparameters fordeterministic and Bayesian NNs.
Revised: Table A.1 provides hyperparameters fordeterministic and Bayesian NNs.

Original: For fair comparison, models with and without spatial smoothingshare hyperparameters such as MC dropout rate.
Revised: For fair comparison, models with and without spatial smoothingshare hyperparameters such as MC dropout rate.

Original: However, Fig.
Revised: However, Fig.

Original: A.1 shows that spatial smoothingimproves predictive performance of ResNet-18 at all dropout rates on CIFAR-100.
Revised: A.1 shows that spatial smoothingimproves predictive performance of ResNet-18 at all dropout rates on CIFAR-100.

Original: The defaultensemble size of MC dropout is 50.
Revised: The defaultensemble size of MC dropout is 50.

Original: We report averages of three evaluations, and error bars in ﬁguresrepresent min and max values.
Revised: We report averages of three evaluations, and error bars in ﬁguresrepresent min and max values.

Original: Standard deviations are omitted from tables for better visualization.
Revised: Standard deviations are omitted from tables for better visualization.

Original: See the source code released on GitHub for other details.
Revised: See the source code released on GitHub for other details.

Original: A. S EMANTIC S EGMENTATION
Revised: A. S EMANTIC S EGMENTATION

Original: We use U-Net (Ronneberger et al., 2015) in semantic segmentation.
Revised: We use U-Net (Ronneberger et al., 2015) in semantic segmentation.

Original: Following Bayesian SegNet(Kendall et al., 2017), Bayesian U-Net contains six MC dropout layers.
Revised: Following Bayesian SegNet(Kendall et al., 2017), Bayesian U-Net contains six MC dropout layers.

Original: We add spatial smoothingbefore each subsampling layer in U-Net encoder.
Revised: We add spatial smoothingbefore each subsampling layer in U-Net encoder.

Original: We use 5 previous predictions and decay rate ofe − 0 .
Revised: We use 5 previous predictions and decay rate ofe − 0 .

Original: 8 for temporal smoothing.
Revised: 8 for temporal smoothing.

Original: [Table]
Revised: [Table]

Original: CamVid consists of 720 × 960 pixels road scene video sequences.
Revised: CamVid consists of 720 × 960 pixels road scene video sequences.

Original: We resize the image bilinearly to360 × 480 pixels.
Revised: We resize the image bilinearly to360 × 480 pixels.

Original: We use a list reduced to 11 labels by following previous works, e.g. (Kendall & Gal,2017).
Revised: We use a list reduced to 11 labels by following previous works, e.g. (Kendall & Gal,2017).

Original: NNs are trained using categorical cross-entropy loss and Adam optimizer with initial learning rate of0 .
Revised: NNs are trained using categorical cross-entropy loss and Adam optimizer with initial learning rate of0 .

Original: 001 and β 1 of 0.9, and β 2 of 0.999.
Revised: 001 and β 1 of 0.9, and β 2 of 0.999.

Original: We train NN for 130 epoch with batch size of 3.
Revised: We train NN for 130 epoch with batch size of 3.

Original: The learningrate decreases to 0 . 

Revised: The learningrate decreases to 0 .

Original: at the 100 epoch.
Revised: at the 100 epoch.

Original: Random cropping and horizontal ﬂipping are used fordata augmentation.
Revised: Random cropping and horizontal ﬂipping are used fordata augmentation.

Original: Median frequency balancing is used to mitigate dataset imbalance.
Revised: Median frequency balancing is used to mitigate dataset imbalance.

Original: Other detailsfollow Park et al.
Revised: Other detailsfollow Park et al.

Original: B A BLATION S TUDY
Revised: B A BLATION S TUDY

Original: ReLU tanh
Revised: [Figure] [Figure] tanh

Original: ReLU
Revised: [Figure]

Original: Constant
Revised: Constant

Original: BatchNorm
Revised: BatchNorm

Original: The probabilistic spatial smoothing proposed in thispaper consists of two components: Prob and Blur .
Revised: The probabilistic spatial smoothing proposed in thispaper consists of two components: Prob and Blur .

Original: Thissection explores several candidates for each componentand their properties.
Revised: Thissection explores several candidates for each componentand their properties.

Original: B. Prob : F EATURE MAPS TO PROBABILITIES
Revised: B. Prob : F EATURE MAPS TO PROBABILITIES

Original: We deﬁne Prob as a composition of an upperbounded function and ReLU , a function that imposesthe lower bound of zero.
Revised: We deﬁne Prob as a composition of an upperbounded function and ReLU , a function that imposesthe lower bound of zero.

Original: Fig.
Revised: Fig.

Original: B.1 shows widely usedupper-bounded functions: tanh τ ( x ) =
Revised: B.1 shows widely usedupper-bounded functions: tanh τ ( x ) =

Original: τ tanh ( x /τ ) ,
Revised: τ tanh ( x /τ ) ,

Original: ReLU6 ( x ) =
Revised: ReLU6 ( x ) =

Original: min ( max ( x , 6) , 0) , and constant scalingwhich is x /τ .
Revised: min ( max ( x , 6) , 0) , and constant scalingwhich is x /τ .

Original: Table B.1 shows the predictive performance improvement by Prob with various upper-bounded functions on
Revised: Table B.1 shows the predictive performance improvement by Prob with various upper-bounded functions on

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N
Revised: N

Original: LL
Revised: LL

Original: [Figure]
Revised: [Figure]

Original: CIFAR-100.
Revised: CIFAR-100.

Original: In this experiment, we use models with MC dropout, and τ = 5 for constant scaling.
Revised: In this experiment, we use models with MC dropout, and τ = 5 for constant scaling.

Original: The results indicate that upper-bounded functions with ReLU tend to improve accuracy and uncertainty at the same time.
Revised: The results indicate that upper-bounded functions with ReLU tend to improve accuracy and uncertainty at the same time.

Original: In addition, they show that Prob and Blur are complementary.
Revised: In addition, they show that Prob and Blur are complementary.

Original: The bestresults are obtained when using both Prob and Blur .
Revised: The bestresults are obtained when using both Prob and Blur .

Original: See Appendix E.1 for a detailed discussionof the complementary relationship between Prob and Blur .
Revised: See Appendix E.1 for a detailed discussionof the complementary relationship between Prob and Blur .

Original: For the main experiments, we use thecomposition of tanh τ and ReLU as Prob , because the hyperparameter of constant scaling is highlydependent on dataset and model.
Revised: For the main experiments, we use thecomposition of tanh τ and ReLU as Prob , because the hyperparameter of constant scaling is highlydependent on dataset and model.

Original: Temperature.
Revised: Temperature.

Original: The characteristics of temperature-scaled tanh depends on τ .
Revised: The characteristics of temperature-scaled tanh depends on τ .

Original: Figure B.2 plotstanh τ
Revised: Figure B.2 plotstanh τ

Original: and
Revised: and

Original: their ﬁrst derivatives with various temperatures.
Revised: their ﬁrst derivatives with various temperatures.

Original: As shown in this ﬁgure, tanh τ has acouple of useful properties.
Revised: As shown in this ﬁgure, tanh τ has acouple of useful properties.

Original: First, tanh τ has an upper bound of τ .
Revised: First, tanh τ has an upper bound of τ .

Original: Second, the ﬁrst derivative oftanh τ at x = 0 does not depend on τ .
Revised: Second, the ﬁrst derivative oftanh τ at x = 0 does not depend on τ .

Original: Fig.
Revised: Fig.

Original: B.3 shows the predictive performance of ResNet-18 with MC dropout and spatial smoothing forthe temperature on CIFAR-100.
Revised: B.3 shows the predictive performance of ResNet-18 with MC dropout and spatial smoothing forthe temperature on CIFAR-100.

Original: In this ﬁgure, the accuracy increases as the temperature increases .
Revised: In this ﬁgure, the accuracy increases as the temperature increases .

Original: Interms of ECE, NN predicts more underconﬁdent results as τ decreases .
Revised: Interms of ECE, NN predicts more underconﬁdent results as τ decreases .

Original: It is a misinterpretation thatthe result is overconﬁdent at low τ because ECE is high.
Revised: It is a misinterpretation thatthe result is overconﬁdent at low τ because ECE is high.

Original: By deﬁnition, ECE relies on the absolutevalue of the difference between conﬁdence and accuracy.
Revised: By deﬁnition, ECE relies on the absolutevalue of the difference between conﬁdence and accuracy.

Original: In this example, at low τ , the accuracy isgreater than the conﬁdence, which leads to a high ECE.
Revised: In this example, at low τ , the accuracy isgreater than the conﬁdence, which leads to a high ECE.

Original: Moreover, at τ = 0 . 2 , ECE with N = 50 isgreater than that with N = 1 , which means that the result is severely underconﬁdent.
Revised: Moreover, at τ = 0 . 2 , ECE with N = 50 isgreater than that with N = 1 , which means that the result is severely underconﬁdent.

Original: NLL, a metricrepresenting both accuracy and uncertainty, is minimized when the accuracy and the uncertainty arebalanced.
Revised: NLL, a metricrepresenting both accuracy and uncertainty, is minimized when the accuracy and the uncertainty arebalanced.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: k = (1 , 2 , 1) 0.
Revised: k = (1 , 2 , 1) 0.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Ensemble Size 0.
Revised: Ensemble Size 0.

Original: N LL 

Revised: N LL

Original: Ensemble Size
Revised: Ensemble Size

Original: A cc u r a c y ( % ) 

Revised: A cc u r a c y ( % )

Original: Ensemble Size 2.
Revised: Ensemble Size 2.

Original: E C E ( % ) 

Revised: E C E ( % )

Original: Ensemble Size 0.
Revised: Ensemble Size 0.

Original: N
Revised: N

Original: LL
Revised: LL

Original: None s s s s
Revised: None s s s s

Original: We report predictive performance of ResNet-18 with 
Revised: We report predictive performance of ResNet-18 with one smoothing

Original: B. Blur : A VERAGING NEIGHBORING PROBABILITIES
Revised: B. Blur : A VERAGING NEIGHBORING PROBABILITIES

Original: Blur is a depth-wise convolution with a kernel.
Revised: Blur is a depth-wise convolution with a kernel.

Original: The kernel given by Eq.
Revised: The kernel given by Eq.

Original: (8) is derived from variousk s such as k ∈ { (1) , (1 , 1) , (1 , 2 , 1) , (1 , 4 , 6 , 4 , 1) } .
Revised: (8) is derived from variousk s such as k ∈ { (1) , (1 , 1) , (1 , 2 , 1) , (1 , 4 , 6 , 4 , 1) } .

Original: In these examples, if | k | is 1, Blur is identity.
Revised: In these examples, if | k | is 1, Blur is identity.

Original: If | k | is 2, Blur is a box blur, which is used in the main experiments.
Revised: If | k | is 2, Blur is a box blur, which is used in the main experiments.

Original: If | k | is 3 or 5, Blur is anapproximated Gaussian blur.
Revised: If | k | is 3 or 5, Blur is anapproximated Gaussian blur.

Original: Table B.2 shows predictive performance of models using spatial smoothing with the kernels on
Revised: Table B.2 shows predictive performance of models using spatial smoothing with the kernels on

Original: CIFAR-100.
Revised: CIFAR-100.

Original: This results show that most kernels improve both accuracy and uncertainty .
Revised: This results show that most kernels improve both accuracy and uncertainty .

Original: However,the most effective kernel size depends on the model.
Revised: However,the most effective kernel size depends on the model.

Original: Refer to Appendix E.1 for discussion of modelproperties and the performance improvements.
Revised: Refer to Appendix E.1 for discussion of modelproperties and the performance improvements.

Original: B. P OSITION OF SPATIAL SMOOTHING .
Revised: B. P OSITION OF SPATIAL SMOOTHING .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, the magnitude of uncertainty tends to increase as the depth increases.
Revised: 5, the magnitude of uncertainty tends to increase as the depth increases.

Original: Therefore,we expect that spatial smoothing close to the output layer will mainly drive performance improvement.
Revised: Therefore,we expect that spatial smoothing close to the output layer will mainly drive performance improvement.

Original: We investigate the predictive performance of models with MC dropout using only one spatial smoothing layer.
Revised: We investigate the predictive performance of models with MC dropout using only one spatial smoothing layer.

Original: Figure B.5 shows the predictive performance of ResNet-18 with one spatial smoothing aftereach stage on CIFAR-100.
Revised: Figure B.5 shows the predictive performance of ResNet-18 with one spatial smoothing aftereach stage on CIFAR-100.

Original: The results suggest that spatial smoothing after s3 is the most importantfor improving performance.
Revised: The results suggest that spatial smoothing after s3 is the most importantfor improving performance.

Original: Surprisingly, spatial smoothing after s4 is the least important.
Revised: Surprisingly, spatial smoothing after s4 is the least important.

Original: This isbecause GAP, the most extreme case of spatial smoothing, already exists there.
Revised: This isbecause GAP, the most extreme case of spatial smoothing, already exists there.

Original: C R EVISITING P RIOR W ORKS
Revised: C R EVISITING P RIOR W ORKS

Original: As mentioned in Section 2, prior works—namely, GAP, pre-activation, and ReLU6 —are spacial casesof spatial smoothing.
Revised: As mentioned in Section 2, prior works—namely, GAP, pre-activation, and ReLU6 —are spacial casesof spatial smoothing.

Original: This section discusses them in detail.
Revised: This section discusses them in detail.

Original: C. G LOBAL A VERAGE P OOLING
Revised: C. G LOBAL A VERAGE P OOLING

Original: The composition of GAP and a fully connected layer is the most popular classiﬁer in classiﬁcationtasks.
Revised: The composition of GAP and a fully connected layer is the most popular classiﬁer in classiﬁcationtasks.

Original: The original motivation and the most widely accepted explanation for the success is that GAPclassiﬁer prevents overﬁtting because it uses signiﬁcantly fewer parameters than MLP (Lin et al.,2013).
Revised: The original motivation and the most widely accepted explanation for the success is that GAPclassiﬁer prevents overﬁtting because it uses signiﬁcantly fewer parameters than MLP (Lin et al.,2014).

Original: To verify this claim, we measure the predictive performance of MLP, GAP, and global maxpooling (GMaxP), a classiﬁer that uses the same number of parameters as GAP, on training dataset.
Revised: To verify this claim, we measure the predictive performance of MLP, GAP, and global maxpooling (GMaxP), a classiﬁer that uses the same number of parameters as GAP, on training dataset.

Original: Predictive performance.
Revised: Predictive performance.

Original: Table C.1 shows the experimental results on the training and the testdataset of CIFAR-100, suggesting that the explanation is poorly supported.
Revised: Table C.1 shows the experimental results on the training and the testdataset of CIFAR-100, suggesting that the explanation is poorly supported.

Original: On both the trainingand the test dataset, most predictive performance of MLP is worse than that of GAP.
Revised: On both the trainingand the test dataset, most predictive performance of MLP is worse than that of GAP.

Original: It is a counterintuitive result meaning that MLP do not overﬁt the training dataset .
Revised: It is a counterintuitive result meaning that MLP do not overﬁt the training dataset .

Original: In addition, the performance
Revised: In addition, the performance

Original: [Table]
Revised: [Table]

Original: MLP 0.
Revised: MLP 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: GMaxP
Revised: GMaxP

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: GMedP
Revised: GMedP

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: N LL
Revised: N LL

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: improvement by GAP is remarkable in VGG, which has irregular loss landscape.
Revised: improvement by GAP is remarkable in VGG, which has irregular loss landscape.

Original: The predictiveperformance of GMaxP is better than that of MLP, but worse than that of GAP.
Revised: The predictiveperformance of GMaxP is better than that of MLP, but worse than that of GAP.

Original: This shows that usingfewer parameters partially helps to improve predictive performance; however, it is insufﬁcient toexplain
Revised: This shows that usingfewer parameters partially helps to improve predictive performance; however, it is insufﬁcient toexplain

Original: the predictive performance improvement by GAP.
Revised: the predictive performance improvement by GAP.

Original: Finally, global median pooling (GMedP)provides better predictive performance than GMaxP. It implies that using other noise reductionmethods instead of average pooling helps to improve predictive performance.
Revised: Finally, global median pooling (GMedP)provides better predictive performance than GMaxP. It implies that using other noise reductionmethods instead of average pooling helps to improve predictive performance.

Original: Robustness.
Revised: Robustness.

Original: To evaluate the robustness of the classiﬁers, we measure the predictive performance of
Revised: To evaluate the robustness of the classiﬁers, we measure the predictive performance of

Original: ResNet-18 using MC dropout with the classiﬁers on CIFAR-100-C. Figure C.1 shows the experimentalresults.
Revised: ResNet-18 using MC dropout with the classiﬁers on CIFAR-100-C. Figure C.1 shows the experimentalresults.

Original: This ﬁgure suggests that MLP is not robust against data corruption, as we would expect.
Revised: This ﬁgure suggests that MLP is not robust against data corruption, as we would expect.

Original: Interms of accuracy, the robustness of GMaxP and GMedP is relatively comparable to that of GAP;however, in terms of uncertainty, GAP is the most robust .
Revised: Interms of accuracy, the robustness of GMaxP and GMedP is relatively comparable to that of GAP;however, in terms of uncertainty, GAP is the most robust .

Original: These are consistent results with otherspatial smoothing experiments.
Revised: These are consistent results with otherspatial smoothing experiments.

Original: Loss landscape visualization.
Revised: Loss landscape visualization.

Original: To understand the mechanism of GAP performance improvement,we investigate the loss landscape.
Revised: To understand the mechanism of GAP performance improvement,we investigate the loss landscape.

Original: Figure C.2 shows the loss landscape sequences of ResNet with
Revised: Figure C.2 shows the loss landscape sequences of ResNet with

Original: MC dropout.
Revised: MC dropout.

Original: In this ﬁgure, each sequence shares the bases, but they ﬂuctuate due to the randomnessof the MC dropout.
Revised: In this ﬁgure, each sequence shares the bases, but they ﬂuctuate due to the randomnessof the MC dropout.

Original: Figure C.2a is the loss landscape of the model using MLP classiﬁer instead of
Revised: Figure C.2a is the loss landscape of the model using MLP classiﬁer instead of

Original: GAP classiﬁer.
Revised: GAP classiﬁer.

Original: The loss landscape is chaotic and irregular, resulting in hindering and destabilizing
Revised: The loss landscape is chaotic and irregular, resulting in hindering and destabilizing

Original: NN optimization.
Revised: NN optimization.

Original: Fig.
Revised: Fig.

Original: C.2b is loss landscape sequence of ResNet with GAP classiﬁer.
Revised: C.2b is loss landscape sequence of ResNet with GAP classiﬁer.

Original: Since GAPensembles all of the feature map points at the last stage, it ﬂattens and stabilizes the loss landscape.
Revised: Since GAPensembles all of the feature map points at the last stage, it ﬂattens and stabilizes the loss landscape.

Original: Likewise, as shown in Fig. C.2c, spatial smoothing layers at the end of all stages also ﬂattens andstabilizes the loss landscape.
Revised: Likewise, as shown in Fig. C.2c, spatial smoothing layers at the end of all stages also ﬂattens andstabilizes the loss landscape.

Original: Hessian eigenvalue spectra.
Revised: Hessian eigenvalue spectra.

Original: To evaluate the smoothness of the loss landscapes quantitatively,we also investigate their Hessians at the optimized weights.
Revised: To evaluate the smoothness of the loss landscapes quantitatively,we also investigate their Hessians at the optimized weights.

Original: In particular, we calculate Hessianeigenvalue spectra (Ghorbani et al., 2019), distributions of Hessian eigenvalues, to show how spatialsmoothing helps NN optimization.
Revised: In particular, we calculate Hessianeigenvalue spectra (Ghorbani et al., 2019), distributions of Hessian eigenvalues, to show how spatialsmoothing helps NN optimization.

Original: To this end, we try to use stochastic Lanczos quadrature algorithm implemented by Yao et al.
Revised: To this end, we try to use stochastic Lanczos quadrature algorithm implemented by Yao et al.

Original: However, the problem is that the model with MLP classiﬁerrequires a lot of memory while the algorithm is memory inefﬁcient.
Revised: However, the problem is that the model with MLP classiﬁerrequires a lot of memory while the algorithm is memory inefﬁcient.

Original: In the training phase, we calculate the mean gradients with respect to mini-batches, rather than theentire dataset.
Revised: In the training phase, we calculate the mean gradients with respect to mini-batches, rather than theentire dataset.

Original: Therefore, it may be reasonable to investigate the properties of the Hessian “minibatch-wisely”.
Revised: Therefore, it may be reasonable to investigate the properties of the Hessian “minibatch-wisely”.

Original: For that purpose, we propose a method, Hessian max eigenvalue spectra , that evaluatesthe distribution of “Hessian’s maximum eigenvalues for one mini-batch”.
Revised: For that purpose, we propose a method, Hessian max eigenvalue spectra , that evaluatesthe distribution of “Hessian’s maximum eigenvalues for one mini-batch”.

Original: We use power iteration toproduce only the greatest eigenvalue of the Hessian.
Revised: We use power iteration toproduce only the greatest eigenvalue of the Hessian.

Original: This algorithm is easy to implement and requiressigniﬁcantly less memory and computational cost, compared with stochastic Lanczos quadraturewith respect to entire dataset.
Revised: This algorithm is easy to implement and requiressigniﬁcantly less memory and computational cost, compared with stochastic Lanczos quadraturewith respect to entire dataset.

Original: With this method, we can investigate the Hessian of NNs with MLPclassiﬁers, which would require a lot of GPU memory.
Revised: With this method, we can investigate the Hessian of NNs with MLPclassiﬁers, which would require a lot of GPU memory.

Original: [Figure]
Revised: [Figure]

Original: We provide Hessian eigenvalue spectra of ResNet-18 with MC dropout on CIFAR-100.
Revised: We provide Hessian eigenvalue spectra of ResNet-18 with MC dropout on CIFAR-100.

Original: Compare to Fig.
Revised: See also

Original: Figure 8 shows the Hessian max eigenvalue spectra of GAP classiﬁer models with and without spatial smoothing layers.
Revised: Figure 8 shows the Hessian max eigenvalue spectra of GAP classiﬁer models with and without spatial smoothing layers.

Original: As Li et al.(2017)
Revised: As Li et al.(2018)

Original: ; Foret et al.
Revised: ; Foret et al.

Original: and Appendix D.1 pointed out, Hessianeigenvalue outliers disturb NN training.
Revised: and Appendix D.3 pointed out, Hessianeigenvalue outliers disturb NN training.

Original: This ﬁgure explicitly showthat the GAP and spatial smoothing reduce the magnitude of the
Revised: This ﬁgure explicitly showthat the GAP and spatial smoothing reduce the magnitude of the

Original: Hessian eigenvalues and suppress the outliers, which leads to thesame result as the previous visualizations: GAP as well as spatialsmoothing smoothen the loss landscape.
Revised: Hessian eigenvalues and suppress the outliers, which leads to thesame result as the previous visualizations: GAP as well as spatialsmoothing smoothen the loss landscape.

Original: In conclusion, averaging feature map points tends to help neural network optimizationby smoothing, ﬂattening, and stabilizing the loss landscape .
Revised: In conclusion, averaging feature map points tends to help neural network optimizationby smoothing, ﬂattening, and stabilizing the loss landscape .

Original: Weobserve a similar phenomenon for deterministic NNs.
Revised: Weobserve a similar phenomenon for deterministic NNs.

Original: We alsoevaluate the Hesse eigenvalue spectrum as shown in Fig.
Revised: We alsoevaluate the Hesse eigenvalue spectrum as shown in Fig.

Original: C.3, andit leads to the same conclusion.
Revised: C.3, andit leads to the same conclusion.

Original: In these experiments, we use MLP incorporating dropout layerswith a rate of 50% as the classiﬁer.
Revised: In these experiments, we use MLP incorporating dropout layerswith a rate of 50% as the classiﬁer.

Original: Since the dropout is one ofthe factors that makes MLP underﬁt the training dataset, we alsoevaluate MLP using dropouts with a rate of 0%.
Revised: Since the dropout is one ofthe factors that makes MLP underﬁt the training dataset, we alsoevaluate MLP using dropouts with a rate of 0%.

Original: Nevertheless, theresults still shows that the predictive performance of MLP is worsethan that of GAP on the training dataset.
Revised: Nevertheless, theresults still shows that the predictive performance of MLP is worsethan that of GAP on the training dataset.

Original: Moreover, it severely degrades predictive performance of
Revised: Moreover, it severely degrades predictive performance of

Original: ResNet on the test dataset.
Revised: ResNet on the test dataset.

Original: C. P RE - ACTIVATION
Revised: C. P RE - ACTIVATION

Original: He et al.
Revised: He et al.

Original: (2016b) experimentally showed that the pre-activation arrangement, in which the activation
Revised: (2016b) experimentally showed that the pre-activation arrangement, in which the activation

Original: ReLU ◦ BatchNorm is placed before the convolution, improves the accuracy of ResNet.
Revised: ReLU ◦ BatchNorm is placed before the convolution, improves the accuracy of ResNet.

Original: Since γ s ofmost BatchNorm s in CNNs are near-zero (Frankle et al., 2020), BatchNorm s reduce the magnitudeof feature maps.
Revised: Since γ s ofmost BatchNorm s in CNNs are near-zero (Frankle et al., 2021), BatchNorm s reduce the magnitudeof feature maps.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: B.1, constant scaling is a non-trainable BatchNorm with nobias, and it also reduces the magnitude of feature map.
Revised: B.1, constant scaling is a non-trainable BatchNorm with nobias, and it also reduces the magnitude of feature map.

Original: In Appendix B.1, we show that constantscaling improves predictive performance.
Revised: In Table B.1, we show that constant scalingimproves predictive performance.

Original: Considering the similarity between Prob with constantscaling and conventional activation, i.e., the similarity between ReLU ◦ ConstantScaling and ReLU ◦ BatchNorm , we ﬁnd that the pre-activation arrangement improves uncertainty as well asaccuracy, because convolutions act as a Blur .
Revised: Considering the similarity between Prob with constant scaling andconventional activation, i.e., the similarity between ReLU ◦ ConstantScaling and ReLU ◦ BatchNorm ,we ﬁnd that the pre-activation arrangement improves uncertainty as well as accuracy, becauseconvolutions act as a Blur .

Original: To show this, we change the post-activation of all layers to pre-activation, and measure the predictiveperformance.
Revised: To show this, we change the post-activation of all layers to pre-activation, and measure the predictiveperformance.

Original: For ResNet, we follow the original paper by He et al.
Revised: For ResNet, we follow the original paper by He et al.

Original: (2016b).
Revised: (2016b).

Original: Table C.2 showsthe predictive performance of models with pre-activation.
Revised: Table C.2 showsthe predictive performance of models with pre-activation.

Original: The results suggests that pre-activationimproves both accuracy and uncertainty in most cases.
Revised: The results suggests that pre-activationimproves both accuracy and uncertainty in most cases.

Original: For deterministic VGG-19, pre-activationsigniﬁcantly degrades accuracy but improves NLL.
Revised: For deterministic VGG-19, pre-activationsigniﬁcantly degrades accuracy but improves NLL.

Original: In conclusion, they imply that pre-activation is aspecial case of spatial smoothing.
Revised: In conclusion, they imply that pre-activation is aspecial case of spatial smoothing.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: (2018) argued that BatchNorm helps in optimization by ﬂattening the loss landscape.
Revised: (2018) argued that BatchNorm helps in optimization by ﬂattening the loss landscape.

Original: We show that spatial smoothing ﬂattens and smoothens the loss landscape, which is a consistentexplanation.
Revised: We show that spatial smoothing ﬂattens and smoothens the loss landscape, which is a consistentexplanation.

Original: It will be interesting to investigate if BatchNorm helps in ensembling feature maps.
Revised: It will be interesting to investigate if BatchNorm helps in ensembling feature maps.

Original: C. R E LU6
Revised: C. R E LU6

Original: ReLU6 was experimentally introduced to improve predictive performance (Krizhevsky & Hinton,2010).
Revised: ReLU6 was experimentally introduced to improve predictive performance (Krizhevsky & Hinton,2010).

Original: Sandler et al.
Revised: Sandler et al.

Original: (2018) used “ ReLU6 as the non linearity because of its robustness when used
Revised: (2018) used “ ReLU6 as the non linearity because of its robustness when used

Original: [Table]
Revised: [Table]

Original: with low-precision computation”.
Revised: with low-precision computation”.

Original: In Appendix B.1, we show that ReLU6 s at the end of stages helps toensemble spatial information by transforming the feature map to Bernoulli distributions.
Revised: In Table B.1, we show that ReLU6 s at the end of stages helps toensemble spatial information by transforming the feature map to Bernoulli distributions.

Original: Since spatialsmoothing improves robustness against data corruption, it seems reasonable that ReLU6 is robust tolow-precision computation.
Revised: Since spatialsmoothing improves robustness against data corruption, it seems reasonable that ReLU6 is robust tolow-precision computation.

Original: A more abundant investigation into this topic is promising future works.
Revised: A more abundant investigation into this topic is promising future works.

Original: We measure the predictive performance of NNs using all activations as ReLU6 instead of ReLU .
Revised: We measure the predictive performance of NNs using all activations as ReLU6 instead of ReLU .

Original: However, in contrast to the results in Appendix B.1, the results are not consistent.
Revised: However, in contrast to the results in Table B.1, the results are not consistent.

Original: We speculate thatthe reason is that a lot of ReLU6 s overly regularize NNs.
Revised: We speculate that thereason is that a lot of ReLU6 s overly regularize NNs.

Original: D E XTENDED A NALYSIS OF H OW SPATIAL SMOOTHING W ORKS
Revised: D E XTENDED A NALYSIS OF H OW SPATIAL SMOOTHING W ORKS

Original: This section provides further explanation of the analysis in Section 2.2.
Revised: This section provides further explanation of the analysis in Section 2.2.

Original: D. 
Revised: D. N EIGHBORING F EATURE M APS IN CNN S A RE S IMILAR

Revised: This work exploits the spatial consistency of feature maps, i.e., neighboring feature maps in CNNsare similar . Below, we theoretically and empirically prove the spatial consistency. Moreover, thisspatial consistency of feature maps holds even if the input data is spatially inconsistent.

Original: D
Revised: Consider a single-layer convolutional neural network with one channel: [Equation] [Figure] where ∗ is convolution operator with a kernel of size k , y is feature map output, w is kernel weight, and x is input random variable .

Original: ROPOUT
Revised: Then, the covariance of two neighboring feature maps is: [Equation] where σ 2 ( x i − l +1 ) is the variance of x i − l +1 . Therefore, Cov ( y i , y i +1 ) is non-zero for randomlyinitialized weights.

Revised: If x is iid , i.e., Cov ( x i , x j )

Original: S
Revised: =

Revised: δ ij σ 2 ( x i ) where δ ij is the Kronecker delta, theremainders in Eq.

Revised: (13) vanish.

Revised: For example, the covariance of two neighboring feature map points in a CNN with a kernel size ofis:

Original: HARPENS THE L
Revised: [Equation] [Equation] When x i is iid , the covariance is: [Equation]

Revised: Since it is non-zero, the neighboring feature maps y 1 and y 2 are correlated.

Original: OSS
Revised: Experiment. To demonstrate the spatial consistency of feature maps empirically, we providefeature map covariances of randomly initialized single-layer CNN and ﬁve-layer CNN with ReLUnon-linearity. In this experiment, the input values are Gaussian random noises. As shown in Fig.

Revised: D.1a,one convolutional layer correlates neighboring feature map points.

Revised: Fig.

Revised: D.1b shows that multipleconvolutional layers correlate one feature map with distant feature maps.

Revised: Moreover, the feature mapsin deep CNNs have a stronger relationship with neighboring feature maps.

Original: L ANDSCAPE
Revised: D. E NSEMBLE F ILTERS H IGH -F REQUENCY S IGNALS

Revised: Following the notation of Eq. (3), the ensemble is convolution of importance π and prediction p : [Equation]

Revised: [Equation]

Revised: where π i,j = π ( x i | x j ) and p i = p ( y | x i , w i ) .

Revised: To show that this ensemble is low-pass ﬁlter, we apply the convolution N times:

Revised: [Equation]

Revised: Since π is probability, i.e., (cid:80)

Revised: i π i,j = 1 , π ∗ · · · ∗ π is the probability for the sum of N randomvariables from π , i.e, φ + · · · + φ ∼ π ∗ · · · ∗ π where φ ∼ π .

Revised: By deﬁnition, an operator islow-pass ﬁlter if and only if the high frequency component vanishes when the operator is appliedinﬁnitely.

Revised: Therefore, ensemble with π is low-pass ﬁlter because Var ( φ + · · · + φ ) =

Original: AND E
Revised: N Var ( φ ) and F [ π ∗ · · · ∗ π ∗ p ] = F [ π ∗ · · · ∗ π ] F [ p ] where F is Fourier transform. Experiment.

Revised: Since blur ﬁlter is low-pass ﬁlter, probabilistic spatial smoothing is also low-passﬁlter.

Revised: In Section 2.2, at the end of the stage 1, we show that MC dropout adds high-frequency noise tofeature maps, and spatial smoothing effectively removes it.

Revised: As shown in Fig.

Original: NSEMBLE S MOOTHENS
Revised: D.2, we observe thesame phenomena at other stages. In addition, Fig. 6c shows that CNNs are vulnerable to high-frequency random noise. Interestingly, italso shows that CNNs are robust against noise with frequencies from 0 .

Original: I T
Revised: 6 π to 0 . 8 π , corresponding toapproximately 3 pixel periods. Since the receptive ﬁelds of convolutions are 3 × 3, the noise with aperiod smaller than the size is averaged out by convolutions. For the same reason, convolutions areparticularly vulnerable against the noise with a frequency of 0 .

Revised: 3 π , corresponding to a period of 6pixel.

Revised: D. R ANDOMNESS S HARPENS L OSS L ANDSCAPE , AND E NSEMBLE S MOOTHENS I T

Original: Ws show that the randomness of BNNs hinder and destabilize NN training because it causes the losslandscape and its gradient to ﬂuctuate from moment to moment.
Revised: Ws show that the randomness of BNNs hinder and destabilize NN training because it causes the losslandscape and its gradient to ﬂuctuate from moment to moment.

Original: In other words, the randomness,such as dropout, sharpens the loss landscape.
Revised: In other words, the randomness,such as dropout, sharpens the loss landscape.

Original: To show the claim theoretically, we use Foret et al.
Revised: To show the claim theoretically, we use Foret et al.

Original: (2020)’s deﬁnition of sharpness with respect to training dataset D :
Revised: (2020)’s deﬁnition of sharpness with respect to training dataset D :

Original: [Equation]
Revised: [Equation]

Original:  where L D is NLL loss, w is NN weight, δ w is small weight perturbation, and ρ is neighborhoodradius.
Revised: 18) where L D is NLL loss, w is NN weight, δ w is small weight perturbation, and ρ is neighborhoodradius.

Original: Therefore, as dropout rate—and the magnitude of δ w —increases, the sharpness increases.
Revised: Therefore, as dropout rate—and the magnitude of δ w —increases, the sharpness increases.

Original: We next calculate the sharpness more rigorously.
Revised: We next calculate the sharpness more rigorously.

Original: Let p i ∈
Revised: Let p i ∈

Original: (0 , 1] be a conﬁdence of one NN prediction, and ¯ p ( N ) be a conﬁdence of N ensemble, i.e., ¯ p ( N ) =
Revised: (0 , 1] be a conﬁdence of one NN prediction, and ¯ p ( N ) be a conﬁdence of N ensemble, i.e., ¯ p ( N ) =

Original: 1 N (cid:80) Ni =1 p i .
Revised: 1 N (cid:80) Ni =1 p i .

Original: Then, the variance of the NLL 
Revised: Then, the variance of the NLL loss is:

Original: [Figure]
Revised: [Equation]

Original: loss is:

Original: [Equation]
Revised: [Equation]

Original:  where µ = ¯ p ( ∞ ) and σ 2 pred is predictive variance of conﬁdence.
Revised: (24) where µ = ¯ p ( ∞ ) and σ 2 pred is predictive variance of conﬁdence.

Original: Here, we use the formula V (cid:104) 1 N (cid:80) Ni =1 ξ (cid:105) =
Revised: We use the formula V (cid:104) 1 N (cid:80) Ni =1 ξ (cid:105) =

Original: 1 N V [ ξ ] for arbitrary random variable ξ , and we take the ﬁrst-order Taylor expansion with an assumption ¯ p ( N ) (cid:39) µ in Eq.
Revised: 1 N V [ ξ ] for arbitrary random variable ξ , and we take the ﬁrst-order Taylor expansion with an assump- [Figure] L o g a m p l i t u d e [Figure]

Original: Therefore, the approximated sharpness is: sharpness
Revised: g [Figure] m p l i t u d e [Figure]

Revised: Eigenvlaue

Revised: L o g d e n s i t y

Revised: Max Eigenvlaue

Revised: D e n s i t y

Revised: Eigenvlaue

Revised: D e n s i t y 0 % 10 % 20 % 30 % tion ¯ p ( N ) (cid:39) µ in Eq.

Revised: Therefore, the approximated sharpness is: sharpness

Original: [Equation]
Revised: [Equation]

Original: In conclusion, the variance of NLL, (the square of) the sharpness, is  inversely proportional to theensemble size N .
Revised: In conclusion, the variance of NLL, (the square of) the sharpness, is proportional to the variance ofpredictions σ 2 pred and inversely proportional to the ensemble size N .

Original: As the ensemble size increases in the training phase, the loss landscape becomessmoother.
Revised: As the ensemble size increasesin the training phase, the loss landscape becomes smoother.

Original: Flat loss landscape results in better predictive performance and generalization (Foret et al.
Revised: Flat loss landscape results in betterpredictive performance and generalization (Foret et al., 2020).

Original: Here, we only consider model uncertainty for the sake of simplicity.
Revised: Here, we only consider model uncertainty for the sake of simplicity.

Original: Extending the formulations to data uncertainty is straightforward.
Revised: Extending the formulations to data uncertainty is straightforward.

Original: The predictive distribution of data-complemented BNN inference (Park et al., 2021) is:
Revised: The predictive distribution of data-complemented BNN inference (Park et al., 2021) is:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where S is proximate data distribution, z = ( x , w ) , and p ( z |S , D )
Revised: (27) where S is proximate data distribution, z = ( x , w ) , and p ( z |S , D )

Original: =
Revised: =

Original: p ( x |S ) p ( w |D ) .
Revised: p ( x |S ) p ( w |D ) .

Original: This equationclearly shows that w and x are symmetric.
Revised: This equationclearly shows that w and x are symmetric.

Original: Therefore, we obtain the formulas including both modeland data uncertainty by replacing w with joint random variable of x and w , i.e. w → z = ( w , x ) .Eigenvlaue L o g d e n s i t y Max Eigenvlaue
Revised: Therefore, we obtain the formulas including both modeland data uncertainty by replacing w with joint random variable of x and w , i.e. w → z = ( w , x ) .

Original: D e n s i t y

Original: Eigenvlaue

Original: D e n s i t y 0 % 10 % 20 % 30 %

Original: Experiment.
Revised: Experiment.

Original: Above, we claimtwo statements.
Revised: Above, we claim two statements.

Original: First, the higherthe dropout rate, the sharper theloss landscape.
Revised: First, the higher the dropout rate, the sharper theloss landscape.

Original: Second, the variance of the loss is inversely proportional to the ensemble size.
Revised: Second, the variance of the loss is inversely proportional to the ensemble size.

Original: To demonstrate the former claimquantitatively, we compare the 
Revised: To demonstrate the former claim quantitatively, we compare the Hessian eigenvalue spectra and the

Original: Hessianeigenvalue spectra andthe Hessian max eigenvalue spectra of MC dropout with various dropout rates.
Revised: Hessian max eigenvalue spectra of MC dropout with various dropout rates.

Original: In these experiments, we use ensemble sizeof one for MC dropout.
Revised: In these experiments,we use ensemble size of one for MC dropout.

Original: Fordetailed explanation of Hessianmax eigenvalue spectrum, see Appendix C.1.
Revised: For detailed explanation of Hessian max eigenvaluespectrum, see Appendix C.1.

Original: Fig.
Revised: Fig.

Original: D.2 represents the spectra, which reveals that as the randomness of the model increases, thenumber of Hessian eigenvalue outliers increases .
Revised: D.3 represents the spectra, which reveals that as the randomness of the model increases, thenumber of Hessian eigenvalue outliers increases .

Original: Since outliers are detrimental to the optimizationprocess (Ghorbani et al., 2019), dropout disturb NN optimization.
Revised: Since outliers are detrimental to the optimizationprocess (Ghorbani et al., 2019), dropout disturb NN optimization.

Original: To show the latter claim, we evaluate the variance of NLL loss for ensemble size N train as shown in
Revised: To show the latter claim, we evaluate the variance of NLL loss for ensemble size N train as shown in

Original: Fig.
Revised: Fig.

Original: D.1a.
Revised: D.4a.

Original: As we would expect, the variance of the NLL loss—the sharpness of the loss landscape—isinversely proportional to the ensemble size for large N train .
Revised: As we would expect, the variance of the NLL loss—the sharpness of the loss landscape—isinversely proportional to the ensemble size for large N train .

Original: D. T RAINING P HASE E NSEMBLE L EADS TO B ETTER P ERFORMANCE
Revised: D. T RAINING P HASE E NSEMBLE L EADS TO B ETTER P ERFORMANCE

Original: Appendix D.1 raises an immediate question: Is there a performance difference between ‘trainingwith prediction ensemble’ and ‘training with a low MC dropout rate, instead of no ensemble’? 
Revised: Appendix D.3 raises an immediate question: Is there a performance difference between ‘trainingwith prediction ensemble’ and ‘training with a low MC dropout rate, instead of no ensemble’? Note [Figure]

Original: Notethat both methods reduce the sharpness of the loss landscape.
Revised: that both methods reduce the sharpness of the loss landscape.

Original: This section answers the questionby providing theoretical and experimental explanations that the ensemble in the training phase canimprove predictive performance.
Revised: This section answers the questionby providing theoretical and experimental explanations that the ensemble in the training phase canimprove predictive performance.

Original: According to Gal & Ghahramani (2016), the total predictive variance (in regression tasks) is:
Revised: According to Gal & Ghahramani (2016), the total predictive variance (in regression tasks) is:

Original: [Equation]
Revised: [Equation]

Original: 19) where σ 2 model is model precision and σ 2 sample is NN prediction variance.
Revised: (28) where σ 2 model is model precision and σ 2 sample is NN prediction variance.

Original: Therefore, the model precision is the lower bound of the predictive variance, i.e.:
Revised: Therefore, the model precision is the lower bound of the predictive variance, i.e.:

Original: [Equation]
Revised: [Equation]

Original: The model precision depends only on the model architecture.
Revised: The model precision depends only on the model architecture.

Original: For example, in the case of MC dropout, σ 2 model is proportional to the dropout rate (Gal & Ghahramani, 2016) as follows:
Revised: For example, in the case of MC dropout, σ 2 model is proportional to the dropout rate (Gal & Ghahramani, 2016) as follows:

Original: [Equation]
Revised: [Equation]

Original: These suggest that model precision dominate predictive variance if the MC dropout rate is largeenough, i.e., even if the number of ensembles is increased in the training phase, the predictive varianceis almost the same.
Revised: These suggest that model precision dominate predictive variance if the MC dropout rate is largeenough, i.e., even if the number of ensembles is increased in the training phase, the predictive varianceis almost the same.

Original: In contrast, decreasing the MC dropout rate reduces prediction diversity, and itobviously leads to performance degradation.
Revised: In contrast, decreasing the MC dropout rate reduces prediction diversity, and itobviously leads to performance degradation.

Original: Therefore, in the training phase, it is better to ensemblepredictions than to lower the MC dropout rate .
Revised: Therefore, in the training phase, it is better to ensemblepredictions than to lower the MC dropout rate .

Original: We believe that the training phase ensemble is stronglycorrelated with Batch Augmentation (Hoffer et al., 2020).
Revised: We believe that the training phase ensemble is stronglycorrelated with Batch Augmentation (Hoffer et al., 2020).

Original: We leave concrete analysis for future work.
Revised: We leave concrete analysis for future work.

Original: Experiments.
Revised: Experiment.

Original: The experiments below support the theoretical analysis.
Revised: The experiments below support the theoretical analysis.

Original: We train MC dropout byusing training-phase ensemble method with various ensemble sizes N train .
Revised: We train MC dropout byusing training-phase ensemble method with various ensemble sizes N train .

Original: As we would expect, Fig.
Revised: As we would expect, Fig.

Original: D.1b shows that training phase ensemble signiﬁcantly improves thepredictive performance .
Revised: D.4b shows that training phase ensemble signiﬁcantly improves thepredictive performance .

Original: In this experiment, we use MC dropout rate of 30%.
Revised: In this experiment, we use MC dropout rate of 30%.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: A.1, itprovides the best predictive performance.
Revised: A.1, itprovides the best predictive performance.

Original: We use ensemble size N test = 50 in test phase.
Revised: We use ensemble size N test = 50 in test phase.

Original: We also measure the predictive variances of NLL.
Revised: We also measure the predictive variances of NLL.

Original: The predictive variances of the model with
Revised: The predictive variances of the model with

Original: N train = 1 and with N train = 3 are V
Revised: N train = 1 and with N train = 3 are V

Original: [ L ] = 0 .
Revised: [ L ] = 0 .

Original: 0169 and V [ L ] = 0 . 0179 , respectively.
Revised: 0169 and V [ L ] = 0 . 0179 , respectively.

Original: Since thepredictive variances of the two models are almost the same, we infer that there exists a lower bound.
Revised: Since thepredictive variances of the two models are almost the same, we infer that there exists a lower bound.

Original: D. F OURIER A NALYSIS OF F EATURE M APS

Original: In Section 2.2, at the end of the stage 1, we show that MC dropout adds high-frequency noise tofeature maps, and spatial smoothing effectively removes it.

Original: As shown in Fig.

Original: D.3, we observe thesame phenomena at other stages.

Original: In addition, Fig.

Original: 6c shows that CNNs are vulnerable to high-frequency random noise.

Original: Interestingly, italso shows that CNNs are robust against noise with frequencies from 0 .

Original: 6 π to 0 .

Original: 8 π , corresponding toapproximately 3 pixel periods.

Original: Since the receptive ﬁelds of convolutions are 3 × 3, the noise with aperiod smaller than the size is averaged out by convolutions.

Original: For the same reason, convolutions areparticularly vulnerable against the noise with a frequency of 0 .

Original: 3 π , corresponding to a period of 6pixel.

Original: E E XTENDED I NFORMATIONS OF E XPERIMENTS
Revised: E E XTENDED I NFORMATIONS OF E XPERIMENTS

Original: This section provides additional information on the experiments in Section 3.
Revised: This section provides additional information on the experiments in Section 3.

Original: E. I MAGE C LASSIFICATION
Revised: E. I MAGE C LASSIFICATION

Original: We present numerical comparisons in the image classiﬁcation experiment and discuss the results indetail. 
Revised: We present numerical comparisons in the image classiﬁcation experiment and discuss the results indetail. [Table] [Table]

Original: Computational performance.
Revised: Ensemble Size [Table] [Table] [Table]

Revised: N L L

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: A cc u r a c y ( % )

Revised: [Table]

Revised: [Table]

Revised: Size

Revised: [Table]

Revised: [Table]

Revised: E C E ( % ) (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: N L L

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: A cc u r a c y ( % )

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: E C E ( % ) (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: N LL

Revised: [Table]

Revised: [Table]

Revised: Size

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: A cc u r a c y ( % )

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: E C E ( % ) (

Revised: [Table]

Revised: ResNet

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: N LL

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: A cc u r a c y ( % )

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size 2.

Revised: E C E ( % ) (d

Revised: [Table]

Revised: on

Revised: [Table]

Revised: Ensemble Size

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: N LL

Revised: [Table]

Revised: [Table]

Revised: Size 76.

Revised: A cc u r a c y ( % )

Revised: [Table]

Revised: [Table]

Revised: Ensemble Size 1.

Revised: [Table]

Revised: E C E ( % ) (e) ResNet-50 on ImageNet 0.

Original:  The throughput of MC dropout and “MC dropout + spatial smoothing” is 755 and 675 image/sec, respectively, in training phase on ImageNet.
Revised: [Figure] [Figure] Computational performance. The throughput of MC dropout and “MC dropout + spatial smoothing” is 755 and 675 image/sec, respectively, in training phase on ImageNet.

Original: As mentioned in lines
Revised: As mentioned in lines

Original: Section 3.1, NLL of “MC dropout + spatial smoothing” with ensemble size of 2 is comparableto or even better than that of MC dropout with ensemble size of 50.
Revised: Section 3.1, NLL of “MC dropout + spatial smoothing” with ensemble size of 2 is comparableto or even better than that of MC dropout with ensemble size of 50.

Original: Therefore, “MC dropout +spatial smoothing” is 22 × faster than MC dropout with similar predictive performance, in terms ofthroughput.
Revised: Therefore, “MC dropout +spatial smoothing” is 22 × faster than MC dropout with similar predictive performance, in terms ofthroughput.

Original: [Figure]
Revised: [Figure]

Original: Predictive performance on test dataset.
Revised: Predictive performance on test dataset.

Original: Fig.
Revised: Fig.

Original: E.2 represents the reliability diagram of ResNet-18 on CIFAR-100,which shows that spatial smoothing improves the uncertaintyof both deterministic and Bayesian NNs.
Revised: E.2 represents the reliability diagram of ResNet-18 on CIFAR-100,which shows that spatial smoothing improves the uncertaintyof both deterministic and Bayesian NNs.

Original: Numerical comparisons are provided below.
Revised: Numerical comparisons are provided below.

Original: Table E.1 shows the predictive performance of various deterministic and Bayesian NNs with and without spatial smoothing on CIFAR-10, CIFAR-100, and ImageNet.
Revised: Table E.1 shows the predictive performance of various deterministic and Bayesian NNs with and without spatial smoothing on CIFAR-10, CIFAR-100, and ImageNet.

Original: This tablesuggests the following: First, spatial smoothing improvesboth accuracy and uncertainty in most cases.
Revised: This tablesuggests the following: First, spatial smoothing improvesboth accuracy and uncertainty in most cases.

Original: In particular, itimproves the predictive performance of all models with MCdropouts .
Revised: In particular, itimproves the predictive performance of all models with MCdropouts .

Original: Second, spatial smoothing signiﬁcantly improvesthe predictive performance of VGG compared with ResNet.
Revised: Second, spatial smoothing signiﬁcantly improvesthe predictive performance of VGG compared with ResNet.

Original: VGG has a chaotic loss landscape, which results in poor predictive performance (Li et al., 2017), and spatial smoothingsmoothens its loss landscape effectively.
Revised: VGG has a chaotic loss landscape, which results in poor predictive performance (Li et al., 2018), and spatial smoothingsmoothens its loss landscape effectively.

Original: Third, as the depthincreases, the performance improvement decreases.
Revised: Third, as the depthincreases, the performance improvement decreases.

Original: Deeper
Revised: Deeper

Original: NNs provide more overconﬁdent results (Guo et al.,
Revised: NNs provide more overconﬁdent results (Guo et al.,

Original: 2017),but the number of spatial smoothing layers calibrating uncertainty is ﬁxed.
Revised: 2017),but the number of spatial smoothing layers calibrating uncertainty is ﬁxed.

Original: Last, the performance improvement of ResNeXt, which includes an ensemble in itsinternal structure, is relatively marginal.
Revised: Last, the performance improvement of ResNeXt, which includes an ensemble in itsinternal structure, is relatively marginal.

Original: Fig.
Revised: Fig.

Original: E.1 shows predictive performance of MC dropout and deep ensemble for ensemble size.
Revised: E.1 shows predictive performance of MC dropout and deep ensemble for ensemble size.

Original: Adeep ensemble with an ensemble size of 1 is a deterministic NN.
Revised: Adeep ensemble with an ensemble size of 1 is a deterministic NN.

Original: This ﬁgure shows that spatialsmoothing improves efﬁciency of ensemble size and the predictive performance at ensemble size of50.
Revised: This ﬁgure shows that spatialsmoothing improves efﬁciency of ensemble size and the predictive performance at ensemble size of [Table]

Original: In addition, spatial smoothing stabilizes NN training.
Revised: In addition, spatial smoothing stabilizes NN training.

Original: It reduces the variance of the performance,especially in VGG.
Revised: It reduces the variance of the performance,especially in VGG.

Original: A peculiarity of the results on ImageNet is that spatial smoothing degrades ECE of ResNet-50.
Revised: A peculiarity of the results on ImageNet is that spatial smoothing degrades ECE of ResNet-50.

Original: Itis because spatial smoothing signiﬁcantly improves the accuracy in this case, and there tends to be [Figure] L o g a m p l i t u d e[Figure]
Revised: Itis because spatial smoothing signiﬁcantly improves the accuracy in this case, and there tends to bea trade-off between accuracy and ECE, e

Original: g

Original: [Figure]

Original: p l i t u d e

Original: [Figure]

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: N L L

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: A cc u r a c y ( % )

Original: [Table]

Original: [Table]

Original: Size

Original: [Table]

Original: [Table]

Original: E C E ( % ) (

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: N L L

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]
Revised: [Table]

Original: A cc u r a c y ( % )

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: E C E ( % ) (

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: N LL

Original: [Table]

Original: [Table]

Original: Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: A cc u r a c y ( % )

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: E C E ( % ) (

Original: [Table]

Original: ResNet

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: N LL

Original: [Table]

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: A cc u r a c y ( % )

Original: [Table]

Original: [Table]

Original: Ensemble Size 2. E C E ( % ) (d [Table] on
Revised: g. as shown in (Guo et al., 2017), Fig.

Original: [Table]

Original: Ensemble Size

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: N LL

Original: [Table]

Original: [Table]

Original: Size 76.

Original: A cc u r a c y ( % ) [Table] [Table] Ensemble Size 1.
Revised: A.

Original: [Table]
Revised: [Table]

Original: E C E ( % ) (e) ResNet-50 on ImageNet 0.

Original: [Figure] [Figure] a trade-off between accuracy and ECE, e.g. as shown in (Guo et al., 2017), Fig. A.1, and Fig.
Revised: and Fig.

Original: B.3.
Revised: B.3.

Original: Instead, spatial smoothing shows the improvement in NLL, another uncertainty metric.
Revised: Instead, spatial smoothing shows the improvement in NLL, another uncertainty metric.

Original: Predictive performance on training datasets.
Revised: Predictive performance on training datasets.

Original: Note that spatial smoothing helps NN learn strongrepresentations .
Revised: Note that spatial smoothing helps NN learn strongrepresentations .

Original: In other words, spatial smoothing does not regularize NNs .
Revised: In other words, spatial smoothing does not regularize NNs .

Original: For example, NLL
Revised: For example, NLL

Original: ResNet-18 with MC dropout on CIFAR-100 training dataset is 2 .
Revised: ResNet-18 with MC dropout on CIFAR-100 training dataset is 2 .

Original: 20 × 10 − 2 .
Revised: 20 × 10 − 2 .

Original: The NLL of the ResNetwith spatial smoothing is 1 .
Revised: The NLL of the ResNetwith spatial smoothing is 1 .

Original: 94 × 10 − 2 .
Revised: 94 × 10 − 2 .

Original: In conclusion, spatial smoothing reduces the training loss.
Revised: In conclusion, spatial smoothing reduces the training loss.

Original: Corruption robustness.
Revised: Corruption robustness.

Original: We measure predictive performance on CIFAR-100-C (Hendrycks &
Revised: We measure predictive performance on CIFAR-100-C (Hendrycks &

Original: Dietterich, 2019) in order to evaluate the robustness of the models against 5 intensities and 15 typesof data corruption.
Revised: Dietterich, 2018) in order to evaluate the robustness of the models against 5 intensities and 15 typesof data corruption.

Original: The top row of Fig.
Revised: The top row of Fig.

Original: E.3 shows the results as a box plot.
Revised: E.3 shows the results as a box plot.

Original: The box plot shows themedian, interquartile range (IQR), minimum, and maximum of predictive performance for types.
Revised: The box plot shows themedian, interquartile range (IQR), minimum, and maximum of predictive performance for types.

Original: They reveal that spatial smoothing improves predictive performance for corrupted data.
Revised: They reveal that spatial smoothing improves predictive performance for corrupted data.

Original: In particular,spatial smoothing undoubtedly helps in predicting reliable uncertainty.
Revised: In particular,spatial smoothing undoubtedly helps in predicting reliable uncertainty.

Original: To summarize the performance of corrupted data in a single value, Hendrycks & Dietterich (2019) introduced a corruption error (CE) for quantitative comparison.
Revised: To summarize the performance of corrupted data in a single value, Hendrycks & Dietterich (2018) introduced a corruption error (CE) for quantitative comparison.

Original: CE f c , which is CE for corruption type c and model f , is as follows:
Revised: CE fc , which is CE for corruption type c and model f , is as follows:

Original: [Equation]
Revised: [Equation]

Original: where E fi,c is top-1 error of f for corruption type c and intensity i , and E AlexNet i,c is the error of AlexNet.
Revised: where E fi,c is top-1 error of f for corruption type c and intensity i , and E AlexNet i,c is the error of AlexNet.

Original: Mean CE or mCE summarizes CE fc by averaging them over 15 corruption types such as Gaussian noise, brightness, and show.
Revised: Mean CE or mCE summarizes CE fc by averaging them over 15 corruption types such as Gaussian noise, brightness, and show.

Original: Likewise, to evaluate robustness in terms of uncertainty, we introduce corruption NLL ( CNLL , ↓ ) and corruption ECE ( CECE , ↓ ) as follows:
Revised: Likewise, to evaluate robustness in terms of uncertainty, we introduce corruption NLL ( CNLL , ↓ ) and corruption ECE ( CECE , ↓ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: where NLL fi,c and ECE fi,c are NLL and ECE of f for c and i , respectively.
Revised: where NLL fi,c and ECE fi,c are NLL and ECE of f for c and i , respectively.

Original: mCNLL and mCECEare averages over corruption types.
Revised: mCNLL and mCECEare averages over corruption types.

Original: Experimental results show that spatial smoothing improves therobustness against data corruption.
Revised: Experimental results show that spatial smoothing improves therobustness against data corruption.

Original: See Fig.
Revised: See Fig.

Original: E.3 for the results.
Revised: E.3 for the results.

Original: The bottom row of Fig.
Revised: The bottom row of Fig.

Original: E.3 shows mCNLL, mCE, and mCECE for ensemble size.
Revised: E.3 shows mCNLL, mCE, and mCECE for ensemble size.

Original: They indicates thatspatial smoothing improves not only the efﬁciency but corruption robustness across a whole range ofensemble size.
Revised: They indicates thatspatial smoothing improves not only the efﬁciency but corruption robustness across a whole range ofensemble size.

Original: Adversarial robustness.
Revised: Adversarial robustness.

Original: We show that spatial smoothing also improves adversarial robustness.
Revised: We show that spatial smoothing also improves adversarial robustness.

Original: First, we measure the robustness, in terms of accuracy and attack success rate (ASR), of ResNet50 on ImageNet against popular adversarial attacks, namely FGSM (Goodfellow et al., 2014) and
Revised: First, we measure the robustness, in terms of accuracy and attack success rate (ASR), of ResNet50 on ImageNet against popular adversarial attacks, namely FGSM (Goodfellow et al., 2015) and

Original: PGD (Madry et al., 2017).
Revised: PGD (Madry et al., 2018).

Original: Table E.2 indicate that both MC dropout and spatial smoothing improverobustness against adversarial attacks.
Revised: Table E.2 indicate that both MC dropout and spatial smoothing improverobustness against adversarial attacks.

Original: Next, we ﬁnd out how spatial smoothing improves adversarial robustness.
Revised: Next, we ﬁnd out how spatial smoothing improves adversarial robustness.

Original: To this end, similarto Section 2.2, we measure the accuracy on the test datasets with frequency-based adversarialperturbations.
Revised: To this end, similarto Section 2.2, we measure the accuracy on the test datasets with frequency-based adversarialperturbations.

Original: In this experiment, we use FGSM attack.
Revised: In this experiment, we use FGSM attack.

Original: This experimental result shows that spatialsmoothing is particularly robust against high frequency ( ≥ 0 .
Revised: This experimental result shows that spatialsmoothing is particularly robust against high frequency ( ≥ 0 .

Original: 3 π ) adversarial attacks.
Revised: 3 π ) adversarial attacks.

Original: This is becausespatial smoothing is a low-pass ﬁlter, as we mentioned in Section 2.2.
Revised: This is becausespatial smoothing is a low-pass ﬁlter, as we mentioned in Section 2.2.

Revised: Since the ResNet is vulnerable

Revised: [Figure]

Revised: We deﬁne relative conﬁdence (See Eq.

Original: Since the ResNet is vulnerableagainst high frequency adversarial attack, an effective defense of spatial smoothing against highfrequency attacks signiﬁcantly improves the robustness.
Revised: (36)), and measure the metric of ResNet-18 on CIFAR- P against high frequency adversarial attack, an effective defense of spatial smoothing against highfrequency attacks signiﬁcantly improves the robust-

Original: Consistency.
Revised: Consistency.

Original: To evaluate the translation invariance of models, we use consistency (Hendrycks &
Revised: To evaluate the translation invariance of models, we use consistency (Hendrycks &

Original: Dietterich, 2019; Zhang, 2019a), a metric representing translation consistency for shift-translated data sequences S = { x 1 , · · · , x M +1 } , as follows:
Revised: Dietterich, 2018; Zhang, 2019a), a metric representing translation consistency for shift-translated data sequences S = { x 1 , · · · , x M +1 } , as follows:

Original: [Equation]
Revised: [Equation]

Original:  where g ( x ) = arg max p ( y | x , D ) .
Revised: 34) where g ( x ) = arg max p ( y | x , D ) .

Original: Table E.3 provides consistency of ResNet-18 on CIFAR-10-P(Hendrycks & Dietterich, 2019).
Revised: Table E.3 provides consistency of ResNet-18 on CIFAR-10-P(Hendrycks & Dietterich, 2018).

Original: The results shows that MC dropout and deep ensemble improveconsistency, and spatial smoothing improves consistency of both deterministic and Bayesian NNs.
Revised: The results showsthat MC dropout and deep ensemble improve consistency, and spatial smoothing improves consistency of both deterministic and Bayesian NNs.

Revised: Prior works (Zhang, 2019a; Azulay & Weiss, 2019) investigated the ﬂuctuation of predictive conﬁdence

Original: [Figure]
Revised: [Table]

Original: [-We deﬁne relative conﬁdence (See Eq. (27)), and measure the metric of ResNet-18 on CIFAR
 -P. Prior works (Zhang, 2019a; Azulay & Weiss, 2019) investigated the ﬂuctuation of predictive conﬁdence-] on shift-translated data sequence.
Revised: [-We deﬁne relative conﬁdence (See Eq. (27)), and measure the metric of ResNet-18 on CIFAR
 -P. Prior works (Zhang, 2019a; Azulay & Weiss, 2019) investigated the ﬂuctuation of predictive conﬁdence-] on shift-translated data sequence.

Original: However, surprisingly, we ﬁnd that conﬁdence ﬂuctuation has little to do with consistency .
Revised: However, surprisingly, we ﬁnd that conﬁdence ﬂuctuation has little to do with consistency .

Original: To demonstrate this claim, we introduce cross-entropy consistency (CEC, ↓ ), a metric that represents the ﬂuctuation of conﬁdence on a shift-translated data sequence
Revised: To demonstrate this claim, we introduce cross-entropy consistency (CEC, ↓ ), a metric that represents the ﬂuctuation of conﬁdence on a shift-translated data sequence

Original: S = { x 1 , · · · , x M +1 } , as follows:
Revised: S = { x 1 , · · · , x M +1 } , as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where f ( x ) =
Revised: where f ( x ) =

Original: p ( y | x , D ) .
Revised: p ( y | x , D ) .

Original: In Table E.3, high consistency does not mean low CEC; conversely, highconsistency tends to be high CEC.
Revised: In Table E.3, high consistency does not mean low CEC; conversely, highconsistency tends to be high CEC.

Original: Canonical NNspredict overconﬁdent probabilities, and their conﬁdence sometimes changes drastically from nearzero to near-one.
Revised: Canonical NNs predict overconﬁdent probabilities, and theirconﬁdence sometimes changes drastically from near-zero to near-one.

Original: Correspondingly, it results in lowconsistency but low CEC.
Revised: Correspondingly, it results inlow consistency but low CEC.

Original: On the contrary, wellcalibrated NNs such as MC dropout provide conﬁdence that oscillates between zero and one, whichresults in high CEC.
Revised: On the contrary, well-calibrated NNs such as MC dropout provideconﬁdence that oscillates between zero and one, which results in high CEC.

Original: To represent the NN reliability properly, we propose relative conﬁdence ( ↑ ) as follows:
Revised: To represent the NN reliability properly, we propose relative conﬁdence ( ↑ ) as follows:

Original: [Equation]
Revised: [Equation]

Original:  where max p ( y | x , D ) is conﬁdence of predictive result and p ( y true | x , D ) is probability of the resultfor true label.
Revised: 36) where max p ( y | x , D ) is conﬁdence of predictive result and p ( y true | x , D ) is probability of the resultfor true label.

Original: It is 1 when NN classiﬁes the image correctly, and less than 1 when NN classiﬁes itincorrectly.
Revised: It is 1 when NN classiﬁes the image correctly, and less than 1 when NN classiﬁes itincorrectly.

Original: Therefore, relative conﬁdence is a metric that indicates the overconﬁdence of a predictionwhen NN’s prediction is incorrect.
Revised: Therefore, relative conﬁdence is a metric that indicates the overconﬁdence of a predictionwhen NN’s prediction is incorrect.

Original: Figure E.4 shows a qualitative example of consistency on CIFAR-10-P by using relative conﬁdence.
Revised: Figure E.4 shows a qualitative example of consistency on CIFAR-10-P by using relative conﬁdence.

Original: This ﬁgure suggests that spatial smoothing improves consistency of both deterministic and Bayesian 
Revised: This ﬁgure suggests that spatial smoothing improves consistency of both deterministic and Bayesian [Table]

Original: NN.
Revised: NN.

Revised: [Table]

Original: E. S EMANTIC S EGMENTATION
Revised: E. S EMANTIC S EGMENTATION

Original: Table E.4 shows the performance of U-Net on the CamVid dataset.
Revised: Table E.4 shows the performance of U-Net on the CamVid dataset.

Original: This table indicates that spatialsmoothing improves accuracy, uncertainty, and consistency of deterministic and Bayesian NNs.
Revised: This table indicates that spatialsmoothing improves accuracy, uncertainty, and consistency of deterministic and Bayesian NNs.

Original: This is consistent with the results in image classiﬁcation.
Revised: This is consistent with the results in image classiﬁcation.

Original: In addition, temporal smoothing leadsto signiﬁcant improvement in efﬁciency of ensemble size, accuracy, uncertainty, and consistencyby exploiting temporal information.
Revised: In addition, temporal smoothing leadsto signiﬁcant improvement in efﬁciency of ensemble size, accuracy, uncertainty, and consistencyby exploiting temporal information.

Original: Moreover, temporal smoothing requires only one ensemble toachieve high predictive performance, since it cooperates with the temporally previous predictions.
Revised: Moreover, temporal smoothing requires only one ensemble toachieve high predictive performance, since it cooperates with the temporally previous predictions.

Original: Weobtain the best predictive and computational performance by using both temporal smoothing andspatial smoothing.
Revised: Weobtain the best predictive and computational performance by using both temporal smoothing andspatial smoothing.

Original: F C OMPARISON W ITH A NTI - ALIASED CNN
Revised: F C OMPARISON W ITH A NTI - ALIASED CNN

Original: As we mentioned in Section 4, local means ( Blur ), also known as anti-aliased CNN (Zhang, 2019a),improve accuracy.
Revised: As we mentioned in Section 4, local means ( Blur ), also known as anti-aliased CNN (Zhang, 2019a),improve accuracy.

Original: Nevertheless, our work ( Prob + Blur ) has novelties in three respects: differentmotivation, improved uncertainty estimation, and analysis of how spatial smoothing works.[Table]
Revised: Nevertheless, our work ( Prob + Blur ) has novelties in three respects: differentmotivation, improved uncertainty estimation, and analysis of how spatial smoothing works.

Original: Different motivation.
Revised: Different motivation.

Original: The motivation of local means was to mitigate the aliasing effect of subsampling and to improve shift invariance.
Revised: The motivation of local means was to mitigate the aliasing effect of subsampling and to improve shift invariance.

Original: In contrast, our spatial smoothing is introduced to aggregateand ensemble nearby feature map points.
Revised: In contrast, our spatial smoothing is introduced to aggregateand ensemble nearby feature map points.

Original: Improved uncertainty estimation.
Revised: Improved uncertainty estimation.

Original: We demonstrate that spatial smoothing improves not onlyaccuracy, but also uncertainty estimation and robustness against natural corruptions and adversarialattacks all at the same time.
Revised: We demonstrate that spatial smoothing improves not onlyaccuracy, but also uncertainty estimation and robustness against natural corruptions and adversarialattacks all at the same time.

Original: Moreover, we show that spatial smoothing signiﬁcantly enhances theperformance of MC dropout.
Revised: Moreover, we show that spatial smoothing signiﬁcantly enhances theperformance of MC dropout.

Original: Since there typically tends to be a trade-off between accuracy and“uncertainty + robustness”—e.g.
Revised: Since there typically tends to be a trade-off between accuracy and“uncertainty + robustness”—e.g.

Original: as shown in (Guo et al., 2017; Zhang et al., 2019;
Revised: as shown in (Guo et al., 2017; Zhang et al., 2019;

Original: Geirhos et al.
Revised: Geirhos et al.

Original: ,2018; Zhang, 2019b), Fig.
Revised: ,2019; Zhang, 2019b), Fig.

Original: A.1, and Fig.
Revised: A.1, and Fig.

Original: B.3—in NN modeling, we believe our simple yet effectivemethod makes major inroads into the uncertainty quantiﬁcation and generalization.
Revised: B.3—in NN modeling, we believe our simple yet effectivemethod makes major inroads into the uncertainty quantiﬁcation and generalization.

Original: Analysis of how spatial smoothing improves performance.
Revised: Analysis of how spatial smoothing improves performance.

Original: We ﬁnd that the predictive performance improvement is not due to the anti-aliasing effect of 
Revised: We ﬁnd that the predictive performance improvement is not due to the anti-aliasing effect of local means.

Original: local means • Prob + Blur —our probabilistic spatial smoothing—improves the performance of preactivation CNNs, but Blur alone—local mean or anti-aliased CNN—does not.
Revised: • Prob + Blur —our probabilistic spatial smoothing—improves the performance of preactivation CNNs, but Blur alone—local mean or anti-aliased CNN—does not.

Original: In fact,contrary to (Zhang, 2019a), local mean degrades the predictive performance since it resultsin loss of information.
Revised: In fact,contrary to (Zhang, 2019a), local mean degrades the predictive performance since it results in loss of information.

Original: It suggests that Prob plays an key role in prediction.
Revised: It suggests that Prob plays an key role in prediction.

Original: For more details,
Revised: For more details,

Original: see Appendix F.1.
Revised: see Appendix F

Original: • Although the local ﬁltering can result in loss of information, Zhang (2019a) experimentallyobserved an increase in both shift-invariance (as expected) and accuracy (which was beyond expectation).
Revised: • Although the local ﬁltering can result in loss of information, Zhang (2019a) experimentallyobserved an increase in both shift-invariance (as expected) and accuracy (which was be yond expectation).

Original: However, “there exist a fundamental trade-off between ‘shift-invarianceplus anti-aliasing’ and performance” (Zhang, 2019b).
Revised: However, “there exist a fundamental trade-off between ‘shift-invarianceplus anti-aliasing’ and performance” (Zhang, 2019b).

Original: Moreover, it is difﬁcult to relateanti-aliasing to improved uncertainty and robustness.
Revised: Moreover, it is difﬁcult to relateanti-aliasing to improved uncertainty and robustness.

Original: Zhang (2019a) did not provide anexplanation for these phenomena.
Revised: Zhang (2019a) did not provide anexplanation for these phenomena.

Original: As discussed in Appendix E.1, spatial smoothing helps
Revised: As discussed in Appendix E.1, spatial smoothing helps

Original: NNs learn strong representations, not regularizes NNs.
Revised: NNs learn strong representations, not regularizes NNs

Original: • Spatial smoothing is, surprisingly, robust against blur corruptions.
Revised: • Spatial smoothing is, surprisingly, robust against blur corruptions.

Original: We analyze how spatial smoothing improves predictive performance, by using loss landscape visualization, Hessian eigenvalue spectra, and Fourier analysis.
Revised: We analyze how spatial smoothing improves predictive performance, by using loss landscape visualization, Hessian eigenvalue spectra, and Fourier analysis.

Original: These analyzes draw the followingconclusions:
Revised: These analyzes draw the followingconclusions:

Original: • Loss landscape visualization : Spatial smoothing stabilizes loss landscape ﬂuc-
Revised: • Loss landscape visualization : Spatial smoothing stabilizes loss landscape ﬂuc-

Original: tuations, caused by e
Revised: tuations, caused by e

Original: g. MC dropout.
Revised: g. MC dropout.

Original: This results in stabilizing NN training
Revised: This results in stabilizing NN training

Original: and improving performance as well as generalization.
Revised: and improving performance as well as generalization.

Original: See Figs.
Revised: See Figs.

Original: 7 and C.2. See also code/resources/losslandscapes/resnet_mcdo_18.gif and code/resources/losslandscapes/resnet_mcdo_smoothing_18.gif in the supplementary material.
Revised: 7 and C.2.See also code/resources/losslandscapes/resnet_mcdo_18.gif andcode/resources/losslandscapes/resnet_mcdo_smoothing_18.gif in thesupplementary material.•

Original: • Hessian eigenvalue spectra : Spatial smoothing suppresses outliers of Hessian eigenvalues,which disrupt NN training.
Revised: Hessian eigenvalue spectra : Spatial smoothing suppresses outliers of Hessian eigenvalues,which disrupt NN training.

Original: See Figs. 
Revised: See Figs. 8 and C.3.

Original: 8 and C.3.• Fourier analysis : Spatial smoothing effectively removes high frequency signals, includingnoise due to MC dropout.
Revised: • Fourier analysis : Spatial smoothing effectively removes high frequency signals, includingnoise due to MC dropout.

Original: We also show that CNNs are vulnerable to high frequency noiseand high frequency adversarial attacks.
Revised: We also show that CNNs are vulnerable to high frequency noise and high frequency adversarial attacks.

Original: See Figs.
Revised: See Figs.

Original: 6 and D.3.
Revised: 6 and D.2.

Original: We also provide theoretical analysis of how spatial smoothing works.
Revised: We also provide theoretical analysis of how spatial smoothing works.

Original: We prove that dropout sharpensthe loss landscape, and ensemble smoothens it .
Revised: We prove that dropout sharpens the loss landscape, and ensemble smoothens it .

Original: Since the spatial smoothing is a spatial ensemble, itsigniﬁcantly enhances the performance of MC dropout.
Revised: Since the spatial smoothing is a spatial ensemble, it signiﬁcantly enhances the performance of MC dropout.

Original: See Appendix D.1 for more details.
Revised: See Appendix D.3 for more details.

Original: Furthermore, we also show that training-phase ensemble signiﬁcantly improves the predictive performancebecause it smoothens the loss landscape without loss of prediction diversity .
Revised: Further- more, we also show that training-phase ensemble signiﬁcantly improves the predictive performancebecause it smoothens the loss landscape without loss of prediction diversity .

Original: Therefore, the spatialsmoothing, which ensembles feature map points at training time, improves the performance effectively.
Revised: Therefore, the spatialsmoothing, which ensembles feature map points at training time, improves the performance effectively.

Original: See Appendix D.
Revised: See Appendix D.4.

Original: F. Prob P LAYS AN I MPORTANT R OLE IN SPATIAL SMOOTHING
Revised: F. Prob P LAYS AN I MPORTANT R OLE IN S PATIAL S MOOTHING

Original: As discussed in Section 2.1, we take the perspective that each point in feature map is a prediction forbinary classiﬁcation by deriving the Bernoulli distributions from the feature map by using Prob .
Revised: As discussed in Section 2.1, we take the perspective that each point in feature map is a prediction forbinary classiﬁcation by deriving the Bernoulli distributions from the feature map by using Prob .

Original: It isin contrast to previous works known as sampling-free BNNs (Hernández-Lobato & Adams, 2015;
Revised: It isin contrast to previous works known as sampling-free BNNs (Hernández-Lobato & Adams, 2015;

Original: Wang et al., 2016; Wu et al., 2018) attempting to approximate the distribution of feature map withone Gaussian distribution.
Revised: Wang et al., 2016; Wu et al., 2019) attempting to approximate the distribution of feature map withone Gaussian distribution.

Original: We do not use any assumptions on the distribution of feature map, andexactly represent the Bernoulli distributions and their averages.
Revised: We do not use any assumptions on the distribution of feature map, andexactly represent the Bernoulli distributions and their averages.

Original: However, sampling-free BNNs areerror-prone because there is no guarantee that feature maps will follow a Gaussian distribution.
Revised: However, sampling-free BNNs areerror-prone because there is no guarantee that feature maps will follow a Gaussian distribution.

Original: This Prob plays an important role in spatial smoothing.
Revised: This Prob plays an important role in spatial smoothing.

Original: CNNs such as VGG, ResNet, and ResNeXt generally use post-activation arrangement.
Revised: CNNs such as VGG, ResNet, and ResNeXt generally use post-activation arrangement.

Original: In other words, their stages end with BatchNorm and
Revised: In other words, their stages end with BatchNorm and

Original: ReLU .
Revised: ReLU .

Original: Therefore, spatial smoothing layers Smooth ( z ) =
Revised: Therefore, spatial smoothing layers Smooth ( z ) =

Original: Blur ◦ Prob ( z ) in CNNs cooperates with
Revised: Blur ◦ Prob ( z ) in CNNs cooperates with

Original: BatchNorm and ReLU as follows:
Revised: BatchNorm and ReLU as follows:

Original: [Equation]
Revised: [Equation]

Original:  since ReLU and tanh τ are commutative, and ReLU ◦ ReLU is ReLU .
Revised: 38) since ReLU and tanh τ are commutative, and ReLU ◦ ReLU is ReLU .

Original: This Prob is trainable and is ageneral form of Eq.
Revised: This Prob is trainable and is ageneral form of Eq.

Original: If we only use Blur as spatial smoothing, the activations BatchNorm – ReLUplay the role of Prob .
Revised: If we only use Blur as spatial smoothing, the activations BatchNorm – ReLUplay the role of Prob .

Original: [Figure]
Revised: [Figure]

Original: In order to analyze the roles of Prob and Blurmore precisely, we measure the predictive performance of the model that does not use the postactivation.
Revised: In order to analyze the roles of Prob and Blurmore precisely, we measure the predictive performance of the model that does not use the postactivation.

Original: Figure F.1 shows NLL of pre-activation
Revised: Figure F.1 shows NLL of pre-activation

Original: VGG-16 on CIFAR-100.
Revised: VGG-16 on CIFAR-100.

Original: The result shows that
Revised: The result shows that

Original: Blur with Prob improves the performance, but
Revised: Blur with Prob improves the performance, but

Original: Blur alone does not.
Revised: Blur alone does not.

Original: In fact, contrary to (Zhang,2019a), blur degrades the predictive performancesince it results in loss of information .
Revised: In fact, contrary to (Zhang,2019a), blur degrades the predictive performancesince it results in loss of information .

Original: We alsomeasure the performance of VGG-19, ResNet-18, ResNet-50, and BlurPool (Zhang, 2019a) with preactivation, and observe the same phenomenon. Inaddition, BatchNorm – ReLU in front of GAP significantly improves the performance of pre-activation ResNet.
Revised: also

Original: As mentioned in Appendix C.2, pre-activation isa special case of spatial smoothing.
Revised: As mentioned in Appendix C.2, pre-activation isa special case of spatial smoothing.

Original: Therefore, theperformance improvement of pre-activation by spatial smoothing is marginal compared to that of post-activation.
Revised: Therefore, theperformance improvement of pre-activation by spatial smoothing is marginal compared to that of postactivation.