Original: Deep Combinatorial Aggregation
Revised: Deep Combinatorial Aggregation

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.
Revised: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.

Original: This includes deepensemble, a simple and effective method that achieves state-of-the-art results foruncertainty-aware learning tasks.
Revised: This includes deep ensemble, a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks.

Original: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).
Revised: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).

Original: DCAcreates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.
Revised: DCA creates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.

Original: DCA components canbe deﬁned at different levels of granularity.
Revised: DCA components can be deﬁned at different levels of granularity.

Original: And we discovered that coarse-grain
Revised: And we discovered that coarse-grain

Original: DCAs can outperform deep ensemble for uncertainty-aware learning both in termsof predictive performance and uncertainty estimation.
Revised: DCAs can outperform deep ensemble for uncertainty-aware learning both in terms of predictive performance and uncertainty estimation.

Original: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weightaveraging (DCWA) can improve the baseline training.
Revised: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weight averaging (DCWA) can improve the baseline training.

Original: It is on par with stochasticweight averaging (SWA) but does not require any custom training schedule oradaptation of BatchNorm layers.
Revised: It is on par with stochastic weight averaging (SWA) but does not require any custom training schedule or adaptation of BatchNorm layers.

Original: Furthermore, we propose a consistency enforcingloss that helps the training of DCWA and modelwise DCA.
Revised: Furthermore, we propose a consistency enforcing loss that helps the training of DCWA and modelwise DCA.

Original: We experiment onin-domain, distributional shift, and out-of-distribution image classiﬁcation tasks,and empirically conﬁrm the effectiveness of DCWA and DCA approaches.
Revised: We experiment on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, and empirically conﬁrm the effectiveness of DCWA and DCA approaches.

Original:  Introduction
Revised:  Introduction

Original: Deep learning has achieved groundbreaking progress and neural networks are now widely used invarious domains [24].
Revised: Deep learning has achieved groundbreaking progress and neural networks are now widely used in various domains [27].

Original: However, they are known to produce poor uncertainty estimations [9, 36, 10] which can be problematic for challenges like safety-critical applications [17, 26] or active learning[34].
Revised: However, they are known to produce poor uncertainty estimations [11, 41, 12], which can be problematic for challenges like safety-critical applications [20, 29] or active learning [38].

Original: Numerous approaches have been proposed to tackle this issue, among which an effective yetsimple method is deep ensemble [23].
Revised: Numerous approaches have been proposed to tackle this issue, among which an effective yet simple method is deep ensemble [26].

Original: Deep ensemble yields state-of-the-art results for uncertaintyaware learning [36, 10] and it does not require elaborate architectural design and hyperparametersearch.
Revised: Deep ensemble yields state-of-the-art results for uncertainty aware learning [41, 12] and it does not require elaborate architectural design and hyperparameter search.

Original: However, while it aggregates multiple separately trained models, it can neither generate newsamples from the posterior to obtain more diverse predictions, nor produce a summarizing averagemodel which improves on the individual models in the ensemble.
Revised: However, while it aggregates multiple separately trained models, it can neither generate new samples from the posterior to obtain more diverse predictions, nor produce a summarizing average model which improves on the individual models in the ensemble.

Original: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation(DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neuralnetworks, we explore the idea of ensembling components of the network architecture and combiningthem to form an enriched collection of model proposals.
Revised: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation (DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neural networks, we explore the idea of ensembling components of the network architecture and combining them to form an enriched collection of model proposals.

Original: DCA inherits the simplicity and effectivenessof deep ensemble, and additionally leads to several new possibilities: Apart from generating adiversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new averageproposal via deep combinatorial weight averaging (DCWA).
Revised: DCA inherits the simplicity and effectiveness of deep ensemble, and additionally leads to several new possibilities: Apart from generating a diversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new average proposal via deep combinatorial weight averaging (DCWA).

Original: Furthermore, DCA training can beneﬁtfrom a consistency enforcing loss, which can produce DCA models that surpass standard deepensemble both in classiﬁcation performance and uncertainty estimation.
Revised: Furthermore, DCA training can beneﬁt from a consistency enforcing loss, which can produce DCA models that surpass standard deep ensemble both in classiﬁcation performance and uncertainty estimation.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: Improving not only the predictive performance but also the uncertainty estimation of neural networkshas been a core objective of Bayesian deep learning, where an abundance of prior work exists.
Revised: Improving not only the predictive performance but also the uncertainty estimation of neural networks has been a core objective of Bayesian deep learning, where an abundance of prior work exists.

Original: Thisincludes methods based on variational inference and weight perturbation such as Bayes by backprop[8, 2] and its variants [40, 4], Bayesian interpretation of dropout including MC dropout [6] andvariational dropout [1, 29], expectation propagation which leads to probabilistic backpropagation[14], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics(SGLD) [39] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [3], stochastic gradientdescent (SGD) as approximate MCMC which results in SWAG [28], as well as Bayesian noisyoptimizers such as variational online Gauss-Newton (VOGN) [18] and approaches using Laplaceapproximation [20, 32].
Revised: This includes methods based on variational inference and weight perturbation such as Bayes by backprop [10, 4] and its variants [45, 6], Bayesian interpretation of dropout including MC dropout [8] and variational dropout [3, 33], expectation propagation which leads to probabilistic backpropagation [16], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics (SGLD) [44] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [5], stochastic gradient descent (SGD) as approximate MCMC which results in SWAG [31], as well as Bayesian noisy optimizers such as variational online Gauss-Newton (VOGN) [21] and approaches using Laplace approximation [23, 36].

Original: Beyond the Bayesian formulation, methods like post-hoc calibration [9, 38]readjust trained networks to produce more calibrated predictions, while approaches like evidentialdeep learning [33] also make use of ideas like subjective logic.
Revised: Beyond the Bayesian formulation, methods like post-hoc calibration [11, 43] readjust trained networks to produce more calibrated predictions, while approaches like evidential deep learning [37] also make use of ideas like subjective logic.

Original: Most relevant to our work is the deep ensemble method [23], which aggregates multiple independentlytrained network models with different initial parameters.
Revised: Most relevant to our work is the deep ensemble method [26], which aggregates multiple independently trained network models with different initial parameters.

Original: Deep ensemble has been shown to producestate-of-the-art results for uncertainty estimation [36, 10] and canalso be combined with methodslike MC dropout [5] or SWAG [42].
Revised: Deep ensemble has been shown to produce state-of-the-art results for uncertainty estimation [41, 12]. It can be combined with methods like MC dropout [7] and SWAG [48], or extended to the hyperparemeter space [47].

Original: Several variants have also been proposed, which often aim atproviding more computationally or memory-efﬁcient alternatives.
Revised: Several variants have also been proposed, which often aim at providing more computationally or memory-efﬁcient alternatives.

Original: This includes snapshot ensemble[15], BatchEnsemble [41], fast geometric ensembling [7], TreeNet [25], etc .
Revised: This includes snapshot ensemble [17], BatchEnsemble [46], fast geometric ensembling [9], TreeNet [28], etc .

Original: In contrast, this workaims at exploring a combinatorial generalization of deep ensemble to obtain new features and improvethe performance of uncertainty estimation.
Revised: In contrast, this work aims at exploring a combinatorial generalization of deep ensemble to obtain new features and improve the performance of uncertainty estimation.

Original: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable tostochastic weight averaging (SWA) [16].
Revised: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable to stochastic weight averaging (SWA) [19].

Original: 1.2 Contributions
Revised: 1.2 Contributions

Original: The main contributions of this paper are the following:
Revised: The main contributions of this paper are the following:

Original: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.
Revised: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.

Original: • We explore DCA at different levels of granularity and propose deep combinatorial weightaveraging (DCWA) for ﬁne-grain DCA models.
Revised: • We explore DCA at different levels of granularity and propose deep combinatorial weight averaging (DCWA) for ﬁne-grain DCA models.

Original: It produces a new average model thatimproves on standard training and is competitive w.r.t.
Revised: It produces a new average model that improves on standard training and is competitive w.r.t.

Original: alternatives like stochastic weightaveraging (SWA) [16].
Revised: alternatives like stochastic weight averaging (SWA) [19].

Original: • We introduce a consistency enforcing loss adapted for DCA training.
Revised: • We introduce a consistency enforcing loss adapted for DCA training.

Original: It strengthens the predictive consistency of DCA model proposals and consistently improves the performance of DCA and DCWA models.
Revised: It strengthens the predictive consistency of DCA model proposals and consistently improves the performance of DCA and DCWA models.

Original: • We conduct experiments on in-domain, distributional shift, and out-of-distribution imageclassiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCAfor uncertainty-aware learning.
Revised: • We conduct experiments on in-domain, distributional shift, and out-of-distribution image classiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCA for uncertainty-aware learning.

Original:  Deep combinatorial aggregation (DCA)
Revised:  Deep combinatorial aggregation (DCA)

Original: In this section, we introduce the deep combinatorial aggregation (DCA) method.
Revised: In this section, we introduce the deep combinatorial aggregation (DCA) method.

Original: To simplify ourdiscussion, we start with a layerwise setting and assume that our base model is a neural network with
Revised: To simplify our discussion, we start with a layerwise setting and assume that our base model is a neural network with

Original: L layers.
Revised: L layers.

Original: This setting can easily be generalized to other DCA variants discussed in Section 2.3.
Revised: This setting can easily be generalized to other DCA variants discussed in Section 2.3.

Original: [Figure]
Revised: [Figure]

Original: 2.1 Methodology
Revised: 2.1 Methodology

Original: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble[23] creates multiple copies of the entire model with different initializations, layerwise DCA insteadcreates multiple instances for each layer in the model.
Revised: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble [26] creates multiple copies of the entire model with different initializations, layerwise DCA instead creates multiple instances for each layer in the model.

Original: Randomly selected instances from networklayers can be combined to form a variety of model proposals.
Revised: Randomly selected instances from network layers can be combined to form a variety of model proposals.

Original: As shown in Figure 1, this results in anexponential number of proposals w.r.t.
Revised: As shown in Figure 1, this results in an exponential number of proposals w.r.t.

Original: the network depth, since n sets of layer instances lead to n Ltotal network proposals.
Revised: the network depth, since n sets of layer instances lead to n L total network proposals.

Original: To ensure consistency among the instance combinations, all layer instances are jointly trained: duringeach feed-forward, a random instance choice from each layer is made to construct a model proposal,and the parameters belonging to the selected layer instances are then updated via backpropagation.
Revised: To ensure consistency among the instance combinations, all layer instances are jointly trained: during each feed-forward, a random instance choice from each layer is made to construct a model proposal, and the parameters belonging to the selected layer instances are then updated via backpropagation.

Original: This differs from deep ensemble [23] where model copies are independently trained.
Revised: This differs from deep ensemble [26] where model copies are independently trained.

Original: In Appendix Awe provide a pseudo-code for layerwise DCA training.
Revised: In Appendix A we provide a pseudo-code for layerwise DCA training.

Original: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCAmodel proposals and aggregate their predictions.
Revised: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCA model proposals and aggregate their predictions.

Original: 2.2 Understanding DCA
Revised: 2.2 Understanding DCA

Original: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoreticalanalysis to understand the assumptions it implicitly makes and their implications.
Revised: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoretical analysis to understand the assumptions it implicitly makes and their implications.

Original: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.
Revised: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.

Original: This translates to the assumption that the weight posterior p ( θ | y, x ) should belayerwise decomposable:
Revised: This translates to the assumption that the weight posterior p ( θ | y, x ) should be layerwise decomposable:

Original: [Equation]
Revised: [Equation]

Original: Is this a valid assumption?
Revised: Is this a valid assumption?

Original: Not exactly.
Revised: Not exactly.

Original: Admittedly, the posterior is proportional to the product ofprior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) oftensatisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to
Revised: Admittedly, the posterior is proportional to the product of prior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) often satisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to

Original: Gaussian prior with constant diagonal covariance matrix.
Revised: Gaussian prior with constant diagonal covariance matrix.

Original: However, the likelihood term p ( y | x, θ ) isin general not decomposable.
Revised: However, the likelihood term p ( y | x, θ ) is in general not decomposable.

Original: In fact, following the directed graphical model [21] interpretation ofstandard neural networks [35], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ )where h l represents hidden neurons in layer l .
Revised: In fact, following the directed graphical model [24] interpretation of standard neural networks [39], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ ) where h l represents hidden neurons in layer l .

Original: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwisedecomposable itself
Revised: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwise decomposable itself

Original: [Equation]
Revised: [Equation]

Original: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h [Equation] L −[-
-]
Revised: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h 1: L −[-
-]

Original: [Equation]
Revised: [Equation]

Original: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.
Revised: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.

Original: Thus approximations are made when we perform layerwise DCA, and experiments show that thisindeed results in some performance penalty.
Revised: Thus approximations are made when we perform layerwise DCA, and experiments show that this indeed results in some performance penalty.

Original: The above analysis also implies that weight aggregationat a coarser multilayer level could alleviate the issue.
Revised: The above analysis also implies that weight aggregation at a coarser multilayer level could alleviate the issue.

Original: This is also conﬁrmed by our empirical ﬁndings( cf .
Revised: This is also conﬁrmed by our empirical ﬁndings ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: 2.3 Granularity of aggregation
Revised: 2.3 Granularity of aggregation

Original: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregationat different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise,multilayer, or modelwise levels.
Revised: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregation at different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise, multilayer, or modelwise levels.

Original: For convolutional neural networks, due to weight-sharing along thespatial dimensions, DCA can work with channel components instead of neurons.
Revised: For convolutional neural networks, due to weight-sharing along the spatial dimensions, DCA can work with channel components instead of neurons.

Original: Note that modelwise
Revised: Note that modelwise

Original: DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained withdistinct subsets of training data which results from the random component selection, and it can beneﬁtfrom consistency enforcing loss introduced in Section 4.
Revised: DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained with distinct subsets of training data which results from the random component selection, and it can beneﬁt from consistency enforcing loss introduced in Section 4.

Original: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more variedpredictions.
Revised: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more varied predictions.

Original: However, the DCA components are more tightly coupled, deviating more signiﬁcantlyfrom the assumption of decomposable posterior, which could lead to worse performance.
Revised: However, the DCA components are more tightly coupled, deviating more signiﬁcantly from the assumption of decomposable posterior, which could lead to worse performance.

Original: This resultsin a tradeoff between performance and prediction variety.
Revised: This results in a tradeoff between performance and prediction variety.

Original: Alternatively, one can also enrich the set ofmodel proposals by using more DCA instances, at the cost of a higher computational budget.
Revised: Alternatively, one can also enrich the set of model proposals by using more DCA instances, at the cost of a higher computational budget.

Original: Thiscan also lead to improved performance ( cf .
Revised: This can also lead to improved performance ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise
Revised: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise

Original: DCA and multilayered DCA.
Revised: DCA and multilayered DCA.

Original: This comes from the fact that component instances from multilayered
Revised: This comes from the fact that component instances from multilayered

Original: DCA variants can have similar behaviors but dissimilar weights.
Revised: DCA variants can have similar behaviors but dissimilar weights.

Original: Neural networks admit a largenumber of equivalent reparameterizations via the reordering of hidden layer neurons.
Revised: Neural networks admit a large number of equivalent reparameterizations via the reordering of hidden layer neurons.

Original: While the jointtraining of DCA enforces consistency among different DCA components of the network model, itdoes not prevent equivalent reordering of hidden neurons inside multilayer components.
Revised: While the joint training of DCA enforces consistency among different DCA components of the network model, it does not prevent equivalent reordering of hidden neurons inside multilayer components.

Original: This issuedoes not occur for (sub)layerwise DCA cases.
Revised: This issue does not occur for (sub)layerwise DCA cases.

Original: To make the distinctions, we refer to (sub)layerwisecases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.
Revised: To make the distinctions, we refer to (sub)layerwise cases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.

Original:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation
Revised:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation

Original: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leadsto an improved parameterization of the base model.
Revised: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leads to an improved parameterization of the base model.

Original: We discuss this procedure here in detail. 
Revised: We discuss this procedure here in detail. Deep

Original: Deep combinatorial weight averaging Since component instances of a ﬁne-grain DCA modelhave compatible weights after the joint training, it is sensible to consider their mean value.
Revised: combinatorial weight averaging Since component instances of a ﬁne-grain DCA model have compatible weights after the joint training, it is sensible to consider their mean value.

Original: Thisproduces a new average parameterization for the base network model.
Revised: This produces a new average parameterization for the base network model.

Original: We refer to this process asdeep combinatorial weight averaging (DCWA) .
Revised: We refer to this process as deep combinatorial weight averaging (DCWA) .

Original: Consider as an example the layerwise DCA model for a base neural network with l layers.
Revised: Consider as an example the layerwise DCA model for a base neural network with l layers.

Original: Afterthe joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . . 
Revised: After the joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . . .

Original: , θ L 1: n ) , DCWAsimply computes the average parameterization ¯ θ =
Revised: , θ L 1: n ) , DCWA simply computes the average parameterization ¯ θ =

Original: (¯ θ 1 , . .
Revised: (¯ θ 1 , . .

Original: , ¯ θ L ) for the base network model, wherefor each layer l , we have ¯ θ l = 1 n (cid:80) ni
Revised: , ¯ θ L ) for the base network model, where for each layer l , we have ¯ θ l = 1 n (cid:80) ni

Original: =1 θ li .
Revised: =1 θ li .

Original: Experiments show that DCWA achieves comparable test accuracy w.r.t.
Revised: Experiments show that DCWA achieves comparable test accuracy w.r.t.

Original: corresponding DCA predictions ( cf .
Revised: corresponding DCA predictions ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: It also consistently outperforms the standard training of the base network, anddelivers comparable results w.r.t.
Revised: It also consistently outperforms the standard training of the base network, and delivers comparable results w.r.t.

Original: to SWA [16] ( cf .
Revised: to SWA [19] ( cf .

Original: Section 5.1).
Revised: Section 5.1).

Original: Comparison to SWA It is interesting to compare DCWA with SWA [16] since they are bothweight averaging schemes that improve the standard training.
Revised: Comparison to SWA It is interesting to compare DCWA with SWA [19] since they are both weight averaging schemes that improve the standard training.

Original: This said, DCWA and SWA are basedon different principles: SWA averages over the SGD trajectory while DCA relies on combiningcomponent aggregations.
Revised: This said, DCWA and SWA are based on different principles: SWA averages over the SGD trajectory while DCA relies on combining component aggregations.

Original: In practice, SWA requires custom learning rate scheduling and carefulchoice of end learning rate.
Revised: In practice, SWA requires custom learning rate scheduling and careful choice of end learning rate.

Original: Also, SWA requires an additional Batch Normalization update [16] toproduce good predictions, which leads to an extra overhead.
Revised: Also, SWA requires an additional Batch Normalization update [19] to produce good predictions, which leads to an extra overhead.

Original: In contrast, DCWA does not have any ofthese issues and is simple to implement and deploy.
Revised: In contrast, DCWA does not have any of these issues and is simple to implement and deploy.

Original:  Consistency enforcing loss for DCA and DCWA
Revised:  Consistency enforcing loss for DCA and DCWA

Original: Through component combination, DCA is able to produce a combinatorial amount of model proposals.
Revised: Through component combination, DCA is able to produce a combinatorial amount of model proposals.

Original: However, during the joint training, each DCA component receives gradient updates from differentmodel proposals.
Revised: However, during the joint training, each DCA component receives gradient updates from different model proposals.

Original: These updates can be inconsistent, which can lead to suboptimal training ofthe DCA model.
Revised: These updates can be inconsistent, which can lead to suboptimal training of the DCA model.

Original: To remedy this issue, we propose in this section a consistency enforcing loss toencourage consistency among DCA model proposals.
Revised: To remedy this issue, we propose in this section a consistency enforcing loss to encourage consistency among DCA model proposals.

Original: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage
Revised: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage

Original: DCA predictions agree with both the ground-truth and the predictions from other model proposals.
Revised: DCA predictions agree with both the ground-truth and the predictions from other model proposals.

Original: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ ,instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )
Revised: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ , instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )

Original: = − log p ( y | x ; ˆ θ ) , the consistencyenforcing loss includes an additional KL divergence term between the predictive output probabilityp ( y | x ; ˆ θ ) and a reference output probability ˜ p :
Revised: = − log p ( y | x ; ˆ θ ) , the consistency enforcing loss includes an additional KL divergence term between the predictive output probability p ( y | x ; ˆ θ ) and a reference output probability ˜ p :

Original: [Equation]
Revised: [Equation]

Original:  Here the reference output probability ˜ p should be chosen to reﬂect the predictions from othermodel proposals.
Revised: ; ˆ θ ) . Here the reference output probability ˜ p should be chosen to reﬂect the predictions from other model proposals.

Original: In practice, we perform multiple feed-forward steps on the same input x usingrandomly chosen model proposals ( ˆ θ ( i ) )
Revised: In practice, we perform multiple feed-forward steps on the same input x using randomly chosen model proposals ( ˆ θ ( i ) )

Original: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simplyuses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) =(cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .
Revised: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simply uses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) = (cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .

Original: Note that the output predictions are reused and one additional feed-forwardpass is needed to compute a ﬁrst reference distribution p (0) .
Revised: Note that the output predictions are reused and one additional feed-forward pass is needed to compute a ﬁrst reference distribution p (0) .

Original: The gradient updates from the multiplefeed-forward steps can be accumulated together for a single optimization step on the global DCAparameter set Θ .
Revised: The gradient updates from the multiple feed-forward steps can be accumulated together for a single optimization step on the global DCA parameter set Θ .

Original: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) toregularize the predictive output distribution p .
Revised: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) to regularize the predictive output distribution p .

Original: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.
Revised: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.

Original: This is helpful for the output predictive distribution pto ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially incases where they strongly disagree with each other.
Revised: This is helpful for the output predictive distribution p to ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially in cases where they strongly disagree with each other.

Original: Relation to existing approaches KL divergence as a regularization term in loss function appearsin many contexts: for Bayesian neural networks [8, 2] it is used to regularize weight posterior; invariational autoencoders (VAE) [19] it regularizes the latent representation.
Revised: Relation to existing approaches KL divergence as a regularization term in loss function appears in many contexts: for Bayesian neural networks [10, 4] it is used to regularize weight posterior; in variational autoencoders (VAE) [22] it regularizes the latent representation.

Original: Applying KL divergenceon output distributions has been proposed by Zhang et al.
Revised: Applying KL divergence on output distributions has been proposed by Zhang et al.

Original: [43] for knowledge distillation, and Songand Chai [37] for collaborative learning.
Revised: [51] for knowledge distillation, and Song and Chai [42] for collaborative learning.

Original: However, to the best of our knowledge, this has not beenextensively explored in the context of uncertainty-aware learning.
Revised: However, to the best of our knowledge, this has not been extensively explored in the context of uncertainty-aware learning.

Original: Also, existing approaches [43, 37]require output predictions from all other models, which is impractical for DCA with a combinatorialamount of model proposals.
Revised: Also, existing approaches [51, 42] require output predictions from all other models, which is impractical for DCA with a combinatorial amount of model proposals.

Original: In comparison, our proposed consistency enforcing loss is efﬁcient andsynergizes well with DCA training.
Revised: In comparison, our proposed consistency enforcing loss is efﬁcient and synergizes well with DCA training.

Original: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions.
Revised: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions.

Original: Thus the consistencyenforcing loss has a mixed effect on uncertainty estimation and could be beneﬁcial in some cases.
Revised: Since model diversity has been found beneﬁcial for ensemble approaches [47, 49, 32], the consistency enforcing loss has a mixed effect on uncertainty estimation and could be advantageous in some cases.

Original: In practice ( cf .
Revised: In practice ( cf .

Original: Section 5.3), we observe that the consistency enforcing loss boosts the predictiveperformance of DCWA models and improves the uncertainty estimation of modelwise DCA.
Revised: Section 5.3), we observe that the consistency enforcing loss boosts the predictive performance of DCWA models and improves the uncertainty estimation of modelwise DCA.

Original: To further analyze the consistency enforcing loss, we empirically study its effect on the classiﬁcationperformance of individual DCA model proposals.
Revised: To further analyze the consistency enforcing loss, we empirically study its effect on the classiﬁcation performance of individual DCA model proposals.

Original: The results are collected in Appendix C. Weobserve that the consistency enforcing loss improves the prediction results of individual models butreduces their diversity.
Revised: The results are collected in Appendix C. We observe that the consistency enforcing loss improves the prediction results of individual models but reduces their diversity.

Original:  Experiments
Revised:  Experiments

Original: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conducta series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 andevaluate the uncertainty estimation of DCA models in Section 5.2.
Revised: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conduct a series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 and evaluate the uncertainty estimation of DCA models in Section 5.2.

Original: Also, ablation studies on DCAvariants are included in Section 5.3 to further analyze several design aspects.
Revised: Also, ablation studies on DCA variants are included in Section 5.3 to further analyze several design aspects.

Original: For all experiments weuse a common preactivation ResNet-20 architecture [11] as the base network, and ﬁve independentruns are executed to produce the ﬁnal results.
Revised: For all experiments we use a common preactivation ResNet-20 architecture [13] as the base network, and ﬁve independent runs are executed to produce the ﬁnal results.

Original: Our implementation uses PyTorch [31] and can run ona single modern GPU with 10Gb VRAM.
Revised: Our implementation uses PyTorch [35] and can run on a single modern GPU with 10Gb VRAM.

Original: 5.1 Image classiﬁcation using DCWA
Revised: 5.1 Image classiﬁcation using DCWA

Original: [Table]
Revised: [Table]

Original: Firstly, we evaluate the performance of the DCWA approach.
Revised: Firstly, we evaluate the performance of the DCWA approach.

Original: For this we use CIFAR-10 [22] and
Revised: For this we use CIFAR-10 [25] and

Original: SVHN [30] datasets and compare against the standard training and SWA [16] as baselines.
Revised: SVHN [34] datasets and compare against the standard training and SWA [19] as baselines.

Original: DCWAmodels are obtained from training layerwise DCA models with ﬁve copies and consistency enforcingloss is used.
Revised: DCWA models are obtained from training layerwise DCA models with ﬁve copies and consistency enforcing loss is used.

Original: SWA models are trained following the settings from Izmailov et al.
Revised: SWA models are trained following the settings from Izmailov et al.

Original: Test accuracyand negative log-likelihood (NLL) on both datasets are reported in Table 1.
Revised: Test accuracy and negative log-likelihood (NLL) on both datasets are reported in Table 1.

Original: We observe that DCWA and SWA have comparable results, and both of them outperform the standardtraining baseline.
Revised: We observe that DCWA and SWA have comparable results, and both of them outperform the standard training baseline.

Original: However, DCWA has the appealing practical advantages over SWA of not requiringcustom learning rate scheduling nor readjustment of batch normalization layers.
Revised: However, DCWA has the appealing practical advantages over SWA of not requiring custom learning rate scheduling nor readjustment of batch normalization layers.

Original: 5.2 DCA for uncertainty-aware learning
Revised: 5.2 DCA for uncertainty-aware learning

Original: We now focus on comparing the uncertainty estimation of DCA models against a selection of practicaluncertainty-aware learning methods, including MC dropout [6], SWAG [28], and deep ensemble[23].
Revised: We now focus on comparing the uncertainty estimation of DCA models against a selection of practical uncertainty-aware learning methods, including MC dropout [8], SWAG [31], and deep ensemble [26].

Original: Also, results from the standard training of the base network are reported for reference.
Revised: Also, results from the standard training of the base network are reported for reference.

Original: Thesebaselines are compared with trunkwise 2 and modelwise DCA models.
Revised: These baselines are compared with trunkwise 2 and modelwise DCA models.

Original: Following the ablation studyresults in Table 3, consistency enforcing loss is used with modelwise DCA models while standardnegative log-likelihood loss is used to train trunkwise DCA models for best performance.
Revised: Following the ablation study results in Table 3, consistency enforcing loss is used with modelwise DCA models while standard negative log-likelihood loss is used to train trunkwise DCA models for best performance.

Original: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain,distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.
Revised: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain, distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.

Original: We considercommon metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score,and expected calibration error (ECE) [9].
Revised: We consider common metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score, and expected calibration error (ECE) [11].

Original: For OOD experiments, we follow the settings of Lianget al.
Revised: For OOD experiments, we follow the settings of Liang et al.

Original: [27] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,
Revised: [30] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,

Original: AUPR-in, AUPR-out, and AUROC in Appendix B.3.
Revised: AUPR-in, AUPR-out, and AUROC in Appendix B.3.

Original: Overall, we observe that modelwise DCA produces the best results both in terms of predictiveperformance and uncertainty estimation.
Revised: Overall, we observe that modelwise DCA produces the best results both in terms of predictive performance and uncertainty estimation.

Original: It is the method of choice if the performance of uncertaintyaware learning is the only concern.
Revised: It is the method of choice if the performance of uncertaintyaware learning is the only concern.

Original: Trunkwise DCA is overall competitive with deep ensemble andmoreover it produces more diverse model proposals.
Revised: Trunkwise DCA is overall competitive with deep ensemble and moreover it produces more diverse model proposals.

Original: It offers a trade-off for more varied predictions.
Revised: It offers a trade-off for more varied predictions.

Original: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [22] and SVHN [30] datasets.
Revised: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [25] and SVHN [34] datasets.

Original: For all trainingwe use SGD with momentum 0 .
Revised: For all training we use SGD with momentum 0 .

Original: We use a drop rate of 0 . 1 for MC dropout, and for SWAG wefollow the settings from Maddox et al.
Revised: We use a drop rate of 0 . 1 for MC dropout, and for SWAG we follow the settings from Maddox et al.

Original: Deep ensemble is performed on ﬁve separate basenetworks.
Revised: Deep ensemble is performed on ﬁve separate base networks.

Original: And trunkwise and modelwise DCA models also use ﬁve copies.
Revised: And trunkwise and modelwise DCA models also use ﬁve copies.

Original: The results on CIFAR-10are summarized in Table 2 and SVHN results are in Appendix B.1.
Revised: The results on CIFARare summarized in Table 2 and SVHN results are in Appendix B.1.

Original: [Table]
Revised: [Table]

Original: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracyand uncertainty estimation.
Revised: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracy and uncertainty estimation.

Original: Also, trunkwise DCA slightly outperforms deep ensemble in this case.
Revised: Also, trunkwise DCA slightly outperforms deep ensemble in this case.

Original: SVHN results in Appendix B.1 also conﬁrm these observations.
Revised: SVHN results in Appendix B.1 also conﬁrm these observations.

Original: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatchbetween the training data and the test set.
Revised: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatch between the training data and the test set.

Original: This calls for additional distributional shift experiments,
Revised: This calls for additional distributional shift experiments,

Original: [Figure]
Revised: [Figure]

Original: which gradually transform the in-domain test set to make it increasingly different than the trainingdata
Revised: which gradually transform the in-domain test set to make it increasingly different than the training data In our case

Original: In our case, we use the CIFAR-10-C [12] dataset, which applies a wide range of corruptions tothe original CIFAR-10 test set.
Revised: [Figure] we use the CIFAR-10-C [14] dataset [Figure] which applies a wide range of corruptions to the original CIFAR-10 test set.

Original: CIFAR-10-C quantizes corruptions into ﬁve severity levels wherehigher levels generate stronger corruptions and are more dissimilar to the in-domain test set.
Revised: CIFAR-10-C quantizes corruptions into ﬁve severity levels where higher levels generate stronger corruptions and are more dissimilar to the in-domain test set.

Original: We usethe models previously trained on CIFAR-10 in-domain training data, and evaluate their performanceon CIFAR-10-C separately for each severity level.
Revised: We use the models previously trained on CIFAR-10 in-domain training data [Figure] and evaluate their performance on CIFAR-10-C separately for each severity level.

Original: Figure 2 summarizes the results.
Revised: Figure 2 summarizes the results.

Original: Similar to the in-domain scenario, modelwise DCA has the best results overall.
Revised: Similar to the in-domain scenario [Figure] modelwise DCA has the best results overall.

Original: Trunkwise DCA hascomparable test accuracies w.r.t.
Revised: Trunkwise DCA has comparable test accuracies w.r [Figure]

Original:  deep ensemble.
Revised: t. deep ensemble.

Original: It tends to outperform deep ensemble in terms ofnegative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by theadditional plots in Appendix B.2. [Figure] o s i t i v e
Revised: It tends to outperform deep ensemble in terms of negative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by the additional plots in Appendix B.

Original: [Figure]

Original: e ( i n

Original: [Figure]
Revised: [Figure]

Original: Modelwise DCA shows the best performance for
Revised: Modelwise DCA shows the best performance for

Original: OOD detection, followed by trunkwise DCA.
Revised: OOD detection, followed by trunkwise DCA.

Original: Out-of-distribution results Additionally, wecomplement the distributional shift experimentswith out-of-distribution image detection [13].
Revised: Out-of-distribution results Additionally, we complement the distributional shift experiments with out-of-distribution image detection [15].

Original: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.
Revised: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.

Original: Thus we have a binary classiﬁcation setting.
Revised: Thus we have a binary classiﬁcation setting.

Original: For our case, we reuse again themodels trained on in-domain CIFAR-10 trainingdata.
Revised: For our case, we reuse again the models trained on in-domain CIFAR-10 training data.

Original: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while testsamples from SVHN are treated as negative outliers.
Revised: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while test samples from SVHN are treated as negative outliers.

Original: We show the ROC curves for DCA modelsand their baselines in Figure 3.
Revised: We show the ROC curves for DCA models and their baselines in Figure 3.

Original: Quantitative metrics following Liang et al.
Revised: Quantitative metrics following Liang et al.

Original: [27], which include
Revised: [30], which include

Original: FPR95, detection error, AUPR-in, AUPR-outand AUROC, are reported in Appendix B.3.
Revised: FPR95, detection error, AUPR-in, AUPR-out and AUROC, are reported in Appendix B.3.

Original: Again, we observe that modelwise DCAachieves the best OOD performance.
Revised: Again, we observe that modelwise DCA achieves the best OOD performance.

Original: Trunkwise DCA has the second-best performance and outperforms deep ensemble and SWAG.
Revised: Trunkwise DCA has the second-best performance and outperforms deep ensemble and SWAG.

Original: Interestingly, MC dropout turns out to perform poorlyin the OOD case and is even worse than the standard training baseline.
Revised: Inter- estingly, MC dropout turns out to perform poorly in the OOD case and is even worse than the standard training baseline.

Original: The quantitative results in Appendix B.3 agree with theobservations from the ROC curve.
Revised: The quantitative results in Appendix B.3 agree with the ROC curve.

Original: 5.3 Ablation studies
Revised: 5.3 Ablation studies

Original: We wrap up the experiment section with a series of ablation studies on variants of DCA models.
Revised: We wrap up the experiment section with a series of ablation studies on variants of DCA models.

Original: We consider three main aspects: the effect of granularity, the usage of consistency enforcing lossduring training, and the inﬂuence of component instance count on DCA predictive results.
Revised: We consider three main aspects: the effect of granularity, the usage of consistency enforcing loss during training, and the inﬂuence of component instance count on DCA predictive results.

Original: We usethe in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.
Revised: We use the in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.

Original: Effect of granularity To understand the effect of DCA component granularity on the performanceof DCA models, we deﬁne ﬁve DCA model variants with different component granularities.
Revised: Effect of granularity To understand the effect of DCA component granularity on the performance of DCA models, we deﬁne ﬁve DCA model variants with different component granularities.

Original: Thisincludes DCA models whose components are channels, layers, residual blocks, residual trunks, andentire models.
Revised: This includes DCA models whose components are channels, layers, residual blocks, residual trunks, and entire models.

Original: All DCA model variants are constructed with ﬁve sets of component instances andtrained with the consistency enforcing loss (NLL loss leads to similar observations).
Revised: All DCA model variants are constructed with ﬁve sets of component instances and trained with the consistency enforcing loss (NLL loss leads to similar observations).

Original: We summarizethe results of these DCA variants and their DCWA models in Figure 4.
Revised: We summarize the results of these DCA variants and their DCWA models in Figure 4.

Original: [Figure]
Revised: [Figure]

Original: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.
Revised: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.

Original: The metrics are all plotted in log scale.
Revised: The metrics are all plotted in log scale.

Original: Several interesting observations can be drawn from this experiment: For DCA models, coarsercomponent granularity generally leads to better results.
Revised: Several interesting observations can be drawn from this experiment: For DCA models, coarser component granularity generally leads to better results.

Original: This agrees with the analysis in Section 2.2.
Revised: This agrees with the analysis in Section 2.2.

Original: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component
Revised: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component

Original: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWAvariants improve on the standard baseline, while coarse-grain DCWA variants result in considerablyworse performance than the standard baseline.
Revised: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWA variants improve on the standard baseline, while coarse-grain DCWA variants result in considerably worse performance than the standard baseline.

Original: This also matches our analysis in Section 2.3.
Revised: This also matches our analysis in Section 2.3.

Original: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better thanrandom ( i.e . 10%) accuracy.
Revised: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better than random ( i.e . 10%) accuracy.

Original: In this case, reparameterization mismatch only happens on the sidebranches of the residual blocks, and the trained weights on the main branch are still able to producemeaningful predictions despite the erroneous parameterization on the residual side branches.[Table] Modelwise
Revised: In this case, reparameterization mismatch only happens on the side branches of the residual blocks, and the trained weights on the main branch are still able to produce meaningful predictions despite the erroneous parameterization on the residual side branches.

Original: [Table]
Revised: [Table]

Original: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained withnegative log-likelihood loss with those trained with consistency enforcing loss.
Revised: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained with negative log-likelihood loss with those trained with consistency enforcing loss.

Original: We summarize theresults on CIFAR-10 in Table 3.
Revised: We summarize the results on CIFAR-10 in Table 3.

Original: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.
Revised: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.

Original: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of
Revised: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of

Original: NLL and ECE.
Revised: NLL and ECE.

Original: On the other hand, negative log-likelihood loss is the better choice for the trunkwise
Revised: On the other hand, negative log-likelihood loss is the better choice for the trunkwise

Original: DCA model.
Revised: DCA model.

Original: In Appendix B.4 we provide additional results for other DCA variants.
Revised: In Appendix B.4 we provide additional results for other DCA variants.

Original: [Figure]
Revised: [Figure]

Original: Number of component instances Finally, weinvestigate the inﬂuence of DCA instance count.
Revised: Number of component instances Finally, we investigate the inﬂuence of DCA instance count.

Original: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial tomultiply the training epochs and per-minibatchgradient backpropagations accordingly.
Revised: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial to multiply the training epochs and per-minibatch gradient backpropagations accordingly.

Original: For example, in our experiments, we schedule 200epochs for standard baseline training.
Revised: For example, in our experiments, we scheduleepochs for standard baseline training.

Original: Then, fortraining a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradientover three backpropagations on randomly selected DCA proposals for parameter update.
Revised: Then, for training a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradient over three backpropagations on randomly selected DCA proposals for parameter update.

Original: Wetrain a series of layerwise DCA models with two,three, four, and ﬁve sets of component instances,and collect their results in Figure 5.
Revised: We train a series of layerwise DCA models with two, three, four, and ﬁve sets of component instances, and collect their results in Figure 5.

Original: In general, we observe that more DCA instanceslead to better results (higher accuracy and lower
Revised: In general, we observe that more DCA instances lead to better results (higher accuracy and lower

Original: NLL loss in Figure 5).
Revised: NLL loss in Figure 5).

Original: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.
Revised: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.

Original: Moreover, we see that a minimal DCA modelwith two sets of components can already signiﬁcantly improve the performance compared to thestandard base network training case.
Revised: Moreover, we see that a minimal DCA model with two sets of components can already signiﬁcantly improve the performance compared to the standard base network training case.

Original:  Discussion
Revised:  Discussion

Original: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that hasalso a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learningrate scheduling of SWAG, requires no elaborate hyperparameter search.
Revised: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that has also a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learning rate scheduling of SWAG, requires no elaborate hyperparameter search.

Original: We ﬁnd that coarse-grain
Revised: We ﬁnd that coarse-grain

Original: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms allother uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios;trunkwise DCA produces more diverse predictions while still delivering competitive performancew.r.t.
Revised: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms all other uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios; trunkwise DCA produces more diverse predictions while still delivering competitive performance w.r.t.

Original: deep ensemble, which is the best among existing baselines.
Revised: deep ensemble, which is the best among existing baselines.

Original: Additionally, we discover thatﬁne-grain DCA can produce a DCWA parameterization for the base network.
Revised: Additionally, we discover that ﬁne-grain DCA can produce a DCWA parameterization for the base network.

Original: DCWA improves thepredictive performance compared to the standard baseline training.
Revised: DCWA improves the predictive performance compared to the standard baseline training.

Original: It has on-par performance with
Revised: It has on-par performance with

Original: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.
Revised: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.

Original: Thus
Revised: Thus

Original: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.
Revised: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.

Original: Similar to deep ensemble, DCA requires multiple times the computational budget to perform thejoint training, compared to the standard baseline.
Revised: Similar to deep ensemble, DCA requires multiple times the computational budget to perform the joint training, compared to the standard baseline.

Original: While in this work we focused on the predictivepower, we plan to improve the efﬁciency of DCA in future work.
Revised: While in this work we focused on the predictive power, we plan to improve the efﬁciency of DCA in future work.

Original: Apart from this, it can also beinteresting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain
Revised: Apart from this, it can also be interesting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain

Original: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimationscan only be attained via coarse-grain DCAs instead.
Revised: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimations can only be attained via coarse-grain DCAs instead.

Original: Additionally, it is possible to combine DCA withother uncertainty-aware methods, which might further improve the performance.
Revised: Additionally, it is possible to combine DCA with other uncertainty-aware methods, which might further improve the performance.

Original: We plan to furtherinvestigate these aspects in future work.
Revised: We plan to further investigate these aspects in future work.

Revised: Acknowledgments and Disclosure of Funding

Revised: This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC

Revised: Advanced Grant SIMULACRON.

Revised: The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers and area chair for their constructive feedback.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: In Section 6.
Revised: In Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?[Yes] cf .
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: footnote (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: Algorithm 1 depicts the pseudocode for training a layerwise DCA model. Inputs : Dataset D , neural network f with n sets of parameters Θ = ( θ 11: n , . .

Original: spent on participant compensation?
Revised: , θ L 1: n ) , with θ li the i -th parameter instance of layer l . sampleDCAProposal Draw random indices i 1 , . . , i L ∈ ; return ˆ θ = ( θ 1 i 1 , . . .

Revised: , θ Li

Revised: ) ; 4 repeat foreach minibatch B ∈ D do ˆ θ ← sampleDCAProposal( Θ ); loss ← feedForward( f, ˆ θ , B ); ∆ ˆ θ ← loss.backward(); Θ ← updateDCAParameters( Θ , ∆ ˆ θ ); end 11 until end of training ;

Revised: B Additional experimental results B.1 In-domain results on SVHN dataset

Revised: Table 4 summarizes the results of in-domain image classiﬁcation on SVHN dataset.

Revised: [Table]

Revised: [Table]

Revised: B.2 Additional plots for distributional shift experiment

Revised: Figure 6 reports Brier scores and ECEs of the distributional shift experiments on CIFAR-10-C.

Revised: B.3 Additional quantitative results for out-of-distribution experiment

Revised: Table 5 provides quantitative metrics for the OOD detection task using CIFAR-10 as the in-domain dataset and SVHN as the OOD outliers.

Revised: The following metrics are reported:

Revised: False positive rate at 95% true positive rate (FPR@95%)

Revised: A threshold metric which reports the false positive rate at the threshold where the true positive rate is 95%; Detection error The minimum of average values of false positive rate and false negative rate evaluated at different decision boundaries d : min d ∈ [0 , 1] ( FPR | d + FNR | d ) / 2 ; Area under the receiver operating characteristic curve (AUROC)

Revised: This metric reports the area under the receiver operating characteristic (ROC) curve;

Revised: [Figure]

Revised: [Table]

Revised: Area under the precision–recall curve (AUPR-in / AUPR-out)

Revised: This metric measures the area under the precision–recall (AUPR) curve, AUPR-in reports the AUPR where the in-domain data are taken as positive samples, whereas AUPR-out instead treats OOD data as positive samples and in-domain data as negative ones.

Revised: B.4 Additional results for ablation study on consistency enforcing loss

Revised: Table 6 displays additional results for channelwise, layerwise and blockwise DCA variants.

Revised: C Performance of individual DCA model proposals

Revised: To further analyze the effect of the consistency enforcing loss for DCA models, we inspect the predictions of the individual model copies from modelwise DCAs training with and without the consistency enforcing loss.

Revised: The results are reported in Table 7.

Revised: We observe that individual models trained with the consistency enforcing loss have better results compared to the NLL case, which shows that CEL improves the training of individual components.

Revised: Compared to the baseline standard training, individual model proposals from trained DCA models

Revised: [Table]

Revised: [Table]

Revised: have worse classiﬁcation performance, however their aggregation yields superior results both in terms of accuracy and uncertainty estimation.

Revised: In Table

Revised: [Table]

Revised: we also see that results from individual models trained without CEL have larger standard deviations, which hints that the predictions might be more diverse in this case.

Revised: To further conﬁrm this, we employ the following metrics to quantify the diversity of individual predictions [1] (assuming we have M individual models and C output classes, KL denotes the KL-divergence, Var denotes the variance, and for a given input x , p mc ( x ) denotes the predicted probability of class c from model m ):

Revised: Pairwise KL divergence

Revised: [Equation]

Revised: Classwise variance

Revised: [Equation]

Revised: Jensen-Shannon divergence

Revised: [Equation]

Revised: The results are collected in Table 8 and conﬁrm that individual models trained without CEL have more diverse predictions.

Revised: [Table]

Revised: D Additional results using VGG and DenseNet

Revised: In previous experiments we focus on the preactivation ResNet-20 base model.

Revised: Since the DCA approach does not rely on speciﬁc choice of network architecture, we expect the conclusions to hold for other network structures.

Revised: To empirically verify this

Revised: [Table]

Revised: we conduct additional in-domain experiments on CIFAR-10 using two more architectures: VGG-16 [40] and DenseNet-40 [18].

Revised: The results are summarized in Table 9 for the VGG backbone and in Table 10 for the DenseNet case.

Revised: We see that modelwise DCA consistently achieves the best results both in terms of accuracy and uncertainty estimation for all tested network architectures.

Revised: [Table]

Revised: [Table]

Revised: E Text classiﬁcation experiments

Revised: Additionally

Revised: [Table]

Revised: we consider the performance of DCA for natural language processing tasks.

Revised: Especially, we address the task of text classiﬁcation on the AG News dataset [50] using a Seq2seq model with attention [2].

Revised: We adapt from an existing implementation 3 and collect the results in Table 11.

Revised: We observe that modelwise DCA achieves the best accuracy when trained with the NLL loss.

Revised: Using the consistency enforcing loss, modelwise DCA achieves the best ECE.

Original: [N/A]
Revised: [Table]