Original: Deep Combinatorial Aggregation
Revised: Deep Combinatorial Aggregation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.
Revised: Neural networks are known to produce poor uncertainty estimations, and a variety of approaches have been proposed to remedy this issue.

Original: This includes deepensemble, a simple and effective method that achieves state-of-the-art results foruncertainty-aware learning tasks.
Revised: This includes deepensemble, a simple and effective method that achieves state-of-the-art results foruncertainty-aware learning tasks.

Original: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).
Revised: In this work, we explore a combinatorial generalization of deep ensemble called deep combinatorial aggregation (DCA).

Original: DCAcreates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.
Revised: DCAcreates multiple instances of network components and aggregates their combinations to produce diversiﬁed model proposals and predictions.

Original: DCA components canbe deﬁned at different levels of granularity.
Revised: DCA components canbe deﬁned at different levels of granularity.

Original: And we discovered that coarse-grain
Revised: And we discovered that coarse-grain

Original: DCAs can outperform deep ensemble for uncertainty-aware learning both in termsof predictive performance and uncertainty estimation.
Revised: DCAs can outperform deep ensemble for uncertainty-aware learning both in termsof predictive performance and uncertainty estimation.

Original: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weightaveraging (DCWA) can improve the baseline training.
Revised: For ﬁne-grain DCAs, we discover that an average parameterization approach named deep combinatorial weightaveraging (DCWA) can improve the baseline training.

Original: It is on par with stochasticweight averaging (SWA) but does not require any custom training schedule oradaptation of BatchNorm layers.
Revised: It is on par with stochasticweight averaging (SWA) but does not require any custom training schedule oradaptation of BatchNorm layers.

Original: Furthermore, we propose a consistency enforcingloss that helps the training of DCWA and modelwise DCA.
Revised: Furthermore, we propose a consistency enforcingloss that helps the training of DCWA and modelwise DCA.

Original: We experiment onin-domain, distributional shift, and out-of-distribution image classiﬁcation tasks,and empirically conﬁrm the effectiveness of DCWA and DCA approaches.
Revised: We experiment onin-domain, distributional shift, and out-of-distribution image classiﬁcation tasks,and empirically conﬁrm the effectiveness of DCWA and DCA approaches.

Original:  Introduction
Revised:  Introduction

Original: Deep learning has achieved groundbreaking progress and neural networks are now widely used invarious domains [24].
Revised: Deep learning has achieved groundbreaking progress and neural networks are now widely used invarious domains [24].

Original: However, they are known to produce poor uncertainty estimations [9, 36, 10]
Revised: However, they are known to produce poor uncertainty estimations [9, 36, 10]

Original: which can be problematic for challenges like safety-critical applications [17, 26] or active learning[34].
Revised: which can be problematic for challenges like safety-critical applications [17, 26] or active learning[34].

Original: Numerous approaches have been proposed to tackle this issue, among which an effective yetsimple method is deep ensemble [23].
Revised: Numerous approaches have been proposed to tackle this issue, among which an effective yetsimple method is deep ensemble [23].

Original: Deep ensemble yields state-of-the-art results for uncertaintyaware learning [36, 10] and it does not require elaborate architectural design and hyperparametersearch.
Revised: Deep ensemble yields state-of-the-art results for uncertaintyaware learning [36, 10] and it does not require elaborate architectural design and hyperparametersearch.

Original: However, while it aggregates multiple separately trained models, it can neither generate newsamples from the posterior to obtain more diverse predictions, nor produce a summarizing averagemodel which improves on the individual models in the ensemble.
Revised: However, while it aggregates multiple separately trained models, it can neither generate newsamples from the posterior to obtain more diverse predictions, nor produce a summarizing averagemodel which improves on the individual models in the ensemble.

Original: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation(DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neuralnetworks, we explore the idea of ensembling components of the network architecture and combiningthem to form an enriched collection of model proposals.
Revised: Motivated by the success of deep ensemble, in this paper, we propose deep combinatorial aggregation(DCA), which generalizes via a combinatorial perspective: given the hierarchical structure of neuralnetworks, we explore the idea of ensembling components of the network architecture and combiningthem to form an enriched collection of model proposals.

Original: DCA inherits the simplicity and effectivenessof deep ensemble, and additionally leads to several new possibilities: Apart from generating adiversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new averageproposal via deep combinatorial weight averaging (DCWA).
Revised: DCA inherits the simplicity and effectivenessof deep ensemble, and additionally leads to several new possibilities: Apart from generating adiversiﬁed set of model proposals, we discover that ﬁne-grain DCA can lead to a new averageproposal via deep combinatorial weight averaging (DCWA).

Original: Furthermore, DCA training can beneﬁtfrom a consistency enforcing loss, which can produce DCA models that surpass standard deepensemble both in classiﬁcation performance and uncertainty estimation.
Revised: Furthermore, DCA training can beneﬁtfrom a consistency enforcing loss, which can produce DCA models that surpass standard deepensemble both in classiﬁcation performance and uncertainty estimation.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: Improving not only the predictive performance but also the uncertainty estimation of neural networkshas been a core objective of Bayesian deep learning, where an abundance of prior work exists.
Revised: Improving not only the predictive performance but also the uncertainty estimation of neural networkshas been a core objective of Bayesian deep learning, where an abundance of prior work exists.

Original: Thisincludes methods based on variational inference and weight perturbation such as Bayes by backprop[8, 2] and its variants [40, 4], Bayesian interpretation of dropout including MC dropout [6] andvariational dropout [1, 29], expectation propagation which leads to probabilistic backpropagation[14], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics(SGLD)
Revised: Thisincludes methods based on variational inference and weight perturbation such as Bayes by backprop[8, 2] and its variants [40, 4], Bayesian interpretation of dropout including MC dropout [6] andvariational dropout [1, 29], expectation propagation which leads to probabilistic backpropagation[14], Markov chain Monte Carlo (MCMC) with methods like stochastic gradient Langevin dynamics(SGLD)

Original: [39] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [3], stochastic gradientdescent (SGD) as approximate MCMC which results in SWAG [28], as well as Bayesian noisyoptimizers such as variational online Gauss-Newton (VOGN) [18] and approaches using Laplaceapproximation [20, 32].
Revised: [39] and stochastic gradient Hamiltonian Monte Carlo (SGHMC) [3], stochastic gradientdescent (SGD) as approximate MCMC which results in SWAG [28], as well as Bayesian noisyoptimizers such as variational online Gauss-Newton (VOGN) [18] and approaches using Laplaceapproximation [20, 32].

Original: Beyond the Bayesian formulation, methods like post-hoc calibration [9, 38]readjust trained networks to produce more calibrated predictions, while approaches like evidentialdeep learning [33] also make use of ideas like subjective logic.
Revised: Beyond the Bayesian formulation, methods like post-hoc calibration [9, 38]readjust trained networks to produce more calibrated predictions, while approaches like evidentialdeep learning [33] also make use of ideas like subjective logic.

Original: Most relevant to our work is the deep ensemble method [23], which aggregates multiple independentlytrained network models with different initial parameters.
Revised: Most relevant to our work is the deep ensemble method [23], which aggregates multiple independentlytrained network models with different initial parameters.

Original: Deep ensemble has been shown to producestate-of-the-art results for uncertainty estimation [36, 10] and can also be combined with methodslike MC dropout [5] or SWAG [42].
Revised: Deep ensemble has been shown to producestate-of-the-art results for uncertainty estimation [36, 10] and can also be combined with methodslike MC dropout [5] or SWAG [42].

Original: Several variants have also been proposed, which often aim atproviding more computationally or memory-efﬁcient alternatives.
Revised: Several variants have also been proposed, which often aim atproviding more computationally or memory-efﬁcient alternatives.

Original: This includes snapshot ensemble[15], BatchEnsemble [41], fast geometric ensembling [7], TreeNet [25], etc .
Revised: This includes snapshot ensemble[15], BatchEnsemble [41], fast geometric ensembling [7], TreeNet [25], etc .

Original: In contrast, this workaims at exploring a combinatorial generalization of deep ensemble to obtain new features and improvethe performance of uncertainty estimation.
Revised: In contrast, this workaims at exploring a combinatorial generalization of deep ensemble to obtain new features and improvethe performance of uncertainty estimation.

Original: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable tostochastic weight averaging (SWA) [16].
Revised: Lastly, our proposed DCWA method provides an alternative weight averaging scheme comparable tostochastic weight averaging (SWA) [16].

Original: 1.2 Contributions
Revised: 1.2 Contributions

Original: The main contributions of this paper are the following:
Revised: The main contributions of this paper are the following:

Original: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.
Revised: • We propose deep combinatorial aggregation, a combinatorial generalization of deep ensemble that can produce more diverse model proposals and predictions.

Original: • We explore DCA at different levels of granularity and propose deep combinatorial weightaveraging (DCWA) for ﬁne-grain DCA models.
Revised: • We explore DCA at different levels of granularity and propose deep combinatorial weightaveraging (DCWA) for ﬁne-grain DCA models.

Original: It produces a new average model thatimproves on standard training and is competitive w.r.t.
Revised: It produces a new average model thatimproves on standard training and is competitive w.r.t.

Original: alternatives like stochastic weightaveraging (SWA) [16].
Revised: alternatives like stochastic weightaveraging (SWA) [16].

Original: • We introduce a consistency enforcing loss adapted for DCA training.
Revised: • We introduce a consistency enforcing loss adapted for DCA training.

Original: It strengthens the
Revised: It strengthens the

Original: predictive consistency of DCA model proposals and consistently improves the performance
Revised: predictive consistency of DCA model proposals and consistently improves the performance

Original: of DCA and DCWA models.
Revised: of DCA and DCWA models.

Original: • We conduct experiments on in-domain, distributional shift, and out-of-distribution imageclassiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCAfor uncertainty-aware learning.
Revised: • We conduct experiments on in-domain, distributional shift, and out-of-distribution imageclassiﬁcation tasks, which validate our analysis and demonstrate the effectiveness of DCAfor uncertainty-aware learning.

Original:  Deep combinatorial aggregation (DCA)
Revised:  Deep combinatorial aggregation (DCA)

Original: In this section, we introduce the deep combinatorial aggregation (DCA) method.
Revised: In this section, we introduce the deep combinatorial aggregation (DCA) method.

Original: To simplify ourdiscussion, we start with a layerwise setting and assume that our base model is a neural network with
Revised: To simplify ourdiscussion, we start with a layerwise setting and assume that our base model is a neural network with

Original: L layers.
Revised: L layers.

Original: This setting can easily be generalized to other DCA variants discussed in Section 2.3.
Revised: This setting can easily be generalized to other DCA variants discussed in Section 2.3.

Original: [Figure]
Revised: [Figure]

Original: 2.1 Methodology
Revised: 2.1 Methodology

Original: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble[23] creates multiple copies of the entire model with different initializations, layerwise DCA insteadcreates multiple instances for each layer in the model.
Revised: The main idea of layerwise deep combinatorial aggregation is straightforward: while deep ensemble[23] creates multiple copies of the entire model with different initializations, layerwise DCA insteadcreates multiple instances for each layer in the model.

Original: Randomly selected instances from networklayers can be combined to form a variety of model proposals.
Revised: Randomly selected instances from networklayers can be combined to form a variety of model proposals.

Original: As shown in Figure 1, this results in anexponential number of proposals w.r.t.
Revised: As shown in Figure 1, this results in anexponential number of proposals w.r.t.

Original: the network depth, since n sets of layer instances lead to n Ltotal network proposals.
Revised: the network depth, since n sets of layer instances lead to n Ltotal network proposals.

Original: To ensure consistency among the instance combinations, all layer instances are jointly trained: duringeach feed-forward, a random instance choice from each layer is made to construct a model proposal,and the parameters belonging to the selected layer instances are then updated via backpropagation.
Revised: To ensure consistency among the instance combinations, all layer instances are jointly trained: duringeach feed-forward, a random instance choice from each layer is made to construct a model proposal,and the parameters belonging to the selected layer instances are then updated via backpropagation.

Original: This differs from deep ensemble [23] where model copies are independently trained.
Revised: This differs from deep ensemble [23] where model copies are independently trained.

Original: In Appendix Awe provide a pseudo-code for layerwise DCA training.
Revised: In Appendix Awe provide a pseudo-code for layerwise DCA training.

Original: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCAmodel proposals and aggregate their predictions.
Revised: During inference time, to obtain uncertainty-aware predictions, one can simply sample multiple DCAmodel proposals and aggregate their predictions.

Original: 2.2 Understanding DCA
Revised: 2.2 Understanding DCA

Original: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoreticalanalysis to understand the assumptions it implicitly makes and their implications.
Revised: While DCA is a simple and intuitive procedure, it is helpful to conduct a more in-depth theoreticalanalysis to understand the assumptions it implicitly makes and their implications.

Original: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.
Revised: Being able to freely combine samples of layer parameters assumes that they follow mutually independent distributions.

Original: This translates to the assumption that the weight posterior p ( θ | y, x ) should belayerwise decomposable:
Revised: This translates to the assumption that the weight posterior p ( θ | y, x ) should belayerwise decomposable:

Original: [Equation]
Revised: [Equation]

Original: Is this a valid assumption?
Revised: Is this a valid assumption?

Original: Not exactly.
Revised: Not exactly.

Original: Admittedly, the posterior is proportional to the product ofprior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) oftensatisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to
Revised: Admittedly, the posterior is proportional to the product ofprior p ( θ ) and likelihood p ( y | x, θ ) ( i.e . p ( θ | y, x ) ∝ p ( θ ) p ( y | x, θ ) ), and the weight prior p ( θ ) oftensatisﬁes a layerwise independence assumption, e.g ., commonly used weight decay is equivalent to

Original: Gaussian prior with constant diagonal covariance matrix.
Revised: Gaussian prior with constant diagonal covariance matrix.

Original: However, the likelihood term p ( y | x, θ ) isin general not decomposable.
Revised: However, the likelihood term p ( y | x, θ ) isin general not decomposable.

Original: In fact, following the directed graphical model [21] interpretation ofstandard neural networks [35], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ )where h l represents hidden neurons in layer l .
Revised: In fact, following the directed graphical model [21] interpretation ofstandard neural networks [35], the base network represents the overall distribution p ( h 1: L − 1 , y | x, θ )where h l represents hidden neurons in layer l .

Original: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwisedecomposable itself
Revised: Interestingly, p ( h 1: L − 1 , y | x, θ ) is actually layerwisedecomposable itself

Original: [Equation]
Revised: [Equation]

Original: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h
Revised: Nevertheless, the likelihood term p ( y | x, θ ) requires marginalization of all hidden neurons h

Original: [Equation]
Revised: [Equation]

Original: L − 

Revised: L −

Original: [Equation]
Revised: [Equation]

Original: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.
Revised: This entangles the layer parameters, and p ( y | x, θ ) is no longer decomposable in general.

Original: Thus approximations are made when we perform layerwise DCA, and experiments show that thisindeed results in some performance penalty.
Revised: Thus approximations are made when we perform layerwise DCA, and experiments show that thisindeed results in some performance penalty.

Original: The above analysis also implies that weight aggregationat a coarser multilayer level could alleviate the issue.
Revised: The above analysis also implies that weight aggregationat a coarser multilayer level could alleviate the issue.

Original: This is also conﬁrmed by our empirical ﬁndings( cf .
Revised: This is also conﬁrmed by our empirical ﬁndings( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: 2.3 Granularity of aggregation
Revised: 2.3 Granularity of aggregation

Original: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregationat different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise,multilayer, or modelwise levels.
Revised: The analysis in Section 2.2 raises an interesting concern about doing deep combinatorial aggregationat different levels of granularity: From ﬁnest to coarsest, DCA can be deﬁned at neuronwise, layerwise,multilayer, or modelwise levels.

Original: For convolutional neural networks, due to weight-sharing along thespatial dimensions, DCA can work with channel components instead of neurons.
Revised: For convolutional neural networks, due to weight-sharing along thespatial dimensions, DCA can work with channel components instead of neurons.

Original: Note that modelwise
Revised: Note that modelwise

Original: DCA is similar to deep ensemble, except that in each epoch DCA model copies are trained withdistinct subsets of training data which results from the random component selection, and it can beneﬁtfrom consistency enforcing loss introduced in Section 4.
Revised: DCA is similar to deep ensemble, except that its model copies are trained with distinct data withineach training epoch, and it can beneﬁt from consistency enforcing loss introduced in Section 4.

Original: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more variedpredictions.
Revised: In general, DCA with ﬁner granularity generates a greater amount of model proposals and more variedpredictions.

Original: However, the DCA components are more tightly coupled, deviating more signiﬁcantlyfrom the assumption of decomposable posterior, which could lead to worse performance.
Revised: However, the DCA components are more tightly coupled, deviating more signiﬁcantlyfrom the assumption of decomposable posterior, which could lead to worse performance.

Original: This resultsin a tradeoff between performance and prediction variety.
Revised: This resultsin a tradeoff between performance and prediction variety.

Original: Alternatively, one can also enrich the set ofmodel proposals by using more DCA instances, at the cost of a higher computational budget.
Revised: Alternatively, one can also enrich the set ofmodel proposals by using more DCA instances, at the cost of a higher computational budget.

Original: Thiscan also lead to improved performance ( cf .
Revised: Thiscan also lead to improved performance ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise
Revised: Among the broad range of granularities, there exists a notable dichotomy between (sub)layerwise

Original: DCA and multilayered DCA.
Revised: DCA and multilayered DCA.

Original: This comes from the fact that component instances from multilayered
Revised: This comes from the fact that component instances from multilayered

Original: DCA variants can have similar behaviors but dissimilar weights.
Revised: DCA variants can have similar behaviors but dissimilar weights.

Original: Neural networks admit a largenumber of equivalent reparameterizations via the reordering of hidden layer neurons.
Revised: Neural networks admit a largenumber of equivalent reparameterizations via the reordering of hidden layer neurons.

Original: While the jointtraining of DCA enforces consistency among different DCA components of the network model, itdoes not prevent equivalent reordering of hidden neurons inside multilayer components.
Revised: While the jointtraining of DCA enforces consistency among different DCA components of the network model, itdoes not prevent equivalent reordering of hidden neurons inside multilayer components.

Original: This issuedoes not occur for (sub)layerwise DCA cases.
Revised: This issuedoes not occur for (sub)layerwise DCA cases.

Original: To make the distinctions, we refer to (sub)layerwisecases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.
Revised: To make the distinctions, we refer to (sub)layerwisecases as ﬁne-grain DCA and multilayered cases as coarse-grain DCA.

Original:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation
Revised:  Deep combinatorial weight averaging (DCWA) for ﬁne-grain aggregation

Original: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leadsto an improved parameterization of the base model.
Revised: For ﬁne-grain DCA models, it turns out that averaging the learned weights of DCA components leadsto an improved parameterization of the base model.

Original: We discuss this procedure here in detail.
Revised: We discuss this procedure here in detail.

Original: Deep combinatorial weight averaging Since component instances of a ﬁne-grain DCA modelhave compatible weights after the joint training, it is sensible to consider their mean value.
Revised: Deep combinatorial weight averaging Since component instances of a ﬁne-grain DCA modelhave compatible weights after the joint training, it is sensible to consider their mean value.

Original: Thisproduces a new average parameterization for the base network model.
Revised: Thisproduces a new average parameterization for the base network model.

Original: We refer to this process asdeep combinatorial weight averaging (DCWA) .
Revised: We refer to this process asdeep combinatorial weight averaging (DCWA) .

Original: Consider as an example the layerwise DCA model for a base neural network with l layers.
Revised: Consider as an example the layerwise DCA model for a base neural network with l layers.

Original: Afterthe joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . .
Revised: Afterthe joint training of n sets of DCA layer instances parameterized by Θ = ( θ 11: n , . .

Original: , θ L 1: n ) , DCWAsimply computes the average parameterization ¯ θ =
Revised: , θ L 1: n ) , DCWAsimply computes the average parameterization ¯ θ =

Original: (¯ θ 1 , . .
Revised: (¯ θ 1 , . .

Original: , ¯ θ L ) for the base network model, wherefor each layer l , we have ¯ θ l = 1 n (cid:80) ni
Revised: , ¯ θ L ) for the base network model, wherefor each layer l , we have ¯ θ l = 1 n (cid:80) ni

Original: =1 θ li .
Revised: =1 θ li .

Original: Experiments show that DCWA achieves comparable test accuracy w.r.t.
Revised: Experiments show that DCWA achieves comparable test accuracy w.r.t.

Original: corresponding DCA predictions ( cf .
Revised: corresponding DCA predictions ( cf .

Original: Section 5.3).
Revised: Section 5.3).

Original: It also consistently outperforms the standard training of the base network, anddelivers comparable results w.r.t.
Revised: It also consistently outperforms the standard training of the base network, anddelivers comparable results w.r.t.

Original: to SWA [16] ( cf .
Revised: to SWA [16] ( cf .

Original: Section 5.1).
Revised: Section 5.1).

Original: Comparison to SWA It is interesting to compare DCWA with SWA [16] since they are bothweight averaging schemes that improve the standard training.
Revised: Comparison to SWA It is interesting to compare DCWA with SWA [16] since they are bothweight averaging schemes that improve the standard training.

Original: This said, DCWA and SWA are basedon different principles: SWA averages over the SGD trajectory while DCA relies on combiningcomponent aggregations.
Revised: This said, DCWA and SWA are basedon different principles: SWA averages over the SGD trajectory while DCA relies on combiningcomponent aggregations.

Original: In practice, SWA requires custom learning rate scheduling and carefulchoice of end learning rate.
Revised: In practice, SWA requires custom learning rate scheduling and carefulchoice of end learning rate.

Original: Also, SWA requires an additional Batch Normalization update [16] toproduce good predictions, which leads to an extra overhead.
Revised: Also, SWA requires an additional Batch Normalization update [16] toproduce good predictions, which leads to an extra overhead.

Original: In contrast, DCWA does not have any ofthese issues and is simple to implement and deploy.
Revised: In contrast, DCWA does not have any ofthese issues and is simple to implement and deploy.

Original:  Consistency enforcing loss for DCA and DCWA
Revised:  Consistency enforcing loss for DCA and DCWA

Original: Through component combination, DCA is able to produce a combinatorial amount of model proposals.
Revised: Through component combination, DCA is able to produce a combinatorial amount of model proposals.

Original: However, during the joint training, each DCA component receives gradient updates from differentmodel proposals.
Revised: However, during the joint training, each DCA component receives gradient updates from differentmodel proposals.

Original: These updates can be inconsistent, which can lead to suboptimal training ofthe DCA model.
Revised: These updates can be inconsistent, which can lead to suboptimal training ofthe DCA model.

Original: To remedy this issue, we propose in this section a consistency enforcing loss toencourage consistency among DCA model proposals.
Revised: To remedy this issue, we propose in this section a consistency enforcing loss toencourage consistency among DCA model proposals.

Original: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage
Revised: Consistency enforcing loss To promote consistency among DCA model proposals, we encourage

Original: DCA predictions agree with both the ground-truth and the predictions from other model proposals.
Revised: DCA predictions agree with both the ground-truth and the predictions from other model proposals.

Original: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ ,instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )
Revised: To achieve this, given an input x with ground-truth y and a DCA model proposal parameterized by ˆ θ ,instead of minimizing the negative log-likelihood (cid:96) NLL ( x, y ; ˆ θ )

Original: = − log p ( y | x ; ˆ θ ) , the consistencyenforcing loss includes an additional KL divergence term between the predictive output probabilityp ( y | x ; ˆ θ ) and a reference output probability ˜ p :
Revised: = − log p ( y | x ; ˆ θ ) , the consistencyenforcing loss includes an additional KL divergence term between the predictive output probabilityp ( y | x ; ˆ θ ) and a reference output probability ˜ p :

Original: [Equation]
Revised: [Equation]

Original: Here the reference output probability ˜ p should be chosen to reﬂect the predictions from othermodel proposals.
Revised: Here the reference output probability ˜ p should be chosen to reﬂect the predictions from othermodel proposals.

Original: In practice, we perform multiple feed-forward steps on the same input x usingrandomly chosen model proposals ( ˆ θ ( i ) )
Revised: In practice, we perform multiple feed-forward steps on the same input x usingrandomly chosen model proposals ( ˆ θ ( i ) )

Original: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simplyuses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) =(cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .
Revised: 1 ≤ i ≤ s , and the i -th consistency enforcing loss (cid:96) ( i ) simplyuses the output probability prediction p ( i − 1) from the previous step as reference distribution: (cid:96) ( i ) =(cid:96) ( x, y, p ( i − 1) ; ˆ θ ( i ) ) .

Original: Note that the output predictions are reused and one additional feed-forwardpass is needed to compute a ﬁrst reference distribution p (0) .
Revised: Note that the output predictions are reused and one additional feed-forwardpass is needed to compute a ﬁrst reference distribution p (0) .

Original: The gradient updates from the multiplefeed-forward steps can be accumulated together for a single optimization step on the global DCAparameter set Θ .
Revised: The gradient updates from the multiplefeed-forward steps can be accumulated together for a single optimization step on the global DCAparameter set Θ .

Original: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) toregularize the predictive output distribution p .
Revised: Also, observe that the consistency enforcing loss uses a forward KL divergence D KL (˜ p (cid:107) p ) toregularize the predictive output distribution p .

Original: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.
Revised: The forward KL divergence D KL (˜ p (cid:107) p ) is “zeroavoiding” and allows p to be ﬂat and uncertain.

Original: This is helpful for the output predictive distribution pto ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially incases where they strongly disagree with each other.
Revised: This is helpful for the output predictive distribution pto ﬁnd a good compromise between the reference prediction ˜ p and the ground-truth y , especially incases where they strongly disagree with each other.

Original: Relation to existing approaches KL divergence as a regularization term in loss function appearsin many contexts: for Bayesian neural networks [8, 2] it is used to regularize weight posterior; invariational autoencoders (VAE) [19] it regularizes the latent representation.
Revised: Relation to existing approaches KL divergence as a regularization term in loss function appearsin many contexts: for Bayesian neural networks [8, 2] it is used to regularize weight posterior; invariational autoencoders (VAE) [19] it regularizes the latent representation.

Original: Applying KL divergenceon output distributions has been proposed by Zhang et al.
Revised: Applying KL divergenceon output distributions has been proposed by Zhang et al.

Original: [43] for knowledge distillation, and Songand Chai [37] for collaborative learning.
Revised: [43] for knowledge distillation, and Songand Chai [37] for collaborative learning.

Original: However, to the best of our knowledge, this has not beenextensively explored in the context of uncertainty-aware learning.
Revised: However, to the best of our knowledge, this has not beenextensively explored in the context of uncertainty-aware learning.

Original: Also, existing approaches [43, 37]require output predictions from all other models, which is impractical for DCA with a combinatorialamount of model proposals.
Revised: Also, existing approaches [43, 37]require output predictions from all other models, which is impractical for DCA with a combinatorialamount of model proposals.

Original: In comparison, our proposed consistency enforcing loss is efﬁcient andsynergizes well with DCA training.
Revised: In comparison, our proposed consistency enforcing loss is efﬁcient andsynergizes well with DCA training.

Original: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions.
Revised: Consistency vs . diversity While strengthening the predictive consistency can help the DCA parameter training, it has the risk of reducing the diversity of model predictions, which might worsenthe uncertainty estimation.

Original: Thus the consistencyenforcing loss has a mixed effect on uncertainty estimation and could be beneﬁcial in some cases.
Revised: In the extreme case where all DCA component instances are identical( ∀ l ∈ [1 : L ] , ∀ i, j ∈ [1 : n ] , θ li = θ lj ), the predictions become identical and perfectly consistent,however, we simply revert back to the baseline model learning.

Original: In practice ( cf .
Revised: In practice ( cf .

Original: Section 5.3), we observe that the consistency enforcing loss boosts the predictiveperformance of DCWA models and improves the uncertainty estimation of modelwise DCA.
Revised: Section 5.3), we observe that the consistency enforcing loss boosts the predictiveperformance of DCWA models and improves the uncertainty estimation of modelwise DCA.

Original: To further analyze the consistency enforcing loss, we empirically study its effect on the classiﬁcationperformance of individual DCA model proposals.

Original: The results are collected in Appendix C. Weobserve that the consistency enforcing loss improves the prediction results of individual models butreduces their diversity.

Original:  Experiments
Revised:  Experiments

Original: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conducta series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 andevaluate the uncertainty estimation of DCA models in Section 5.2.
Revised: To empirically evaluate the effectiveness of the proposed DCA and DCWA models, we conducta series of experiments: we benchmark the predictive performance of DCWA in Section 5.1 andevaluate the uncertainty estimation of DCA models in Section 5.2.

Original: Also, ablation studies on DCAvariants are included in Section 5.3 to further analyze several design aspects.
Revised: Also, ablation studies on DCAvariants are included in Section 5.3 to further analyze several design aspects.

Original: For all experiments weuse a common preactivation ResNet-20 architecture [11] as the base network, and ﬁve independentruns are executed to produce the ﬁnal results.
Revised: For all experiments weuse a common preactivation ResNet-20 architecture [11] as the base network, and ﬁve independentruns are executed to produce the ﬁnal results.

Original: Our implementation uses PyTorch [31] and can run ona single modern GPUwith 10Gb VRAM.
Revised: Our implementation uses PyTorch [31] and can run ona single modern GPU 10Gb VRAM.

Original: 5.1 Image classiﬁcation using DCWA
Revised: 5.1 Image classiﬁcation using DCWA

Original: [Table]
Revised: [Table]

Original: Firstly, we evaluate the performance of the DCWA approach.
Revised: Firstly, we evaluate the performance of the DCWA approach.

Original: For this we use CIFAR-10 [22] and
Revised: For this we use CIFAR-10 [22] and

Original: SVHN [30] datasets and compare against the standard training and SWA [16] as baselines.
Revised: SVHN [30] datasets and compare against the standard training and SWA [16] as baselines.

Original: DCWAmodels are obtained from training layerwise DCA models with ﬁve copies and consistency enforcingloss is used.
Revised: DCWA models are obtained from training layerwise DCA models with ﬁve copies and consistency enforcingloss is used.

Original: SWA models are trained following the settings from Izmailov et al.
Revised: SWA models are trained following the settings from Izmailov et al.

Original: Test accuracyand negative log-likelihood (NLL) on both datasets are reported in Table 1.
Revised: Test accuracyand negative log-likelihood (NLL) on both datasets are reported in Table 1.

Original: We observe that DCWA and SWA have comparable results, and both of them outperform the standardtraining baseline.
Revised: We observe that DCWA and SWA have comparable results, and both of them outperform the standardtraining baseline.

Original: However, DCWA has the appealing practical advantages over SWA of not requiringcustom learning rate scheduling nor readjustment of batch normalization layers.
Revised: However, DCWA has the appealing practical advantages over SWA of not requiringcustom learning rate scheduling nor readjustment of batch normalization layers.

Original: 5.2 DCA for uncertainty-aware learning
Revised: 5.2 DCA for uncertainty-aware learning

Original: We now focus on comparing the uncertainty estimation of DCA models against a selection of practicaluncertainty-aware learning methods, including MC dropout [6], SWAG [28], and deep ensemble[23].
Revised: We now focus on comparing the uncertainty estimation of DCA models against a selection of practicaluncertainty-aware learning methods, including MC dropout [6], SWAG [28], and deep ensemble[23].

Original: Also, results from the standard training of the base network are reported for reference.
Revised: Also, results from the standard training of the base network are reported for reference.

Original: Thesebaselines are compared with trunkwise 2 and modelwise DCA models.
Revised: Thesebaselines are compared with trunkwise 2 and modelwise DCA models.

Original: Following the ablation studyresults in Table 3, consistency enforcing loss is used with modelwise DCA models while standardnegative log-likelihood loss is used to train trunkwise DCA models for best performance.
Revised: Following the ablation studyresults in Table 3, consistency enforcing loss is used with modelwise DCA models while standardnegative log-likelihood loss is used to train trunkwise DCA models for best performance.

Original: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain,distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.
Revised: To evaluate the quality of uncertainty estimation, we conduct a series of experiments on in-domain,distributional shift, and out-of-distribution (OOD) image classiﬁcation problems.

Original: We considercommon metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score,and expected calibration error (ECE) [9].
Revised: We considercommon metrics for uncertainty quantiﬁcation, including negative log-likelihood (NLL), Brier score,and expected calibration error (ECE) [9].

Original: For OOD experiments, we follow the settings of Lianget al.
Revised: For OOD experiments, we follow the settings of Lianget al.

Original: [27] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,
Revised: [27] and plot the ROC-curve, and report quantitative metrics such as FPR95, detection error,

Original: AUPR-in, AUPR-out, and AUROC in Appendix B.3.
Revised: AUPR-in, AUPR-out, and AUROC in Appendix B.3.

Original: Overall, we observe that modelwise DCA produces the best results both in terms of predictiveperformance and uncertainty estimation.
Revised: Overall, we observe that modelwise DCA produces the best results both in terms of predictiveperformance and uncertainty estimation.

Original: It is the method of choice if the performance of uncertaintyaware learning is the only concern.
Revised: It is the method of choice if the performance of uncertaintyaware learning is the only concern.

Original: Trunkwise DCA is overall competitive with deep ensemble andmoreover it produces more diverse model proposals.
Revised: Trunkwise DCA is overall competitive with deep ensemble andmoreover it produces more diverse model proposals.

Original: It offers a trade-off for more varied predictions.
Revised: It offers a trade-off for more varied predictions.

Original: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [22] and SVHN [30] datasets.
Revised: In-domain comparison We start by training the baselines and DCA models for the standard indomain image classiﬁcation problems on the CIFAR-10 [22] and SVHN [30] datasets.

Original: For all trainingwe use SGD with momentum 0 .
Revised: For all trainingwe use SGD with momentum 0 .

Original: We use a drop rate of 0 . 1 for MC dropout, and for SWAG wefollow the settings from Maddox et al.
Revised: We use a drop rate of 0 . 1 for MC dropout, and for SWAG wefollow the settings from Maddox et al.

Original: Deep ensemble is performed on ﬁve separate basenetworks.
Revised: Deep ensemble is performed on ﬁve separate basenetworks.

Original: And trunkwise and modelwise DCA models also use ﬁve copies.
Revised: And trunkwise and modelwise DCA models also use ﬁve copies.

Original: The results on CIFAR-10are summarized in Table 2 and SVHN results are in Appendix B.1.
Revised: The results on CIFAR-10are summarized in Table 2 and SVHN results are in Appendix B.1.

Original: [Table]
Revised: [Table]

Original: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracyand uncertainty estimation.
Revised: In general, modelwise DCA obtains the best performance both in terms of classiﬁcation accuracyand uncertainty estimation.

Original: Also, trunkwise DCA slightly outperforms deep ensemble in this case.
Revised: Also, trunkwise DCA slightly outperforms deep ensemble in this case.

Original: SVHN results in Appendix B.1 also conﬁrm these observations.
Revised: SVHN results in Appendix B.1 also conﬁrm these observations.

Original: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatchbetween the training data and the test set.
Revised: Distributional shift experiments In-domain experiments alone are not sufﬁcient to fully characterize the quality of uncertainty estimation, since they do not reﬂect the case when there is a mismatchbetween the training data and the test set.

Original: This calls for additional distributional shift experiments, [Figure] which gradually transform the in-domain test set to make it increasingly different than the trainingdata
Revised: This calls for additional distributional shift experiments,which gradually transform the in-domain test set to make it increasingly different than the trainingdata.

Original: In our case, we use the CIFAR-10-C [12] dataset, which applies a wide range of corruptions tothe original CIFAR-10 test set.
Revised: In our case, we use the CIFAR-10-C [12] dataset, which applies a wide range of corruptions to [Figure] the original CIFAR-10 test set.

Original: CIFAR-10-C quantizes corruptions into ﬁve severity levels wherehigher levels generate stronger corruptions and are more dissimilar to the in-domain test set.
Revised: CIFAR-10-C quantizes corruptions into ﬁve severity levels wherehigher levels generate stronger corruptions and are more dissimilar to the in-domain test set.

Original: We usethe models previously trained on CIFAR-10 in-domain training data, and evaluate their performanceon CIFAR-10-C separately for each severity level.
Revised: We usethe models previously trained on CIFAR-10 in-domain training data, and evaluate their performanceon CIFAR-10-C separately for each severity level.

Original: Figure 2 summarizes the results.
Revised: Figure 2 summarizes the results.

Original: Similar to the in-domain scenario, modelwise DCA has the best results overall.
Revised: Similar to the in-domain scenario, modelwise DCA has the best results overall.

Original: Trunkwise DCA hascomparable test accuracies w.r.t.
Revised: Trunkwise DCA hascomparable test accuracies w.r.t.

Original: deep ensemble.
Revised: deep ensemble.

Original: It tends to outperform deep ensemble in terms ofnegative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by theadditional plots in Appendix B.2.
Revised: It tends to outperform deep ensemble in terms ofnegative log-likelihood, however, it is worse in terms of Brier score and ECE, as indicated by theadditional plots in Appendix B.2.

Original: [Figure]
Revised: [Figure]

Original: o s i t i v e [Figure] e ( i n
Revised: - domain dataset and SVHN as the OOD dataset.

Original: [Figure]

Original: Modelwise DCA shows the best performance for
Revised: Modelwise DCA shows the best performance for

Original: OOD detection, followed by trunkwise DCA.
Revised: OOD detection, followed by trunkwise DCA.

Original: Out-of-distribution results Additionally, wecomplement the distributional shift experimentswith out-of-distribution image detection [13].
Revised: Out-of-distribution results Additionally, wecomplement the distributional shift experimentswith out-of-distribution image detection [13].

Original: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.
Revised: The goal is to distinguish in-domain test samples from outliers that are drawn from a different dataset.

Original: Thus we have a binary classiﬁcation setting.
Revised: Thus we have a binary classiﬁcation setting.

Original: For our case, we reuse again themodels trained on in-domain CIFAR-10 trainingdata.
Revised: For our case, we reuse again themodels trained on in-domain CIFAR-10 trainingdata.

Original: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while testsamples from SVHN are treated as negative outliers.
Revised: Samples from the CIFAR-10 test set are regarded as in-domain positive samples while testsamples from SVHN are treated as negative outliers.

Original: We show the ROC curves for DCA modelsand their baselines in Figure 3.
Revised: We show the ROC curves for DCA modelsand their baselines in Figure 3.

Original: Quantitative metrics following Liang et al.
Revised: Quantitative metrics following Liang et al.

Original: [27], which include
Revised: [27], which include

Original: FPR95, detection error, AUPR-in, AUPR-outand AUROC, are reported in Appendix B.3.
Revised: FPR95, detection error, AUPR-in, AUPR-outand AUROC, are reported in Appendix B.3.

Original: Again, we observe that modelwise DCAachieves the best OOD performance.
Revised: Again, we observe that modelwise DCAachieves the best OOD performance.

Original: Trunkwise DCA has the second-best performance and outperforms deep ensemble and SWAG.
Revised: Trunkwise DCA has the second-best performance andoutperforms deep ensemble and SWAG.

Original: Interestingly, MC dropout turns out to perform poorlyin the OOD case and is even worse than the standard training baseline.
Revised: Interestingly, MC dropout turns out to perform poorly in the OOD case and is even worse than the standard training baseline.

Original: The quantitative results in
Revised: The quantitative results in

Original: Appendix B.3 agree with the observations from the ROC curve.
Revised: Appendix B.3 agree with the observations from the ROC curve.

Original: 5.3 Ablation studies
Revised: 5.3 Ablation studies

Original: We wrap up the experiment section with a series of ablation studies on variants of DCA models.
Revised: We wrap up the experiment section with a series of ablation studies on variants of DCA models.

Original: We consider three main aspects: the effect of granularity, the usage of consistency enforcing lossduring training, and the inﬂuence of component instance count on DCA predictive results.
Revised: We consider three main aspects: the effect of granularity, the usage of consistency enforcing lossduring training, and the inﬂuence of component instance count on DCA predictive results.

Original: We usethe in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.
Revised: We usethe in-domain CIFAR-10 classiﬁcation task to compare the performance of different DCA variants.

Original: Effect of granularity To understand the effect of DCA component granularity on the performanceof DCA models, we deﬁne ﬁve DCA model variants with different component granularities.
Revised: Effect of granularity To understand the effect of DCA component granularity on the performanceof DCA models, we deﬁne ﬁve DCA model variants with different component granularities.

Original: Thisincludes DCA models whose components are channels, layers, residual blocks, residual trunks, andentire models.
Revised: Thisincludes DCA models whose components are channels, layers, residual blocks, residual trunks, andentire models.

Original: All DCA model variants are constructed with ﬁve sets of component instances andtrained with the consistency enforcing loss (NLL loss leads to similar observations).
Revised: All DCA model variants are constructed with ﬁve sets of component instances andtrained with the consistency enforcing loss (NLL loss leads to similar observations).

Original: We summarizethe results of these DCA variants and their DCWA models in Figure 4.
Revised: We summarizethe results of these DCA variants and their DCWA models in Figure 4.

Original: [Figure]
Revised: [Figure]

Original: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.
Revised: As references, the results of standard baseline training (dashed lines), as well as deep ensemble (dotted lines), are also provided.

Original: The metrics are all plotted in log scale.
Revised: The metrics are all plotted in log scale.

Original: Several interesting observations can be drawn from this experiment: For DCA models, coarsercomponent granularity generally leads to better results.
Revised: Several interesting observations can be drawn from this experiment: For DCA models, coarsercomponent granularity generally leads to better results.

Original: This agrees with the analysis in Section 2.2.
Revised: This agrees with the analysis in Section 2.2.

Original: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component
Revised: For DCWA models we observe a clear distinction between ﬁne-grain (channel and layer component

Original: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWAvariants improve on the standard baseline, while coarse-grain DCWA variants result in considerablyworse performance than the standard baseline.
Revised: DCAs) and coarse-grain (block, trunk, and model component DCAs) variants: ﬁne-grain DCWAvariants improve on the standard baseline, while coarse-grain DCWA variants result in considerablyworse performance than the standard baseline.

Original: This also matches our analysis in Section 2.3.
Revised: This also matches our analysis in Section 2.3.

Original: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better thanrandom ( i.e . 10%) accuracy.
Revised: Interestingly, the DCWA model with block component is still able to yield signiﬁcantly better thanrandom ( i.e . 10%) accuracy.

Original: In this case, reparameterization mismatch only happens on the sidebranches of the residual blocks, and the trained weights on the main branch are still able to producemeaningful predictions despite the erroneous parameterization on the residual side branches.[Table] Modelwise
Revised: In this case, reparameterization mismatch only happens on the sidebranches of the residual blocks, and the trained weights on the main branch are still able to producemeaningful predictions despite the erroneous parameterization on the residual side branches.

Original: [Table]
Revised: [Table]

Original: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained withnegative log-likelihood loss with those trained with consistency enforcing loss.
Revised: Consistency enforcing loss To analyze the inﬂuence of consistency enforcing loss for DCA training, we compare DCWA, trunkwise DCA, and modelwise DCA models which are trained with negative log-likelihood loss with those trained with consistency enforcing loss.

Original: We summarize theresults on CIFAR-10 in Table 3.
Revised: We summarize theresults on CIFAR-10 in Table 3.

Original: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.
Revised: We observe that the consistency enforcing loss improves the overall performance of the DCWA model.

Original: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of
Revised: It is also beneﬁcial for the uncertainty estimation of the modelwise DCA model, both in terms of

Original: NLL and ECE.
Revised: NLL and ECE.

Original: On the other hand, negative log-likelihood loss is the better choice for the trunkwise
Revised: On the other hand, negative log-likelihood loss is the better choice for the trunkwise

Original: DCA model.
Revised: DCA model.

Original: In Appendix B.4 we provide additional results for other DCA variants.
Revised: In Appendix B.4 we provide additional results for other DCA variants.

Original: [Figure]
Revised: [Figure]

Original: Number of component instances Finally, weinvestigate the inﬂuence of DCA instance count.
Revised: Number of component instances Finally, we investigate the inﬂuence of DCA instance count.

Original: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial tomultiply the training epochs and per-minibatchgradient backpropagations accordingly.
Revised: To properly train all the DCA component instances jointly, we ﬁnd out that it is beneﬁcial tomultiply the training epochs and per-minibatchgradient backpropagations accordingly.

Original: For example, in our experiments, we schedule 200epochs for standard baseline training.
Revised: For example, in our experiments, we schedule 200epochs for standard baseline training.

Original: Then, fortraining a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradientover three backpropagations on randomly selected DCA proposals for parameter update.
Revised: Then, fortraining a DCA model with three sets of component instances we use 600 epochs, and accumulate for each minibatch an average gradientover three backpropagations on randomly selected DCA proposals for parameter update.

Original: Wetrain a series of layerwise DCA models with two,three, four, and ﬁve sets of component instances,and collect their results in Figure 5.
Revised: Wetrain a series of layerwise DCA models with two,three, four, and ﬁve sets of component instances,and collect their results in Figure 5.

Original: In general, we observe that more DCA instanceslead to better results (higher accuracy and lower
Revised: In general, we observe that more DCA instanceslead to better results (higher accuracy and lower

Original: NLL loss in Figure 5).
Revised: NLL loss in Figure 5).

Original: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.
Revised: Therefore, DCA provides a reasonable trade-off between the computational footprint and the predictive power.

Original: Moreover, we see that a minimal DCA modelwith two sets of components can already signiﬁcantly improve the performance compared to thestandard base network training case.
Revised: Moreover, we see that a minimal DCA modelwith two sets of components can already signiﬁcantly improve the performance compared to thestandard base network training case.

Original:  Discussion
Revised:  Discussion

Original: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that hasalso a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learningrate scheduling of SWAG, requires no elaborate hyperparameter search.
Revised: In this work, we propose deep combinatorial aggregation, a generalization of deep ensemble that hasalso a simple setup and, unlike the choice of dropout rate for MC dropout or the custom learningrate scheduling of SWAG, requires no elaborate hyperparameter search.

Original: We ﬁnd that coarse-grain
Revised: We ﬁnd that coarse-grain

Original: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms allother uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios;trunkwise DCA produces more diverse predictions while still delivering competitive performancew.r.t.
Revised: DCAs are well suited for uncertainty-aware learning: modelwise DCA consistently outperforms allother uncertainty-aware learning methods for in-domain, distributional shift, and OOD scenarios;trunkwise DCA produces more diverse predictions while still delivering competitive performancew.r.t.

Original: deep ensemble, which is the best among existing baselines.
Revised: deep ensemble, which is the best among existing baselines.

Original: Additionally, we discover thatﬁne-grain DCA can produce a DCWA parameterization for the base network.
Revised: Additionally, we discover thatﬁne-grain DCA can produce a DCWA parameterization for the base network.

Original: DCWA improves thepredictive performance compared to the standard baseline training.
Revised: DCWA improves thepredictive performance compared to the standard baseline training.

Original: It has on-par performance with
Revised: It has on-par performance with

Original: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.
Revised: SWA but does not need custom learning rate scheduling and readjustment of batchnorm layers.

Original: Thus
Revised: Thus

Original: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.
Revised: DCWA can be an attractive alternative to SWA that is easy to implement and deploy.

Original: Similar to deep ensemble, DCA requires multiple times the computational budget to perform thejoint training, compared to the standard baseline.
Revised: Similar to deep ensemble, DCA requires multiple times the computational budget to perform thejoint training, compared to the standard baseline.

Original: While in this work we focused on the predictivepower, we plan to improve the efﬁciency of DCA in future work.
Revised: While in this work we focused on the predictivepower, we plan to improve the efﬁciency of DCA in future work.

Original: Apart from this, it can also beinteresting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain
Revised: Apart from this, it can also beinteresting to search for a variant that combines the beneﬁts of both the ﬁne-grain and coarse-grain

Original: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimationscan only be attained via coarse-grain DCAs instead.
Revised: DCAs, as currently DCWA is only possible for ﬁne-grain DCAs, while good uncertainty estimationscan only be attained via coarse-grain DCAs instead.

Original: Additionally, it is possible to combine DCA withother uncertainty-aware methods, which might further improve the performance.
Revised: Additionally, it is possible to combine DCA withother uncertainty-aware methods, which might further improve the performance.

Original: We plan to furtherinvestigate these aspects in future work.
Revised: We plan to furtherinvestigate these aspects in future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: In Section 6.
Revised: In Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] cf .
Revised: [Yes] cf .

Original: footnote
Revised: footnote

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]